2024-03-08T22:40:40.479 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-08T22:40:40.479 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi --owner scheduled_rishabh@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} --archive /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268 -- /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/orig.config.yaml 2024-03-08T22:40:40.536 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T22:40:40.624 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268 branch: rishabh-2024mar8 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} email: null first_in_suite: false job_id: '7588268' kernel: client: branch: testing flavor: default kdb: true sha1: dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-2024mar8 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 7d63482aa89628d733e298984a762389b625c761 subvols: create: 2 subvol_options: --namespace-isolated ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 7d63482aa89628d733e298984a762389b625c761 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2024mar8 sha1: 7d63482aa89628d733e298984a762389b625c761 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1249 sha1: 7d63482aa89628d733e298984a762389b625c761 sleep_before_teardown: 0 subset: 481/512 suite: fs suite_branch: rishabh-2024mar8 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7d63482aa89628d733e298984a762389b625c761 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi046.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALBRr494KWIsmHoG8SNXCt8oaJNNWBZyX2bnIn3439QfzY/L09+/PZFEz4wvhNKUhpD6T3z1xyvpAkzsBaH8cQ= smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApETSWUc4zesRT8aG+3tvjZpZMzdaf+a+WX2EAxGi94LDYCONTl3r/H69D8WFbpi2Gwr5lKc56lZHYskcBu3XE= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLttwb5bQnofE4tfKvDurVh8KmCGkXTY0TehT5eoqDhg1ji2Ug5C9xLD52pTrlFzc45Y0CuEVvZ8qxAfd77R0I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_19:54:47 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-08T22:40:40.624 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa; will attempt to use it 2024-03-08T22:40:40.625 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks 2024-03-08T22:40:40.625 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-08T22:40:40.627 INFO:teuthology.task.internal:Checking packages... 2024-03-08T22:40:40.649 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7d63482aa89628d733e298984a762389b625c761' 2024-03-08T22:40:40.649 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-08T22:40:40.650 INFO:teuthology.packaging:ref: None 2024-03-08T22:40:40.650 INFO:teuthology.packaging:tag: None 2024-03-08T22:40:40.650 INFO:teuthology.packaging:branch: rishabh-2024mar8 2024-03-08T22:40:40.650 INFO:teuthology.packaging:sha1: 7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:40:40.650 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-2024mar8 2024-03-08T22:40:40.869 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1965.g7d63482a 2024-03-08T22:40:40.871 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-08T22:40:40.877 INFO:teuthology.task.internal:no buildpackages task found 2024-03-08T22:40:40.877 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-08T22:40:40.929 INFO:teuthology.task.internal:Saving configuration 2024-03-08T22:40:40.952 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-08T22:40:40.963 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T22:40:40.983 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.147602', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALBRr494KWIsmHoG8SNXCt8oaJNNWBZyX2bnIn3439QfzY/L09+/PZFEz4wvhNKUhpD6T3z1xyvpAkzsBaH8cQ='} 2024-03-08T22:40:41.007 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.144262', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApETSWUc4zesRT8aG+3tvjZpZMzdaf+a+WX2EAxGi94LDYCONTl3r/H69D8WFbpi2Gwr5lKc56lZHYskcBu3XE='} 2024-03-08T22:40:41.030 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.146016', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLttwb5bQnofE4tfKvDurVh8KmCGkXTY0TehT5eoqDhg1ji2Ug5C9xLD52pTrlFzc45Y0CuEVvZ8qxAfd77R0I='} 2024-03-08T22:40:41.031 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-08T22:40:41.037 INFO:teuthology.task.internal:roles: ubuntu@smithi046.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-08T22:40:41.037 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-08T22:40:41.038 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-08T22:40:41.038 INFO:teuthology.run_tasks:Running task console_log... 2024-03-08T22:40:41.160 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe912abe0d0>, signals=[15]) 2024-03-08T22:40:41.160 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-08T22:40:41.167 INFO:teuthology.task.internal:Opening connections... 2024-03-08T22:40:41.167 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T22:40:41.169 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:40:41.276 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-03-08T22:40:41.277 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:40:41.381 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-03-08T22:40:41.382 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:40:41.470 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-08T22:40:41.477 DEBUG:teuthology.orchestra.run.smithi046:> uname -m 2024-03-08T22:40:41.495 INFO:teuthology.orchestra.run.smithi046.stdout:x86_64 2024-03-08T22:40:41.496 DEBUG:teuthology.orchestra.run.smithi046:> cat /etc/os-release 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:NAME="CentOS Stream" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION="9" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:ID="centos" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:ID_LIKE="rhel fedora" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:VERSION_ID="9" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:PLATFORM_ID="platform:el9" 2024-03-08T22:40:41.554 INFO:teuthology.orchestra.run.smithi046.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:ANSI_COLOR="0;31" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:LOGO="fedora-logo-icon" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:HOME_URL="https://centos.org/" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T22:40:41.555 INFO:teuthology.orchestra.run.smithi046.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T22:40:41.555 INFO:teuthology.lock.ops:Updating smithi046.front.sepia.ceph.com on lock server 2024-03-08T22:40:41.575 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-03-08T22:40:41.595 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-03-08T22:40:41.595 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-03-08T22:40:41.653 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T22:40:41.654 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T22:40:41.654 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-03-08T22:40:41.675 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-03-08T22:40:41.692 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-03-08T22:40:41.693 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-03-08T22:40:41.750 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-03-08T22:40:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T22:40:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-03-08T22:40:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T22:40:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T22:40:41.751 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T22:40:41.751 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-03-08T22:40:41.776 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-08T22:40:41.786 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-08T22:40:41.830 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-08T22:40:41.830 DEBUG:teuthology.orchestra.run.smithi046:> test '!' -e /home/ubuntu/cephtest 2024-03-08T22:40:41.833 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-03-08T22:40:41.836 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-03-08T22:40:41.852 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-08T22:40:41.858 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-08T22:40:41.858 DEBUG:teuthology.orchestra.run.smithi046:> test -z $(ls -A /var/lib/ceph) 2024-03-08T22:40:41.890 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-03-08T22:40:41.892 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-03-08T22:40:42.011 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-08T22:40:42.084 INFO:teuthology.run_tasks:Running task kernel... 2024-03-08T22:40:42.098 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': 'dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412'} 2024-03-08T22:40:42.098 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-08T22:40:42.098 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-08T22:40:42.099 DEBUG:teuthology.orchestra.run.smithi046:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T22:40:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:40:42.116 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-08T22:40:42.356 DEBUG:teuthology.task.kernel:sha1 for client.0 is dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-08T22:40:42.356 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412"... 2024-03-08T22:40:42.357 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-08T22:40:42.374 INFO:teuthology.orchestra.run.smithi046.stdout:5.14.0-361.el9.x86_64 2024-03-08T22:40:42.375 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-08T22:40:42.375 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-08T22:40:42.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum upgrade -y linux-firmware 2024-03-08T22:40:44.736 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-08T22:40:46.269 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-08T22:40:47.126 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-08T22:41:08.026 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 975 kB/s | 19 MB 00:19 2024-03-08T22:41:13.034 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-08T22:41:14.911 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-03-08T22:41:16.703 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-08T22:41:21.820 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-08T22:41:24.218 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:================================================================================ 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 1 Package 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 375 M 2024-03-08T22:41:24.231 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-08T22:41:32.621 INFO:teuthology.orchestra.run.smithi046.stdout:linux-firmware-20240219-143.el9.noarch.rpm 45 MB/s | 375 MB 00:08 2024-03-08T22:41:32.622 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-08T22:41:32.622 INFO:teuthology.orchestra.run.smithi046.stdout:Total 45 MB/s | 375 MB 00:08 2024-03-08T22:41:32.622 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-08T22:41:32.707 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-08T22:41:32.707 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-08T22:41:35.204 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-08T22:41:35.204 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-08T22:41:41.621 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-08T22:41:41.868 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-08T22:41:41.955 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-08T22:41:43.288 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-08T22:41:43.288 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-08T22:41:43.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-08T22:41:43.732 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:41:43.732 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-03-08T22:41:43.732 INFO:teuthology.orchestra.run.smithi046.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-08T22:41:43.732 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:41:43.732 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-08T22:41:44.023 INFO:teuthology.task.kernel:Downloading kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-08T22:41:44.024 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 2024-03-08T22:41:44.171 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ 2024-03-08T22:41:44.171 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-08T22:41:44.174 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi046.front.sepia.ceph.com to complete... 2024-03-08T22:41:59.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08 22:41:59 URL:https://2.chacra.ceph.com/r/kernel/testing/dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_gdd5c1cd17167-1.x86_64.rpm [336216379/336216379] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-08T22:41:59.958 INFO:teuthology.orchestra.run.smithi046.stderr:FINISHED --2024-03-08 22:41:59-- 2024-03-08T22:41:59.958 INFO:teuthology.orchestra.run.smithi046.stderr:Total wall clock time: 16s 2024-03-08T22:41:59.959 INFO:teuthology.orchestra.run.smithi046.stderr:Downloaded: 1 files, 321M in 15s (21.1 MB/s) 2024-03-08T22:41:59.962 INFO:teuthology.task.kernel:Installing kernel dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 on client.0... 2024-03-08T22:41:59.962 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-08T22:42:01.987 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying... ######################################## 2024-03-08T22:42:02.062 INFO:teuthology.orchestra.run.smithi046.stdout:Preparing... ######################################## 2024-03-08T22:42:02.147 INFO:teuthology.orchestra.run.smithi046.stdout:Updating / installing... 2024-03-08T22:42:09.987 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-6.8.0_rc7_gdd5c1cd17167-1 ######################################## 2024-03-08T22:42:26.086 INFO:teuthology.orchestra.run.smithi046.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-08T22:42:49.508 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi046.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-08T22:42:49.509 DEBUG:teuthology.orchestra.run.smithi046:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-08T22:42:49.609 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:49.609 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/System.map 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/config 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-08T22:42:49.610 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-08T22:42:49.611 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-08T22:42:49.612 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/arch/x86/kvm/kvm.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ansi_cprng.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/anubis.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/arc4.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_memcpy.ko 2024-03-08T22:42:49.613 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_pq.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_tx.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/async_xor.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/async_tx/raid6test.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authenc.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/authencesn.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blake2b_generic.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_common.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/blowfish_generic.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/camellia_generic.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast5_generic.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast6_generic.ko 2024-03-08T22:42:49.614 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cast_common.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/ccm.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cmac.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crc32_generic.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_engine.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/crypto_user.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/cts.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/des_generic.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/dh_generic.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/echainiv.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/essiv.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/fcrypt.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/khazad.ko 2024-03-08T22:42:49.615 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/lz4hc.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/md4.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/michael_mic.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcbc.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/pcrypt.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/rmd160.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/seed.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/serpent_generic.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tcrypt.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/tea.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_common.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/twofish_generic.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/vmac.ko 2024-03-08T22:42:49.616 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/wp512.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xcbc.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xor.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/crypto/xxhash_generic.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/acpi_pad.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/custom_method.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/ec_sys.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/platform_profile.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbs.ko 2024-03-08T22:42:49.617 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/sbshc.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/acpi/video.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/acard-ahci.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ahci_platform.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/ata_generic.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/libahci_platform.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_acpi.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ali.ko 2024-03-08T22:42:49.618 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_amd.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_artop.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atiixp.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_atp867x.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd640.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cmd64x.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_cypress.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_efar.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt366.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt37x.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it8213.ko 2024-03-08T22:42:49.619 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_it821x.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_jmicron.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_marvell.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_mpiix.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_netcell.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ninja32.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87410.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_ns87415.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_oldpiix.ko 2024-03-08T22:42:49.620 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_opti.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_optidma.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pcmcia.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_piccolo.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_rdc.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sch.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_serverworks.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sil680.ko 2024-03-08T22:42:49.621 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sis.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_sl82c105.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_triflex.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pata_via.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/pdc_adma.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_inic162x.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_mv.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_nv.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_promise.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_qstor.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sil24.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sis.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_svw.ko 2024-03-08T22:42:49.622 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_sx4.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_uli.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_via.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ata/sata_vsc.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/auxdisplay/ks0108.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/bcma/bcma.ko 2024-03-08T22:42:49.623 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/aoe/aoe.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/brd.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/drbd/drbd.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/floppy.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/loop.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/nbd.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/null_blk/null_blk.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/pktcdvd.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/rbd.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/virtio_blk.ko 2024-03-08T22:42:49.624 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/xen-blkfront.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/block/zram/zram.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hangcheck-timer.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/via-rng.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-08T22:42:49.625 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/lp.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/mwave/mwave.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/ppdev.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tlclk.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-08T22:42:49.626 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/uv_mmtimer.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/char/virtio_console.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp 2024-03-08T22:42:49.627 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/ccp/ccp.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-aes.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/padlock-sha.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dca/dca.ko 2024-03-08T22:42:49.628 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/amd64_edac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/e752x_edac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/edac_mce_amd.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3000_edac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i3200_edac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5100_edac.ko 2024-03-08T22:42:49.629 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i5400_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7300_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i7core_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/i82975x_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/ie31200_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/sb_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/edac/x38_edac.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/edd.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-ich.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid 2024-03-08T22:42:49.630 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-appleir.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-aureal.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-axff.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-dr.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elecom.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-elo.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-emsff.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gaff.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gt683r.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-gyration.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-holtekff.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-hyperv.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-icade.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-keytouch.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-kye.ko 2024-03-08T22:42:49.631 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lcpower.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-led.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-lenovo.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-multitouch.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-ortek.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-penmount.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-petalynx.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-picolcd.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-pl.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-plantronics.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-primax.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-rmi.ko 2024-03-08T22:42:49.632 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-common.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-08T22:42:49.633 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-roccat.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-saitek.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-samsung.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sjoy.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sony.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-speedlink.ko 2024-03-08T22:42:49.634 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-steelseries.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-sunplus.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tivo.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-tmff.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-topseed.ko 2024-03-08T22:42:49.635 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-twinhan.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-uclogic.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-waltop.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-wiimote.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-xinmo.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zpff.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/hid-zydacron.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/uhid.ko 2024-03-08T22:42:49.636 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hid/wacom.ko 2024-03-08T22:42:49.637 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv 2024-03-08T22:42:49.637 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_balloon.ko 2024-03-08T22:42:49.637 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_utils.ko 2024-03-08T22:42:49.637 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hv/hv_vmbus.ko 2024-03-08T22:42:49.637 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru.ko 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/abituguru3.ko 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7414.ko 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ad7418.ko 2024-03-08T22:42:49.638 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adc128d818.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1025.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1026.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1029.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm1031.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adm9240.ko 2024-03-08T22:42:49.639 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ads7828.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7410.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7411.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7462.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7470.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7475.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/adt7x10.ko 2024-03-08T22:42:49.640 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/amc6821.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/applesmc.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asb100.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asc7621.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/atxp1.ko 2024-03-08T22:42:49.641 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/coretemp.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/dme1737.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds1621.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ds620.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc1403.ko 2024-03-08T22:42:49.642 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/emc6w201.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71805f.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f71882fg.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/f75375s.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fam15h_power.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/fschmd.ko 2024-03-08T22:42:49.643 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g760a.ko 2024-03-08T22:42:49.644 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/g762.ko 2024-03-08T22:42:49.644 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl518sm.ko 2024-03-08T22:42:49.644 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/gl520sm.ko 2024-03-08T22:42:49.644 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-08T22:42:49.644 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5500_temp.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/i5k_amb.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmaem.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ibmpex.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina209.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ina2xx.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/it87.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k10temp.ko 2024-03-08T22:42:49.645 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/k8temp.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lineage-pem.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm63.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm73.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm75.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm77.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm78.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm80.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm83.ko 2024-03-08T22:42:49.646 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm85.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm87.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm90.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm92.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm93.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95234.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95241.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/lm95245.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc2945.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4151.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4215.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4222.ko 2024-03-08T22:42:49.647 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4245.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4260.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/ltc4261.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max16065.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1619.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max1668.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max197.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6639.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6650.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/max6697.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/mcp3021.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6683.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775-core.ko 2024-03-08T22:42:49.648 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct6775.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/nct7802.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87360.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pc87427.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pcf8591.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-08T22:42:49.649 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/powr1220.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5627.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch5636.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht15.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sht21.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/shtc1.ko 2024-03-08T22:42:49.650 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/sis5595.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47b397.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m1.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/smsc47m192.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/thmc50.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp102.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp103.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp401.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/tmp421.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via-cputemp.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/via686a.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt1211.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/vt8231.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627ehf.ko 2024-03-08T22:42:49.651 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83627hf.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83781d.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83791d.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83792d.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83793.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83795.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l785ts.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/hwmon/w83l786ng.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-08T22:42:49.652 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-08T22:42:49.653 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-dev.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-mux.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-smbus.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/i2c/i2c-stub.ko 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core 2024-03-08T22:42:49.654 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_core.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-08T22:42:49.655 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input 2024-03-08T22:42:49.656 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/gameport/gameport.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/joydev.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/appletouch.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/bcm5974.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/cyapatp.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/sermouse.ko 2024-03-08T22:42:49.657 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/altera_ps2.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/arc_ps2.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/serio/serio_raw.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/input/sparse-keymap.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-blinkm.ko 2024-03-08T22:42:49.658 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lm3530.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-lp3944.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/leds-ss4200.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md 2024-03-08T22:42:49.659 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/bcache/bcache.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-bio-prison.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache-smq.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-cache.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-crypt.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-delay.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-flakey.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-log-userspace.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-multipath.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-queue-length.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-raid.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-round-robin.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-service-time.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-switch.ko 2024-03-08T22:42:49.660 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-thin-pool.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/dm-verity.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid0.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid1.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid10.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/md/raid456.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptbase.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptctl.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptfc.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptlan.ko 2024-03-08T22:42:49.661 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptsas.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptscsih.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/message/fusion/mptspi.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_ich.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/lpc_sch.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/mfd-core.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/sm501.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/viperboard.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/vx855.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/mfd/wl1273-core.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds9802als.ko 2024-03-08T22:42:49.662 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/apds990x.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/bh1770glc.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/cb710/cb710.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/at24.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/eeprom/max6875.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/enclosure.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/hpilo.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29003.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/isl29020.ko 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d 2024-03-08T22:42:49.663 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-me.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei-txe.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/mei/mei.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_7xx1.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tifm_core.ko 2024-03-08T22:42:49.664 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/tsl2550.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_balloon.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/bonding/bonding.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx 2024-03-08T22:42:49.665 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/dummy.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/eql.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-08T22:42:49.666 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-08T22:42:49.667 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-08T22:42:49.668 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-08T22:42:49.669 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip 2024-03-08T22:42:49.670 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/dnet.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-08T22:42:49.671 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ethoc.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/fealnx.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-08T22:42:49.672 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/jme.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-08T22:42:49.673 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-08T22:42:49.674 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-08T22:42:49.675 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-08T22:42:49.676 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-08T22:42:49.677 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-08T22:42:49.678 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/geneve.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ifb.ko 2024-03-08T22:42:49.679 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvlan.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/macvtap.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/mii.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/net_failover.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/netconsole.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/nlmon.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-08T22:42:49.680 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/amd.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/bcm87xx.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/broadcom.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/cicada.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/davicom.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/dp83640.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/et1011c.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/icplus.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/lxt.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/marvell.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/micrel.ko 2024-03-08T22:42:49.681 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/national.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/phylink.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/qsemi.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/realtek.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/smsc.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/ste10Xp.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/phy/vitesse.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_async.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppoe.ko 2024-03-08T22:42:49.682 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pppox.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/ppp/pptp.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slhc.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/slip/slip.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/sungem_phy.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tap.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_random.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/tun.ko 2024-03-08T22:42:49.683 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/veth.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/virtio_net.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/vxlan/vxlan.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/net/xen-netfront.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ntb/ntb.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme-core.ko 2024-03-08T22:42:49.684 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/nvme/host/nvme.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_cs.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_pc.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/parport/parport_serial.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pci/xen-pcifront.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/i82092.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/pd6729.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-08T22:42:49.685 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/acerhdf.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-08T22:42:49.686 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/hdaps.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/intel_ips.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-08T22:42:49.687 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/platform/x86/wmi.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/clients/pps_parport.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/pps/pps_core.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ptp/ptp_kvm.ko 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc 2024-03-08T22:42:49.688 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-em3027.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-08T22:42:49.689 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-max6900.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-08T22:42:49.690 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/rtc/rtc-x1205.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-9xxx.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-sas.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/3w-xxxx.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/BusLogic.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/a100u2w.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/advansys.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-08T22:42:49.691 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/am53c974.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/atp870u.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bfa/bfa.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ch.ko 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor 2024-03-08T22:42:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dc395x.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/dmx3191d.ko 2024-03-08T22:42:49.693 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/esp_scsi.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/fnic/fnic.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hpsa.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hptiop.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/hv_storvsc.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/initio.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ipr.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ips.ko 2024-03-08T22:42:49.694 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/isci/isci.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libfc/libfc.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/libsas/libsas.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid 2024-03-08T22:42:49.695 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/mvumi.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/pmcraid.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla1280.ko 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx 2024-03-08T22:42:49.696 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/raid_class.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_debug.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/ses.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/st.ko 2024-03-08T22:42:49.697 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/stex.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/virtio_scsi.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/scsi/wd719x.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ssb/ssb.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-08T22:42:49.698 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_file.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_iblock.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_mod.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_pscsi.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/target_core_user.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-08T22:42:49.699 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-08T22:42:49.700 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_gsm.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/n_hdlc.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/nozomi.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/arc_uart.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core 2024-03-08T22:42:49.701 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_aec.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_cif.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_pci_generic.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/uio/uio_sercos3.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-acm.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usblp.ko 2024-03-08T22:42:49.702 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/class/usbtmc.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/mdc800.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/image/microtek.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/adutux.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/appledisplay.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi26.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/emi62.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ezusb.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/idmouse.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/iowarrior.ko 2024-03-08T22:42:49.703 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/ldusb.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/legousbtower.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usb3503.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usblcd.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/uss720.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/misc/yurex.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/phy/phy-generic.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/aircable.ko 2024-03-08T22:42:49.704 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ark3116.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ch341.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cp210x.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cyberjack.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/empeg.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/io_ti.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipaq.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ipw.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ir-usb.ko 2024-03-08T22:42:49.705 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mct_u232.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7720.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/mos7840.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/navman.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/omninet.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/opticon.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/option.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/oti6858.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/pl2303.ko 2024-03-08T22:42:49.706 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcaux.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/qcserial.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/quatech2.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/safe_serial.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/sierra.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ssu100.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/symbolserial.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_debug.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/visor.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/whiteheat.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-08T22:42:49.707 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/uas.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-karma.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-08T22:42:49.708 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/storage/usb-storage.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_net.ko 2024-03-08T22:42:49.709 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/vhost/vhost_scsi.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/apple_bl.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/lcd.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_balloon.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_mmio.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/virtio/virtio_ring.ko 2024-03-08T22:42:49.710 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2482.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/masters/ds2490.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-08T22:42:49.711 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/w1/wire.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/hpwdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/i6300esb.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ib700wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ibmasr.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-08T22:42:49.712 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/it87_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/machzwd.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/nv_tco.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/softdog.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/via_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-08T22:42:49.713 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/wdt_pci.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/watchdog/xen_wdt.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-evtchn.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntalloc.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-gntdev.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xen-privcmd.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs 2024-03-08T22:42:49.714 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/9p/9p.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/affs/affs.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/befs/befs.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/binfmt_misc.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/btrfs/btrfs.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cachefiles/cachefiles.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ceph/ceph.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/coda/coda.ko 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs 2024-03-08T22:42:49.715 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/cramfs/cramfs.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/dlm/dlm.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/f2fs/f2fs.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/fat.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/msdos.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fat/vfat.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/cuse.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/fuse/fuse.ko 2024-03-08T22:42:49.716 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/gfs2/gfs2.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfs/hfs.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/hfsplus/hfsplus.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/isofs/isofs.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/jfs/jfs.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/lockd/lockd.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/minix/minix.ko 2024-03-08T22:42:49.717 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/netfs/netfs.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfs.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv3.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs/nfsv4.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/grace.ko 2024-03-08T22:42:49.718 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfs_common/nfs_acl.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nfsd/nfsd.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nilfs2/nilfs2.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-celtic.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-centeuro.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-croatian.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-cyrillic.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-gaelic.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-greek.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-iceland.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-inuit.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-roman.ko 2024-03-08T22:42:49.719 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-romanian.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/mac-turkish.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1250.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1251.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp1255.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp737.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp775.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp850.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp852.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp855.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp857.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp860.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp861.ko 2024-03-08T22:42:49.720 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp862.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp863.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp864.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp865.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp866.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp869.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp874.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp932.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp936.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp949.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_cp950.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_euc-jp.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-1.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-13.ko 2024-03-08T22:42:49.721 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-14.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-15.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-2.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-3.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-4.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-5.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-6.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-7.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_iso8859-9.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-r.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-ru.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_koi8-u.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/nls/nls_utf8.ko 2024-03-08T22:42:49.722 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/overlayfs/overlay.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/pstore/ramoops.ko 2024-03-08T22:42:49.723 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/reiserfs/reiserfs.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/romfs/romfs.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/client/cifs.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_arc4.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/smb/common/cifs_md4.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/squashfs/squashfs.ko 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv 2024-03-08T22:42:49.724 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/sysv/sysv.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/udf/udf.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/ufs/ufs.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/fs/xfs/xfs.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/rcu/rcutorture.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/torture.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/asn1_encoder.ko 2024-03-08T22:42:49.725 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc-itu-t.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crc8.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libarc4.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/crypto/libdes.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/libcrc32c.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lru_cache.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4_compress.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/lz4/lz4hc_compress.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/math/cordic.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/raid6/raid6_pq.ko 2024-03-08T22:42:49.726 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_bm.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_fsm.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/ts_kmp.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/lib/zstd/zstd_compress.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/mm/hwpoison-inject.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/6lowpan.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_dest.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_fragment.ko 2024-03-08T22:42:49.727 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_hop.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_mobility.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_routing.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/6lowpan/nhc_udp.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/garp.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/mrp.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/p8022.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/psnap.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/802/stp.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/8021q/8021q.ko 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p 2024-03-08T22:42:49.728 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_fd.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_rdma.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/9p/9pnet_virtio.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/batman-adv/batman-adv.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/br_netfilter.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/bridge.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-08T22:42:49.729 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-08T22:42:49.730 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/ebtables.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ceph/libceph.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/failover.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/core/pktgen.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_diag.ko 2024-03-08T22:42:49.731 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv4.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dccp/dccp_ipv6.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dns_resolver/dns_resolver.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/dsa_core.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_brcm.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_dsa.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_none.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/dsa/tag_trailer.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4 2024-03-08T22:42:49.732 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ah4.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/esp4.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/fou.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/gre.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/inet_diag.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_gre.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_tunnel.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ip_vti.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipcomp.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/ipip.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-08T22:42:49.733 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-08T22:42:49.734 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_bic.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_dctcp.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_diag.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_highspeed.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_htcp.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_hybla.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_illinois.ko 2024-03-08T22:42:49.735 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_lp.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_scalable.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_vegas.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_veno.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_westwood.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tcp_yeah.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/tunnel4.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_diag.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/udp_tunnel.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6 2024-03-08T22:42:49.736 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ah6.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/esp6.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/fou6.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_tunnel.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ip6_vti.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/ipcomp6.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-08T22:42:49.737 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-08T22:42:49.738 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/sit.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/tunnel6.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/key/af_key.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_core.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_eth.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip.ko 2024-03-08T22:42:49.739 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ip6.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_netlink.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/l2tp/l2tp_ppp.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/llc/llc.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mac802154/mac802154.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/mpls/mpls_gso.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-08T22:42:49.740 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-08T22:42:49.741 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conncount.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack.ko 2024-03-08T22:42:49.742 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table.ko 2024-03-08T22:42:49.743 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_log_syslog.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_irc.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_sip.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nf_tables.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_log.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-08T22:42:49.744 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_chain_nat.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_compat.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_ct.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fib.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_hash.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_limit.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_log.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_masq.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_nat.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_numgen.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_queue.ko 2024-03-08T22:42:49.745 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_quota.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_redir.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_reject_inet.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_socket.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/nft_tunnel.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_AUDIT.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_CT.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_DSCP.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HL.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_HMARK.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-08T22:42:49.746 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LED.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_LOG.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NETMAP.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFLOG.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_RATEEST.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_SECMARK.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TEE.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TPROXY.ko 2024-03-08T22:42:49.747 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_TRACE.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_addrtype.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_bpf.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cgroup.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cluster.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_comment.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connbytes.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlabel.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connlimit.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_connmark.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_conntrack.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_cpu.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dccp.ko 2024-03-08T22:42:49.748 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_devgroup.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_dscp.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ecn.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_esp.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hashlimit.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_helper.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_hl.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipcomp.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_iprange.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_ipvs.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_l2tp.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_length.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_limit.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mac.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_mark.ko 2024-03-08T22:42:49.749 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_multiport.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nat.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_nfacct.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_osf.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_owner.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_physdev.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_pkttype.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_policy.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_quota.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_rateest.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_realm.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_recent.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_sctp.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_set.ko 2024-03-08T22:42:49.750 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_socket.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_state.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_statistic.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_string.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_tcpmss.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_time.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netfilter/xt_u32.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/netlink/netlink_diag.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/nsh/nsh.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/openvswitch.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-geneve.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-gre.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/openvswitch/vport-vxlan.ko 2024-03-08T22:42:49.751 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/packet/af_packet_diag.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_rdma.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/rds/rds_tcp.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_csum.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_gact.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_mirred.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_nat.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_pedit.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_police.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_simple.ko 2024-03-08T22:42:49.752 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_skbedit.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/act_vlan.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_basic.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_bpf.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_flow.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_fw.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_route.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/cls_u32.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_cmp.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_ipset.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_meta.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_nbyte.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_text.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/em_u32.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_choke.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_codel.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_drr.ko 2024-03-08T22:42:49.753 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_fq.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_gred.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hfsc.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_hhf.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_htb.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_ingress.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_mqprio_lib.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_multiq.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_netem.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_pie.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_plug.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_prio.ko 2024-03-08T22:42:49.754 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_qfq.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_red.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfb.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_sfq.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_tbf.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sched/sch_teql.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sctp/sctp_diag.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/sunrpc.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc 2024-03-08T22:42:49.755 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/diag.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/tipc/tipc.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/unix/unix_diag.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security 2024-03-08T22:42:49.756 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/security/keys/trusted-keys/trusted.ko 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/kernel/virt/lib/irqbypass.ko 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.alias.bin 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.alias.bin 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.bin 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.builtin.modinfo 2024-03-08T22:42:49.757 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.dep.bin 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.devname 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.order 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.softdep 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/modules.symbols.bin 2024-03-08T22:42:49.758 INFO:teuthology.orchestra.run.smithi046.stdout:/lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-08T22:42:49.762 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:49.763 DEBUG:teuthology.orchestra.run.smithi046:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:postinstall scriptlet (using /bin/sh): 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: /usr/bin/kernel-install add 6.8.0-rc7-gdd5c1cd17167 /lib/modules/6.8.0-rc7-gdd5c1cd17167/vmlinuz 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:fi 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:for file in vmlinuz System.map config; do 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167"; then 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: cp "/lib/modules/6.8.0-rc7-gdd5c1cd17167/${file}" "/boot/${file}-6.8.0-rc7-gdd5c1cd17167" 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout: fi 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:done 2024-03-08T22:42:49.786 INFO:teuthology.orchestra.run.smithi046.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:new-kernel-pkg --remove 6.8.0-rc7-gdd5c1cd17167 --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-gdd5c1cd17167.img 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:kernel-install remove 6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:fi 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:49.787 INFO:teuthology.orchestra.run.smithi046.stdout:fi 2024-03-08T22:42:49.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo rpm -qi grub2-tools 2024-03-08T22:42:49.867 INFO:teuthology.orchestra.run.smithi046.stdout:Name : grub2-tools 2024-03-08T22:42:49.867 INFO:teuthology.orchestra.run.smithi046.stdout:Epoch : 1 2024-03-08T22:42:49.867 INFO:teuthology.orchestra.run.smithi046.stdout:Version : 2.06 2024-03-08T22:42:49.868 INFO:teuthology.orchestra.run.smithi046.stdout:Release : 68.el9 2024-03-08T22:42:49.868 INFO:teuthology.orchestra.run.smithi046.stdout:Architecture: x86_64 2024-03-08T22:42:49.868 INFO:teuthology.orchestra.run.smithi046.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-08T22:42:49.868 INFO:teuthology.orchestra.run.smithi046.stdout:Group : Unspecified 2024-03-08T22:42:49.868 INFO:teuthology.orchestra.run.smithi046.stdout:Size : 8264254 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:License : GPLv3+ 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-08T22:42:49.869 INFO:teuthology.orchestra.run.smithi046.stdout:Packager : builder@centos.org 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:Vendor : CentOS 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:Summary : Support tools for GRUB. 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:Description : 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:42:49.870 INFO:teuthology.orchestra.run.smithi046.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-08T22:42:49.871 INFO:teuthology.orchestra.run.smithi046.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-08T22:42:49.871 INFO:teuthology.orchestra.run.smithi046.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-08T22:42:49.871 INFO:teuthology.orchestra.run.smithi046.stdout:hardware devices. 2024-03-08T22:42:49.871 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:42:49.871 INFO:teuthology.orchestra.run.smithi046.stdout:This subpackage provides tools for support of all platforms. 2024-03-08T22:42:49.873 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-08T22:42:49.873 INFO:teuthology.task.kernel:Updating grub on smithi046 to boot 6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:49.873 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-08T22:42:50.627 INFO:teuthology.orchestra.run.smithi046.stderr:Generating grub configuration file ... 2024-03-08T22:42:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[36134]) leaked on vgs invocation. Parent PID 14917: grub2-probe 2024-03-08T22:42:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[27922]) leaked on vgs invocation. Parent PID 14917: grub2-probe 2024-03-08T22:42:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 3 (pipe:[36134]) leaked on vgs invocation. Parent PID 14917: grub2-probe 2024-03-08T22:42:52.196 INFO:teuthology.orchestra.run.smithi046.stderr:File descriptor 9 (pipe:[27922]) leaked on vgs invocation. Parent PID 14917: grub2-probe 2024-03-08T22:42:52.454 INFO:teuthology.orchestra.run.smithi046.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-08T22:42:52.469 INFO:teuthology.orchestra.run.smithi046.stderr:done 2024-03-08T22:42:52.471 DEBUG:teuthology.orchestra.run.smithi046:> mktemp 2024-03-08T22:42:52.486 INFO:teuthology.orchestra.run.smithi046.stdout:/tmp/tmp.dpUwrFnkgI 2024-03-08T22:42:52.486 DEBUG:teuthology.orchestra.run.smithi046:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dpUwrFnkgI 2024-03-08T22:42:52.582 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0666 /tmp/tmp.dpUwrFnkgI 2024-03-08T22:42:52.686 DEBUG:teuthology.orchestra.remote:smithi046:/tmp/tmp.dpUwrFnkgI is 6KB 2024-03-08T22:42:52.734 DEBUG:teuthology.orchestra.run.smithi046:> rm -fr /tmp/tmp.dpUwrFnkgI 2024-03-08T22:42:52.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo /bin/ls /boot/loader/entries || true 2024-03-08T22:42:52.822 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-08T22:42:52.823 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-08T22:42:52.823 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-08T22:42:52.823 INFO:teuthology.orchestra.run.smithi046.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167.conf 2024-03-08T22:42:52.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:42:52.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo shutdown -r now 2024-03-08T22:43:22.990 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-08T22:43:22.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:43:22.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:43:41.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-08T22:43:50.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:43:50.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:43:53.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-08T22:44:05.536 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:44:05.536 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:44:23.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.46 2024-03-08T22:44:38.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:44:38.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:44:39.515 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-08T22:44:40.225 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:44:40.225 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-gdd5c1cd17167"... 2024-03-08T22:44:40.225 DEBUG:teuthology.orchestra.run.smithi046:> uname -r 2024-03-08T22:44:40.285 INFO:teuthology.orchestra.run.smithi046.stdout:6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:44:40.286 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-gdd5c1cd17167 vs 6.8.0-rc7-gdd5c1cd17167 2024-03-08T22:44:40.286 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-08T22:44:40.286 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-08T22:44:41.288 DEBUG:teuthology.parallel:result is None 2024-03-08T22:44:41.288 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-08T22:44:41.296 INFO:teuthology.task.internal:Creating test directory... 2024-03-08T22:44:41.296 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T22:44:41.301 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T22:44:41.306 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T22:44:41.341 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-08T22:44:41.387 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-08T22:44:41.420 INFO:teuthology.task.internal:Creating archive directory... 2024-03-08T22:44:41.420 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T22:44:41.425 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T22:44:41.427 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T22:44:41.471 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-08T22:44:41.482 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-08T22:44:41.482 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T22:44:41.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:44:41.513 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T22:44:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:44:41.529 DEBUG:teuthology.orchestra.run.smithi046:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T22:44:41.534 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T22:44:41.555 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T22:44:41.616 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.636 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.640 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.657 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.716 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.769 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T22:44:41.773 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-08T22:44:41.781 INFO:teuthology.task.internal:Configuring sudo... 2024-03-08T22:44:41.782 DEBUG:teuthology.orchestra.run.smithi046:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T22:44:41.786 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T22:44:41.788 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T22:44:41.854 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-08T22:44:41.864 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-08T22:44:41.865 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T22:44:41.870 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T22:44:41.872 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T22:44:41.912 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:41.974 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:42.093 DEBUG:teuthology.orchestra.run.smithi046:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.169 DEBUG:teuthology.orchestra.run.smithi046:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.243 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:44:42.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T22:44:42.346 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:42.388 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:42.454 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.499 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.547 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:44:42.547 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T22:44:42.619 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:42.664 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T22:44:42.724 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T22:44:42.813 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:44:42.813 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T22:44:42.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo service rsyslog restart 2024-03-08T22:44:42.888 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-03-08T22:44:42.890 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-03-08T22:44:42.934 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T22:44:42.960 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T22:44:43.001 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T22:44:43.422 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-08T22:44:43.429 INFO:teuthology.task.internal:Starting timer... 2024-03-08T22:44:43.429 INFO:teuthology.run_tasks:Running task pcp... 2024-03-08T22:44:43.479 INFO:teuthology.run_tasks:Running task selinux... 2024-03-08T22:44:43.509 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-08T22:44:43.510 DEBUG:teuthology.orchestra.run.smithi046:> sudo service auditd rotate 2024-03-08T22:44:43.642 INFO:teuthology.orchestra.run.smithi046.stdout:Rotating logs: 2024-03-08T22:44:43.646 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-03-08T22:44:43.699 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-03-08T22:44:43.701 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-03-08T22:44:43.759 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-03-08T22:44:43.761 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-08T22:44:43.762 DEBUG:teuthology.orchestra.run.smithi046:> /usr/sbin/getenforce 2024-03-08T22:44:43.810 INFO:teuthology.orchestra.run.smithi046.stdout:Permissive 2024-03-08T22:44:43.810 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-03-08T22:44:43.844 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-03-08T22:44:43.844 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-03-08T22:44:43.869 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-03-08T22:44:43.869 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi046.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-03-08T22:44:43.869 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T22:44:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:44:43.955 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T22:44:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:44:43.987 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T22:44:44.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T22:44:44.016 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-08T22:44:44.016 DEBUG:teuthology.orchestra.run.smithi046:> sudo /usr/sbin/setenforce permissive 2024-03-08T22:44:44.118 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-03-08T22:44:44.163 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-03-08T22:44:44.196 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-08T22:44:44.205 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-08T22:44:44.211 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-08T22:44:44.335 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-08T22:44:44.350 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-08T22:44:44.351 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi046.front.sepia.ceph.com,smithi062.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-08T22:50:58.838 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi046.front.sepia.ceph.com'), Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-03-08T22:50:58.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:50:58.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi046.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:50:58.990 DEBUG:teuthology.orchestra.run.smithi046:> true 2024-03-08T22:50:59.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi046.front.sepia.ceph.com' 2024-03-08T22:50:59.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-08T22:50:59.150 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:50:59.256 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-03-08T22:50:59.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-03-08T22:50:59.331 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-08T22:50:59.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T22:50:59.420 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-03-08T22:50:59.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-03-08T22:50:59.490 INFO:teuthology.run_tasks:Running task clock... 2024-03-08T22:50:59.502 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-08T22:50:59.502 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T22:50:59.503 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T22:50:59.510 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T22:50:59.511 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T22:50:59.513 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T22:50:59.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T22:50:59.551 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T22:50:59.560 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T22:50:59.572 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T22:50:59.577 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T22:50:59.606 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-03-08T22:50:59.610 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-03-08T22:50:59.616 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T22:50:59.619 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-03-08T22:50:59.628 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-03-08T22:50:59.635 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T22:50:59.649 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T22:50:59.652 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T22:50:59.669 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T22:50:59.671 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T22:50:59.696 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.699 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.718 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-08T22:50:59.721 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T22:50:59.721 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-08T22:50:59.721 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.722 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.722 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.722 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:50:59.766 INFO:teuthology.orchestra.run.smithi046.stderr:sudo: ntpd: command not found 2024-03-08T22:50:59.812 INFO:teuthology.orchestra.run.smithi046.stdout:506 Cannot talk to daemon 2024-03-08T22:50:59.865 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T22:50:59.915 INFO:teuthology.orchestra.run.smithi046.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T22:51:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:51:00.050 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T22:51:00.052 INFO:teuthology.run_tasks:Running task install... 2024-03-08T22:51:00.060 DEBUG:teuthology.task.install:project ceph 2024-03-08T22:51:00.060 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7d63482aa89628d733e298984a762389b625c761'}} 2024-03-08T22:51:00.060 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '7d63482aa89628d733e298984a762389b625c761'} 2024-03-08T22:51:00.060 INFO:teuthology.task.install:Using flavor: default 2024-03-08T22:51:00.066 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-08T22:51:00.066 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-08T22:51:00.067 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T22:51:00.067 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:51:00.069 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T22:51:00.069 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:51:00.070 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T22:51:00.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:51:00.219 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T22:51:00.219 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T22:51:00.228 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T22:51:00.228 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T22:51:00.367 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T22:51:00.367 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:51:00.367 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T22:51:00.371 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T22:51:00.371 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:51:00.371 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T22:51:00.405 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T22:51:00.405 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T22:51:00.407 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T22:51:00.407 DEBUG:teuthology.orchestra.run.smithi062:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T22:51:00.470 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T22:51:00.470 DEBUG:teuthology.orchestra.run.smithi046:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T22:51:00.481 DEBUG:teuthology.orchestra.run.smithi062:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T22:51:00.571 DEBUG:teuthology.orchestra.run.smithi062:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T22:51:00.602 INFO:teuthology.orchestra.run.smithi062.stdout:check_obsoletes = 1 2024-03-08T22:51:00.603 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum clean all 2024-03-08T22:51:00.604 DEBUG:teuthology.orchestra.run.smithi046:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T22:51:00.665 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T22:51:00.665 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:51:00.665 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T22:51:00.697 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T22:51:00.697 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T22:51:00.734 DEBUG:teuthology.orchestra.run.smithi046:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T22:51:00.769 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T22:51:00.812 INFO:teuthology.orchestra.run.smithi046.stdout:check_obsoletes = 1 2024-03-08T22:51:00.815 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum clean all 2024-03-08T22:51:00.851 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T22:51:00.852 INFO:teuthology.orchestra.run.smithi062.stdout:74 files removed 2024-03-08T22:51:00.879 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T22:51:00.919 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-03-08T22:51:00.920 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-03-08T22:51:01.157 INFO:teuthology.orchestra.run.smithi192.stdout:74 files removed 2024-03-08T22:51:01.179 INFO:teuthology.orchestra.run.smithi046.stdout:74 files removed 2024-03-08T22:51:01.184 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T22:51:01.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T22:51:01.870 INFO:teuthology.orchestra.run.smithi062.stdout:ceph packages for x86_64 120 kB/s | 82 kB 00:00 2024-03-08T22:51:01.916 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 186 kB/s | 82 kB 00:00 2024-03-08T22:51:02.245 INFO:teuthology.orchestra.run.smithi062.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-08T22:51:02.300 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-08T22:51:02.347 INFO:teuthology.orchestra.run.smithi046.stdout:ceph packages for x86_64 135 kB/s | 82 kB 00:00 2024-03-08T22:51:02.604 INFO:teuthology.orchestra.run.smithi062.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-08T22:51:02.642 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-08T22:51:02.914 INFO:teuthology.orchestra.run.smithi046.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-08T22:51:03.215 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-08T22:51:03.304 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-08T22:51:03.457 INFO:teuthology.orchestra.run.smithi046.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-08T22:51:04.485 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - AppStream 7.0 MB/s | 6.7 MB 00:00 2024-03-08T22:51:04.807 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-08T22:51:04.888 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-08T22:51:06.086 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-08T22:51:06.386 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-08T22:51:06.773 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 4.9 MB/s | 8.0 MB 00:01 2024-03-08T22:51:07.889 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - BaseOS 7.4 MB/s | 8.0 MB 00:01 2024-03-08T22:51:10.865 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:02 2024-03-08T22:51:30.727 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 819 kB/s | 19 MB 00:23 2024-03-08T22:51:32.030 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - AppStream 842 kB/s | 19 MB 00:22 2024-03-08T22:51:32.123 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 355 kB/s | 6.1 MB 00:17 2024-03-08T22:51:33.582 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-03-08T22:51:36.764 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-03-08T22:51:38.144 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-08T22:51:38.474 INFO:teuthology.orchestra.run.smithi046.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 15 kB 00:00 2024-03-08T22:51:39.828 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 15 kB 00:00 2024-03-08T22:51:40.663 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-08T22:51:45.721 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-08T22:51:48.243 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:51:48.244 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:51:48.259 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T22:51:48.260 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T22:51:48.261 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T22:51:48.262 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T22:51:48.264 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T22:51:48.267 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T22:51:48.268 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T22:51:48.350 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-03-08T22:51:48.365 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-08T22:51:48.365 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T22:51:48.366 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T22:51:48.367 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T22:51:48.368 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T22:51:48.369 INFO:teuthology.orchestra.run.smithi192.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-03-08T22:51:48.370 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T22:51:48.371 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T22:51:48.372 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T22:51:48.373 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T22:51:48.374 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T22:51:48.375 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T22:51:48.376 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T22:51:48.377 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T22:51:48.378 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout:============================================================================================ 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout:Install 150 Packages 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 24 Packages 2024-03-08T22:51:48.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:51:48.380 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 223 M 2024-03-08T22:51:48.380 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-03-08T22:51:49.756 INFO:teuthology.orchestra.run.smithi192.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 13 kB/s | 6.6 kB 00:00 2024-03-08T22:51:50.215 INFO:teuthology.orchestra.run.smithi192.stdout:(2/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 1.9 MB/s | 889 kB 00:00 2024-03-08T22:51:50.315 INFO:teuthology.orchestra.run.smithi192.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 1.4 MB/s | 145 kB 00:00 2024-03-08T22:51:50.857 INFO:teuthology.orchestra.run.smithi192.stdout:(4/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 3.9 MB/s | 2.1 MB 00:00 2024-03-08T22:51:51.250 INFO:teuthology.orchestra.run.smithi192.stdout:(5/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 3.8 MB/s | 1.5 MB 00:00 2024-03-08T22:51:51.474 INFO:teuthology.orchestra.run.smithi192.stdout:(6/174): ceph-common-19.0.0-1965.g7d63482a.el9. 8.9 MB/s | 20 MB 00:02 2024-03-08T22:51:51.557 INFO:teuthology.orchestra.run.smithi192.stdout:(7/174): ceph-base-19.0.0-1965.g7d63482a.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-03-08T22:51:52.241 INFO:teuthology.orchestra.run.smithi192.stdout:(8/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 4.7 MB/s | 4.7 MB 00:00 2024-03-08T22:51:52.366 INFO:teuthology.orchestra.run.smithi192.stdout:(9/174): ceph-selinux-19.0.0-1965.g7d63482a.el9 201 kB/s | 25 kB 00:00 2024-03-08T22:51:52.925 INFO:teuthology.orchestra.run.smithi192.stdout:(10/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x8 12 MB/s | 17 MB 00:01 2024-03-08T22:51:53.042 INFO:teuthology.orchestra.run.smithi192.stdout:(11/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 1.8 MB/s | 220 kB 00:00 2024-03-08T22:51:53.142 INFO:teuthology.orchestra.run.smithi192.stdout:(12/174): cephfs-shell-19.0.0-1965.g7d63482a.el 271 kB/s | 27 kB 00:00 2024-03-08T22:51:53.242 INFO:teuthology.orchestra.run.smithi192.stdout:(13/174): libcephfs-devel-19.0.0-1965.g7d63482a 311 kB/s | 31 kB 00:00 2024-03-08T22:51:53.517 INFO:teuthology.orchestra.run.smithi192.stdout:(14/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 2.6 MB/s | 745 kB 00:00 2024-03-08T22:51:53.893 INFO:teuthology.orchestra.run.smithi192.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 435 kB/s | 163 kB 00:00 2024-03-08T22:51:53.985 INFO:teuthology.orchestra.run.smithi192.stdout:(16/174): librados-devel-19.0.0-1965.g7d63482a. 1.3 MB/s | 127 kB 00:00 2024-03-08T22:51:54.102 INFO:teuthology.orchestra.run.smithi192.stdout:(17/174): libradosstriper1-19.0.0-1965.g7d63482 4.2 MB/s | 501 kB 00:00 2024-03-08T22:51:54.844 INFO:teuthology.orchestra.run.smithi192.stdout:(18/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 6.9 MB/s | 5.1 MB 00:00 2024-03-08T22:51:54.952 INFO:teuthology.orchestra.run.smithi192.stdout:(19/174): ceph-radosgw-19.0.0-1965.g7d63482a.el 2.9 MB/s | 9.8 MB 00:03 2024-03-08T22:51:54.978 INFO:teuthology.orchestra.run.smithi192.stdout:(20/174): python3-ceph-argparse-19.0.0-1965.g7d 338 kB/s | 45 kB 00:00 2024-03-08T22:51:55.470 INFO:teuthology.orchestra.run.smithi192.stdout:(21/174): python3-ceph-common-19.0.0-1965.g7d63 254 kB/s | 131 kB 00:00 2024-03-08T22:51:55.653 INFO:teuthology.orchestra.run.smithi192.stdout:(22/174): python3-rados-19.0.0-1965.g7d63482a.e 1.7 MB/s | 320 kB 00:00 2024-03-08T22:51:55.695 INFO:teuthology.orchestra.run.smithi192.stdout:(23/174): python3-cephfs-19.0.0-1965.g7d63482a. 226 kB/s | 162 kB 00:00 2024-03-08T22:51:55.871 INFO:teuthology.orchestra.run.smithi192.stdout:(24/174): python3-rbd-19.0.0-1965.g7d63482a.el9 1.3 MB/s | 298 kB 00:00 2024-03-08T22:51:55.962 INFO:teuthology.orchestra.run.smithi192.stdout:(25/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 932 kB/s | 85 kB 00:00 2024-03-08T22:51:55.988 INFO:teuthology.orchestra.run.smithi192.stdout:(26/174): python3-rgw-19.0.0-1965.g7d63482a.el9 339 kB/s | 99 kB 00:00 2024-03-08T22:51:56.163 INFO:teuthology.orchestra.run.smithi192.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 975 kB/s | 171 kB 00:00 2024-03-08T22:51:56.247 INFO:teuthology.orchestra.run.smithi192.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1965.g 407 kB/s | 34 kB 00:00 2024-03-08T22:51:56.405 INFO:teuthology.orchestra.run.smithi192.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 875 kB/s | 139 kB 00:00 2024-03-08T22:51:57.156 INFO:teuthology.orchestra.run.smithi192.stdout:(30/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 2.6 MB/s | 3.1 MB 00:01 2024-03-08T22:51:57.256 INFO:teuthology.orchestra.run.smithi192.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 2.1 MB/s | 1.8 MB 00:00 2024-03-08T22:51:57.415 INFO:teuthology.orchestra.run.smithi192.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1965.g7d 1.5 MB/s | 248 kB 00:00 2024-03-08T22:51:57.506 INFO:teuthology.orchestra.run.smithi192.stdout:(33/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 538 kB/s | 49 kB 00:00 2024-03-08T22:51:57.590 INFO:teuthology.orchestra.run.smithi192.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1965.g7 198 kB/s | 16 kB 00:00 2024-03-08T22:51:57.740 INFO:teuthology.orchestra.run.smithi192.stdout:(35/174): ceph-volume-19.0.0-1965.g7d63482a.el9 1.7 MB/s | 264 kB 00:00 2024-03-08T22:51:58.033 INFO:teuthology.orchestra.run.smithi192.stdout:(36/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 2.5 MB/s | 748 kB 00:00 2024-03-08T22:51:58.124 INFO:teuthology.orchestra.run.smithi192.stdout:(37/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 199 kB/s | 18 kB 00:00 2024-03-08T22:51:58.475 INFO:teuthology.orchestra.run.smithi192.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-03-08T22:51:58.591 INFO:teuthology.orchestra.run.smithi192.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 7.7 MB/s | 920 kB 00:00 2024-03-08T22:51:58.658 INFO:teuthology.orchestra.run.smithi192.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-08T22:51:58.725 INFO:teuthology.orchestra.run.smithi192.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-08T22:51:58.784 INFO:teuthology.orchestra.run.smithi192.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 543 kB/s | 32 kB 00:00 2024-03-08T22:51:58.977 INFO:teuthology.orchestra.run.smithi192.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-08T22:51:59.044 INFO:teuthology.orchestra.run.smithi192.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 257 kB/s | 17 kB 00:00 2024-03-08T22:51:59.110 INFO:teuthology.orchestra.run.smithi192.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 150 kB/s | 10 kB 00:00 2024-03-08T22:51:59.177 INFO:teuthology.orchestra.run.smithi192.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.3 MB/s | 159 kB 00:00 2024-03-08T22:51:59.244 INFO:teuthology.orchestra.run.smithi192.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 663 kB/s | 44 kB 00:00 2024-03-08T22:51:59.329 INFO:teuthology.orchestra.run.smithi192.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-08T22:51:59.403 INFO:teuthology.orchestra.run.smithi192.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-03-08T22:51:59.462 INFO:teuthology.orchestra.run.smithi192.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 797 kB/s | 46 kB 00:00 2024-03-08T22:51:59.521 INFO:teuthology.orchestra.run.smithi192.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-08T22:51:59.562 INFO:teuthology.orchestra.run.smithi192.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 4.4 MB/s | 187 kB 00:00 2024-03-08T22:51:59.605 INFO:teuthology.orchestra.run.smithi192.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-08T22:51:59.647 INFO:teuthology.orchestra.run.smithi192.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-08T22:51:59.830 INFO:teuthology.orchestra.run.smithi192.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-08T22:51:59.905 INFO:teuthology.orchestra.run.smithi192.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2024-03-08T22:51:59.956 INFO:teuthology.orchestra.run.smithi192.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 16 MB/s | 811 kB 00:00 2024-03-08T22:51:59.998 INFO:teuthology.orchestra.run.smithi192.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.1 MB/s | 134 kB 00:00 2024-03-08T22:52:00.165 INFO:teuthology.orchestra.run.smithi192.stdout:(59/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-08T22:52:00.242 INFO:teuthology.orchestra.run.smithi192.stdout:(60/174): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-03-08T22:52:00.282 INFO:teuthology.orchestra.run.smithi192.stdout:(61/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.8 MB/s | 78 kB 00:00 2024-03-08T22:52:00.457 INFO:teuthology.orchestra.run.smithi192.stdout:(62/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-08T22:52:00.491 INFO:teuthology.orchestra.run.smithi192.stdout:(63/174): python3-jsonpatch-1.21-16.el9.noarch. 736 kB/s | 24 kB 00:00 2024-03-08T22:52:00.525 INFO:teuthology.orchestra.run.smithi192.stdout:(64/174): python3-jsonpointer-2.0-4.el9.noarch. 541 kB/s | 18 kB 00:00 2024-03-08T22:52:00.567 INFO:teuthology.orchestra.run.smithi192.stdout:(65/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-08T22:52:00.598 INFO:teuthology.orchestra.run.smithi192.stdout:(66/174): python3-markupsafe-1.1.1-12.el9.x86_6 1.0 MB/s | 32 kB 00:00 2024-03-08T22:52:00.732 INFO:teuthology.orchestra.run.smithi192.stdout:(67/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-08T22:52:00.774 INFO:teuthology.orchestra.run.smithi192.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.1 MB/s | 217 kB 00:00 2024-03-08T22:52:00.816 INFO:teuthology.orchestra.run.smithi192.stdout:(69/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-08T22:52:00.908 INFO:teuthology.orchestra.run.smithi192.stdout:(70/174): ceph-mgr-diskprediction-local-19.0.0- 2.0 MB/s | 7.4 MB 00:03 2024-03-08T22:52:00.933 INFO:teuthology.orchestra.run.smithi192.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 585 kB/s | 69 kB 00:00 2024-03-08T22:52:00.950 INFO:teuthology.orchestra.run.smithi192.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 986 kB/s | 41 kB 00:00 2024-03-08T22:52:00.984 INFO:teuthology.orchestra.run.smithi192.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-08T22:52:01.009 INFO:teuthology.orchestra.run.smithi192.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2024-03-08T22:52:01.035 INFO:teuthology.orchestra.run.smithi192.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-08T22:52:01.060 INFO:teuthology.orchestra.run.smithi192.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 948 kB/s | 47 kB 00:00 2024-03-08T22:52:01.085 INFO:teuthology.orchestra.run.smithi192.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 855 kB/s | 43 kB 00:00 2024-03-08T22:52:01.194 INFO:teuthology.orchestra.run.smithi192.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 361 kB/s | 39 kB 00:00 2024-03-08T22:52:01.315 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 762 kB/s | 21 MB 00:27 2024-03-08T22:52:01.403 INFO:teuthology.orchestra.run.smithi192.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-08T22:52:01.444 INFO:teuthology.orchestra.run.smithi192.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 165 kB/s | 41 kB 00:00 2024-03-08T22:52:01.465 INFO:teuthology.orchestra.run.smithi046.stdout:Extra Packages for Enterprise Linux 923 kB/s | 21 MB 00:22 2024-03-08T22:52:01.470 INFO:teuthology.orchestra.run.smithi192.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-08T22:52:01.495 INFO:teuthology.orchestra.run.smithi192.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-08T22:52:01.521 INFO:teuthology.orchestra.run.smithi192.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-08T22:52:01.537 INFO:teuthology.orchestra.run.smithi192.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-08T22:52:01.563 INFO:teuthology.orchestra.run.smithi192.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-08T22:52:01.588 INFO:teuthology.orchestra.run.smithi192.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-08T22:52:01.630 INFO:teuthology.orchestra.run.smithi192.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-08T22:52:01.656 INFO:teuthology.orchestra.run.smithi192.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-08T22:52:01.682 INFO:teuthology.orchestra.run.smithi192.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-08T22:52:01.706 INFO:teuthology.orchestra.run.smithi192.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-08T22:52:01.848 INFO:teuthology.orchestra.run.smithi192.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-08T22:52:01.899 INFO:teuthology.orchestra.run.smithi192.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-08T22:52:01.966 INFO:teuthology.orchestra.run.smithi192.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-03-08T22:52:01.991 INFO:teuthology.orchestra.run.smithi192.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.3 MB/s | 125 kB 00:00 2024-03-08T22:52:02.016 INFO:teuthology.orchestra.run.smithi192.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 437 kB/s | 21 kB 00:00 2024-03-08T22:52:02.175 INFO:teuthology.orchestra.run.smithi192.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 138 kB/s | 25 kB 00:00 2024-03-08T22:52:02.783 INFO:teuthology.orchestra.run.smithi192.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 403 kB/s | 245 kB 00:00 2024-03-08T22:52:03.284 INFO:teuthology.orchestra.run.smithi192.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 496 kB/s | 248 kB 00:00 2024-03-08T22:52:03.375 INFO:teuthology.orchestra.run.smithi192.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-08T22:52:03.426 INFO:teuthology.orchestra.run.smithi192.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 697 kB/s | 35 kB 00:00 2024-03-08T22:52:03.851 INFO:teuthology.orchestra.run.smithi192.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 105 kB/s | 44 kB 00:00 2024-03-08T22:52:04.293 INFO:teuthology.orchestra.run.smithi192.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 212 kB/s | 94 kB 00:00 2024-03-08T22:52:04.393 INFO:teuthology.orchestra.run.smithi192.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-08T22:52:04.535 INFO:teuthology.orchestra.run.smithi192.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-03-08T22:52:04.719 INFO:teuthology.orchestra.run.smithi192.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 1.5 MB/s | 4.1 MB 00:02 2024-03-08T22:52:05.061 INFO:teuthology.orchestra.run.smithi192.stdout:(106/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-08T22:52:05.086 INFO:teuthology.orchestra.run.smithi192.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 845 kB/s | 309 kB 00:00 2024-03-08T22:52:05.120 INFO:teuthology.orchestra.run.smithi192.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 333 kB/s | 19 kB 00:00 2024-03-08T22:52:05.278 INFO:teuthology.orchestra.run.smithi192.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-08T22:52:05.345 INFO:teuthology.orchestra.run.smithi192.stdout:(110/174): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-08T22:52:05.370 INFO:teuthology.orchestra.run.smithi192.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 533 kB/s | 49 kB 00:00 2024-03-08T22:52:05.421 INFO:teuthology.orchestra.run.smithi192.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 893 kB/s | 67 kB 00:00 2024-03-08T22:52:05.447 INFO:teuthology.orchestra.run.smithi192.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.9 MB/s | 151 kB 00:00 2024-03-08T22:52:05.505 INFO:teuthology.orchestra.run.smithi192.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 838 kB 00:00 2024-03-08T22:52:05.539 INFO:teuthology.orchestra.run.smithi192.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 5.8 MB/s | 548 kB 00:00 2024-03-08T22:52:05.581 INFO:teuthology.orchestra.run.smithi192.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 585 kB/s | 43 kB 00:00 2024-03-08T22:52:05.618 INFO:teuthology.orchestra.run.smithi192.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 407 kB/s | 32 kB 00:00 2024-03-08T22:52:05.652 INFO:teuthology.orchestra.run.smithi192.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 211 kB/s | 15 kB 00:00 2024-03-08T22:52:05.694 INFO:teuthology.orchestra.run.smithi192.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-08T22:52:05.727 INFO:teuthology.orchestra.run.smithi192.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 4.7 MB/s | 358 kB 00:00 2024-03-08T22:52:05.769 INFO:teuthology.orchestra.run.smithi192.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-03-08T22:52:05.795 INFO:teuthology.orchestra.run.smithi192.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-03-08T22:52:05.853 INFO:teuthology.orchestra.run.smithi192.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-03-08T22:52:05.928 INFO:teuthology.orchestra.run.smithi192.stdout:(124/174): python3-grpcio-1.46.7-10.el9.x86_64. 15 MB/s | 2.0 MB 00:00 2024-03-08T22:52:05.945 INFO:teuthology.orchestra.run.smithi192.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-03-08T22:52:05.996 INFO:teuthology.orchestra.run.smithi192.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 161 kB/s | 11 kB 00:00 2024-03-08T22:52:06.104 INFO:teuthology.orchestra.run.smithi192.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 111 kB/s | 18 kB 00:00 2024-03-08T22:52:06.205 INFO:teuthology.orchestra.run.smithi192.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 111 kB/s | 23 kB 00:00 2024-03-08T22:52:06.230 INFO:teuthology.orchestra.run.smithi192.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 155 kB/s | 19 kB 00:00 2024-03-08T22:52:06.263 INFO:teuthology.orchestra.run.smithi192.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 336 kB/s | 20 kB 00:00 2024-03-08T22:52:06.322 INFO:teuthology.orchestra.run.smithi192.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-03-08T22:52:06.347 INFO:teuthology.orchestra.run.smithi192.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-03-08T22:52:06.373 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-03-08T22:52:06.414 INFO:teuthology.orchestra.run.smithi192.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-03-08T22:52:06.439 INFO:teuthology.orchestra.run.smithi192.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-03-08T22:52:06.507 INFO:teuthology.orchestra.run.smithi192.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-03-08T22:52:06.532 INFO:teuthology.orchestra.run.smithi192.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-03-08T22:52:06.599 INFO:teuthology.orchestra.run.smithi192.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 978 kB/s | 90 kB 00:00 2024-03-08T22:52:06.624 INFO:teuthology.orchestra.run.smithi192.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-03-08T22:52:06.691 INFO:teuthology.orchestra.run.smithi192.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-03-08T22:52:06.716 INFO:teuthology.orchestra.run.smithi192.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-03-08T22:52:06.792 INFO:teuthology.orchestra.run.smithi192.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-03-08T22:52:06.817 INFO:teuthology.orchestra.run.smithi192.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-08T22:52:06.884 INFO:teuthology.orchestra.run.smithi192.stdout:(143/174): python3-typing-extensions-4.4.0-2.el 557 kB/s | 51 kB 00:00 2024-03-08T22:52:06.917 INFO:teuthology.orchestra.run.smithi192.stdout:(144/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-08T22:52:06.985 INFO:teuthology.orchestra.run.smithi192.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 894 kB/s | 90 kB 00:00 2024-03-08T22:52:07.026 INFO:teuthology.orchestra.run.smithi192.stdout:(146/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.8 MB/s | 427 kB 00:00 2024-03-08T22:52:07.102 INFO:teuthology.orchestra.run.smithi192.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 171 kB/s | 20 kB 00:00 2024-03-08T22:52:07.135 INFO:teuthology.orchestra.run.smithi192.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-08T22:52:07.252 INFO:teuthology.orchestra.run.smithi192.stdout:(149/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-08T22:52:07.473 INFO:teuthology.orchestra.run.smithi046.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-08T22:52:08.918 INFO:teuthology.orchestra.run.smithi062.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:52:08.919 INFO:teuthology.orchestra.run.smithi062.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:52:08.931 INFO:teuthology.orchestra.run.smithi062.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T22:52:08.932 INFO:teuthology.orchestra.run.smithi062.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T22:52:08.933 INFO:teuthology.orchestra.run.smithi062.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T22:52:08.934 INFO:teuthology.orchestra.run.smithi062.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T22:52:08.936 INFO:teuthology.orchestra.run.smithi062.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T22:52:08.938 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T22:52:08.939 INFO:teuthology.orchestra.run.smithi062.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T22:52:09.026 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-03-08T22:52:09.061 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-08T22:52:09.061 INFO:teuthology.orchestra.run.smithi062.stdout: Package Arch Version Repository Size 2024-03-08T22:52:09.061 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T22:52:09.062 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T22:52:09.063 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrading: 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T22:52:09.064 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:52:09.065 INFO:teuthology.orchestra.run.smithi062.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T22:52:09.066 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T22:52:09.067 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T22:52:09.068 INFO:teuthology.orchestra.run.smithi062.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T22:52:09.069 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T22:52:09.070 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T22:52:09.071 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T22:52:09.072 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T22:52:09.073 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout:Installing weak dependencies: 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout:============================================================================================ 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout:Install 150 Packages 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout:Upgrade 24 Packages 2024-03-08T22:52:09.074 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:52:09.075 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 223 M 2024-03-08T22:52:09.075 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-03-08T22:52:09.094 INFO:teuthology.orchestra.run.smithi192.stdout:(150/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 1.7 MB/s | 3.1 MB 00:01 2024-03-08T22:52:09.144 INFO:teuthology.orchestra.run.smithi192.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-08T22:52:09.703 INFO:teuthology.orchestra.run.smithi192.stdout:(152/174): librados2-19.0.0-1965.g7d63482a.el9. 1.3 MB/s | 3.4 MB 00:02 2024-03-08T22:52:09.816 INFO:teuthology.orchestra.run.smithi062.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-08T22:52:10.200 INFO:teuthology.orchestra.run.smithi062.stdout:(2/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 2.3 MB/s | 889 kB 00:00 2024-03-08T22:52:10.283 INFO:teuthology.orchestra.run.smithi062.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 1.7 MB/s | 145 kB 00:00 2024-03-08T22:52:10.300 INFO:teuthology.orchestra.run.smithi046.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:52:10.302 INFO:teuthology.orchestra.run.smithi046.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T22:52:10.326 INFO:teuthology.orchestra.run.smithi046.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T22:52:10.327 INFO:teuthology.orchestra.run.smithi046.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T22:52:10.329 INFO:teuthology.orchestra.run.smithi046.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T22:52:10.330 INFO:teuthology.orchestra.run.smithi046.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T22:52:10.333 INFO:teuthology.orchestra.run.smithi046.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T22:52:10.337 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T22:52:10.338 INFO:teuthology.orchestra.run.smithi046.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T22:52:10.433 INFO:teuthology.orchestra.run.smithi046.stdout:Dependencies resolved. 2024-03-08T22:52:10.463 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-08T22:52:10.463 INFO:teuthology.orchestra.run.smithi046.stdout: Package Arch Version Repository Size 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout:Installing: 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T22:52:10.464 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T22:52:10.465 INFO:teuthology.orchestra.run.smithi046.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T22:52:10.466 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrading: 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T22:52:10.467 INFO:teuthology.orchestra.run.smithi046.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout:Installing dependencies: 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T22:52:10.468 INFO:teuthology.orchestra.run.smithi046.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T22:52:10.469 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T22:52:10.470 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T22:52:10.471 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T22:52:10.472 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T22:52:10.473 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T22:52:10.474 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T22:52:10.475 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T22:52:10.476 INFO:teuthology.orchestra.run.smithi046.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout:Installing weak dependencies: 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction Summary 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout:============================================================================================ 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout:Install 150 Packages 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout:Upgrade 24 Packages 2024-03-08T22:52:10.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:52:10.478 INFO:teuthology.orchestra.run.smithi046.stdout:Total download size: 223 M 2024-03-08T22:52:10.478 INFO:teuthology.orchestra.run.smithi046.stdout:Downloading Packages: 2024-03-08T22:52:10.709 INFO:teuthology.orchestra.run.smithi062.stdout:(4/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 5.0 MB/s | 2.1 MB 00:00 2024-03-08T22:52:10.826 INFO:teuthology.orchestra.run.smithi062.stdout:(5/174): ceph-base-19.0.0-1965.g7d63482a.el9.x8 4.1 MB/s | 5.4 MB 00:01 2024-03-08T22:52:10.985 INFO:teuthology.orchestra.run.smithi062.stdout:(6/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 5.4 MB/s | 1.5 MB 00:00 2024-03-08T22:52:11.435 INFO:teuthology.orchestra.run.smithi062.stdout:(7/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 7.7 MB/s | 4.7 MB 00:00 2024-03-08T22:52:11.892 INFO:teuthology.orchestra.run.smithi046.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 12 kB/s | 6.6 kB 00:00 2024-03-08T22:52:12.693 INFO:teuthology.orchestra.run.smithi062.stdout:(8/174): ceph-radosgw-19.0.0-1965.g7d63482a.el9 7.8 MB/s | 9.8 MB 00:01 2024-03-08T22:52:12.793 INFO:teuthology.orchestra.run.smithi062.stdout:(9/174): ceph-selinux-19.0.0-1965.g7d63482a.el9 253 kB/s | 25 kB 00:00 2024-03-08T22:52:13.059 INFO:teuthology.orchestra.run.smithi046.stdout:(2/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 763 kB/s | 889 kB 00:01 2024-03-08T22:52:13.167 INFO:teuthology.orchestra.run.smithi046.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 1.3 MB/s | 145 kB 00:00 2024-03-08T22:52:13.202 INFO:teuthology.orchestra.run.smithi062.stdout:(10/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x8 7.6 MB/s | 17 MB 00:02 2024-03-08T22:52:13.427 INFO:teuthology.orchestra.run.smithi062.stdout:(11/174): ceph-common-19.0.0-1965.g7d63482a.el9 5.1 MB/s | 20 MB 00:03 2024-03-08T22:52:13.436 INFO:teuthology.orchestra.run.smithi192.stdout:(153/174): ceph-test-19.0.0-1965.g7d63482a.el9. 2.3 MB/s | 47 MB 00:21 2024-03-08T22:52:13.452 INFO:teuthology.orchestra.run.smithi062.stdout:(12/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 879 kB/s | 220 kB 00:00 2024-03-08T22:52:13.494 INFO:teuthology.orchestra.run.smithi062.stdout:(13/174): cephfs-shell-19.0.0-1965.g7d63482a.el 405 kB/s | 27 kB 00:00 2024-03-08T22:52:13.520 INFO:teuthology.orchestra.run.smithi062.stdout:(14/174): libcephfs-devel-19.0.0-1965.g7d63482a 463 kB/s | 31 kB 00:00 2024-03-08T22:52:13.612 INFO:teuthology.orchestra.run.smithi062.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 1.7 MB/s | 163 kB 00:00 2024-03-08T22:52:13.645 INFO:teuthology.orchestra.run.smithi062.stdout:(16/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 4.8 MB/s | 745 kB 00:00 2024-03-08T22:52:13.687 INFO:teuthology.orchestra.run.smithi062.stdout:(17/174): librados-devel-19.0.0-1965.g7d63482a. 1.6 MB/s | 127 kB 00:00 2024-03-08T22:52:13.721 INFO:teuthology.orchestra.run.smithi062.stdout:(18/174): libradosstriper1-19.0.0-1965.g7d63482 6.5 MB/s | 501 kB 00:00 2024-03-08T22:52:13.805 INFO:teuthology.orchestra.run.smithi062.stdout:(19/174): python3-ceph-argparse-19.0.0-1965.g7d 542 kB/s | 45 kB 00:00 2024-03-08T22:52:13.897 INFO:teuthology.orchestra.run.smithi062.stdout:(20/174): python3-ceph-common-19.0.0-1965.g7d63 1.4 MB/s | 131 kB 00:00 2024-03-08T22:52:13.920 INFO:teuthology.orchestra.run.smithi192.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 88 kB/s | 42 kB 00:00 2024-03-08T22:52:13.959 INFO:teuthology.orchestra.run.smithi046.stdout:(4/174): ceph-base-19.0.0-1965.g7d63482a.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-03-08T22:52:13.989 INFO:teuthology.orchestra.run.smithi062.stdout:(21/174): python3-cephfs-19.0.0-1965.g7d63482a. 1.7 MB/s | 162 kB 00:00 2024-03-08T22:52:14.097 INFO:teuthology.orchestra.run.smithi062.stdout:(22/174): python3-rados-19.0.0-1965.g7d63482a.e 2.9 MB/s | 320 kB 00:00 2024-03-08T22:52:14.222 INFO:teuthology.orchestra.run.smithi062.stdout:(23/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 9.6 MB/s | 5.1 MB 00:00 2024-03-08T22:52:14.248 INFO:teuthology.orchestra.run.smithi062.stdout:(24/174): python3-rbd-19.0.0-1965.g7d63482a.el9 1.9 MB/s | 298 kB 00:00 2024-03-08T22:52:14.295 INFO:teuthology.orchestra.run.smithi192.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 662 kB/s | 248 kB 00:00 2024-03-08T22:52:14.298 INFO:teuthology.orchestra.run.smithi062.stdout:(25/174): python3-rgw-19.0.0-1965.g7d63482a.el9 1.3 MB/s | 99 kB 00:00 2024-03-08T22:52:14.315 INFO:teuthology.orchestra.run.smithi062.stdout:(26/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 1.2 MB/s | 85 kB 00:00 2024-03-08T22:52:14.342 INFO:teuthology.orchestra.run.smithi046.stdout:(5/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 1.8 MB/s | 2.1 MB 00:01 2024-03-08T22:52:14.423 INFO:teuthology.orchestra.run.smithi062.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 1.5 MB/s | 171 kB 00:00 2024-03-08T22:52:14.501 INFO:teuthology.orchestra.run.smithi046.stdout:(6/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-03-08T22:52:14.540 INFO:teuthology.orchestra.run.smithi062.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1965.g 291 kB/s | 34 kB 00:00 2024-03-08T22:52:14.616 INFO:teuthology.orchestra.run.smithi062.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 1.8 MB/s | 139 kB 00:00 2024-03-08T22:52:14.645 INFO:teuthology.orchestra.run.smithi192.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 204 kB/s | 71 kB 00:00 2024-03-08T22:52:14.666 INFO:teuthology.orchestra.run.smithi062.stdout:(30/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 8.4 MB/s | 3.1 MB 00:00 2024-03-08T22:52:14.795 INFO:teuthology.orchestra.run.smithi192.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 243 kB/s | 36 kB 00:00 2024-03-08T22:52:14.833 INFO:teuthology.orchestra.run.smithi062.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 8.3 MB/s | 1.8 MB 00:00 2024-03-08T22:52:14.925 INFO:teuthology.orchestra.run.smithi062.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1965.g7d 2.6 MB/s | 248 kB 00:00 2024-03-08T22:52:15.009 INFO:teuthology.orchestra.run.smithi062.stdout:(33/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 592 kB/s | 49 kB 00:00 2024-03-08T22:52:15.084 INFO:teuthology.orchestra.run.smithi062.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1965.g7 219 kB/s | 16 kB 00:00 2024-03-08T22:52:15.176 INFO:teuthology.orchestra.run.smithi062.stdout:(35/174): ceph-volume-19.0.0-1965.g7d63482a.el9 2.8 MB/s | 264 kB 00:00 2024-03-08T22:52:15.304 INFO:teuthology.orchestra.run.smithi192.stdout:(158/174): NetworkManager-libnm-1.46.0-1.el9.x8 332 kB/s | 1.8 MB 00:05 2024-03-08T22:52:15.334 INFO:teuthology.orchestra.run.smithi062.stdout:(36/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-08T22:52:15.418 INFO:teuthology.orchestra.run.smithi062.stdout:(37/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 219 kB/s | 18 kB 00:00 2024-03-08T22:52:15.568 INFO:teuthology.orchestra.run.smithi062.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 8.2 MB/s | 7.4 MB 00:00 2024-03-08T22:52:15.605 INFO:teuthology.orchestra.run.smithi192.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 238 kB/s | 192 kB 00:00 2024-03-08T22:52:15.744 INFO:teuthology.orchestra.run.smithi062.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 5.2 MB/s | 920 kB 00:00 2024-03-08T22:52:15.763 INFO:teuthology.orchestra.run.smithi192.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 145 kB/s | 23 kB 00:00 2024-03-08T22:52:15.777 INFO:teuthology.orchestra.run.smithi062.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 662 kB 00:00 2024-03-08T22:52:15.803 INFO:teuthology.orchestra.run.smithi062.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-08T22:52:15.828 INFO:teuthology.orchestra.run.smithi062.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-08T22:52:15.853 INFO:teuthology.orchestra.run.smithi062.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 631 kB/s | 32 kB 00:00 2024-03-08T22:52:15.904 INFO:teuthology.orchestra.run.smithi062.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 348 kB/s | 17 kB 00:00 2024-03-08T22:52:15.979 INFO:teuthology.orchestra.run.smithi062.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-08T22:52:16.005 INFO:teuthology.orchestra.run.smithi062.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 102 kB/s | 10 kB 00:00 2024-03-08T22:52:16.030 INFO:teuthology.orchestra.run.smithi062.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-08T22:52:16.055 INFO:teuthology.orchestra.run.smithi062.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-08T22:52:16.089 INFO:teuthology.orchestra.run.smithi062.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-08T22:52:16.114 INFO:teuthology.orchestra.run.smithi062.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-08T22:52:16.139 INFO:teuthology.orchestra.run.smithi062.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 941 kB/s | 46 kB 00:00 2024-03-08T22:52:16.165 INFO:teuthology.orchestra.run.smithi062.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2024-03-08T22:52:16.190 INFO:teuthology.orchestra.run.smithi062.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-08T22:52:16.204 INFO:teuthology.orchestra.run.smithi192.stdout:(161/174): libblkid-2.37.4-18.el9.x86_64.rpm 245 kB/s | 108 kB 00:00 2024-03-08T22:52:16.215 INFO:teuthology.orchestra.run.smithi062.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-08T22:52:16.240 INFO:teuthology.orchestra.run.smithi062.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-08T22:52:16.372 INFO:teuthology.orchestra.run.smithi192.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 239 kB/s | 256 kB 00:01 2024-03-08T22:52:16.391 INFO:teuthology.orchestra.run.smithi062.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-08T22:52:16.433 INFO:teuthology.orchestra.run.smithi062.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.9 MB/s | 1.5 MB 00:00 2024-03-08T22:52:16.466 INFO:teuthology.orchestra.run.smithi062.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-08T22:52:16.488 INFO:teuthology.orchestra.run.smithi192.stdout:(163/174): NetworkManager-1.46.0-1.el9.x86_64.r 319 kB/s | 2.3 MB 00:07 2024-03-08T22:52:16.491 INFO:teuthology.orchestra.run.smithi062.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-08T22:52:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:(7/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-08T22:52:16.522 INFO:teuthology.orchestra.run.smithi192.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 488 kB/s | 155 kB 00:00 2024-03-08T22:52:16.581 INFO:teuthology.orchestra.run.smithi192.stdout:(165/174): libuuid-2.37.4-18.el9.x86_64.rpm 492 kB/s | 29 kB 00:00 2024-03-08T22:52:16.598 INFO:teuthology.orchestra.run.smithi192.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 587 kB/s | 64 kB 00:00 2024-03-08T22:52:16.642 INFO:teuthology.orchestra.run.smithi062.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-08T22:52:16.664 INFO:teuthology.orchestra.run.smithi192.stdout:(167/174): libmount-2.37.4-18.el9.x86_64.rpm 464 kB/s | 136 kB 00:00 2024-03-08T22:52:16.675 INFO:teuthology.orchestra.run.smithi062.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-08T22:52:16.692 INFO:teuthology.orchestra.run.smithi062.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-08T22:52:16.843 INFO:teuthology.orchestra.run.smithi062.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-08T22:52:16.868 INFO:teuthology.orchestra.run.smithi062.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 139 kB/s | 24 kB 00:00 2024-03-08T22:52:16.893 INFO:teuthology.orchestra.run.smithi062.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-08T22:52:16.919 INFO:teuthology.orchestra.run.smithi062.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-08T22:52:16.935 INFO:teuthology.orchestra.run.smithi062.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 760 kB/s | 32 kB 00:00 2024-03-08T22:52:17.070 INFO:teuthology.orchestra.run.smithi062.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-08T22:52:17.095 INFO:teuthology.orchestra.run.smithi062.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 217 kB 00:00 2024-03-08T22:52:17.120 INFO:teuthology.orchestra.run.smithi062.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-08T22:52:17.146 INFO:teuthology.orchestra.run.smithi062.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-08T22:52:17.171 INFO:teuthology.orchestra.run.smithi062.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 815 kB/s | 41 kB 00:00 2024-03-08T22:52:17.197 INFO:teuthology.orchestra.run.smithi062.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-08T22:52:17.222 INFO:teuthology.orchestra.run.smithi062.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-08T22:52:17.256 INFO:teuthology.orchestra.run.smithi062.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 211 kB 00:00 2024-03-08T22:52:17.281 INFO:teuthology.orchestra.run.smithi062.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-03-08T22:52:17.306 INFO:teuthology.orchestra.run.smithi062.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 852 kB/s | 43 kB 00:00 2024-03-08T22:52:17.448 INFO:teuthology.orchestra.run.smithi062.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 274 kB/s | 39 kB 00:00 2024-03-08T22:52:17.479 INFO:teuthology.orchestra.run.smithi046.stdout:(8/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x86 5.6 MB/s | 17 MB 00:02 2024-03-08T22:52:17.527 INFO:teuthology.orchestra.run.smithi046.stdout:(9/174): ceph-selinux-19.0.0-1965.g7d63482a.el9 539 kB/s | 25 kB 00:00 2024-03-08T22:52:17.640 INFO:teuthology.orchestra.run.smithi192.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 1.1 MB/s | 1.2 MB 00:01 2024-03-08T22:52:17.682 INFO:teuthology.orchestra.run.smithi062.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-03-08T22:52:17.707 INFO:teuthology.orchestra.run.smithi062.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-08T22:52:17.732 INFO:teuthology.orchestra.run.smithi062.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-08T22:52:17.758 INFO:teuthology.orchestra.run.smithi062.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-08T22:52:17.783 INFO:teuthology.orchestra.run.smithi062.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-08T22:52:17.809 INFO:teuthology.orchestra.run.smithi062.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-08T22:52:17.834 INFO:teuthology.orchestra.run.smithi062.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-08T22:52:17.859 INFO:teuthology.orchestra.run.smithi062.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-08T22:52:17.901 INFO:teuthology.orchestra.run.smithi062.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 17 MB/s | 1.1 MB 00:00 2024-03-08T22:52:17.926 INFO:teuthology.orchestra.run.smithi062.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-08T22:52:17.952 INFO:teuthology.orchestra.run.smithi062.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-08T22:52:17.977 INFO:teuthology.orchestra.run.smithi062.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 187 kB 00:00 2024-03-08T22:52:18.402 INFO:teuthology.orchestra.run.smithi062.stdout:(91/174): ceph-test-19.0.0-1965.g7d63482a.el9.x 8.5 MB/s | 47 MB 00:05 2024-03-08T22:52:18.511 INFO:teuthology.orchestra.run.smithi062.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 335 kB/s | 187 kB 00:00 2024-03-08T22:52:18.561 INFO:teuthology.orchestra.run.smithi062.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-08T22:52:18.586 INFO:teuthology.orchestra.run.smithi062.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-03-08T22:52:18.612 INFO:teuthology.orchestra.run.smithi062.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 428 kB/s | 21 kB 00:00 2024-03-08T22:52:18.637 INFO:teuthology.orchestra.run.smithi062.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 506 kB/s | 25 kB 00:00 2024-03-08T22:52:18.688 INFO:teuthology.orchestra.run.smithi062.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-08T22:52:18.713 INFO:teuthology.orchestra.run.smithi062.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-08T22:52:18.738 INFO:teuthology.orchestra.run.smithi062.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-03-08T22:52:18.763 INFO:teuthology.orchestra.run.smithi062.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-08T22:52:18.788 INFO:teuthology.orchestra.run.smithi062.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 695 kB/s | 35 kB 00:00 2024-03-08T22:52:18.814 INFO:teuthology.orchestra.run.smithi062.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-08T22:52:18.831 INFO:teuthology.orchestra.run.smithi062.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-08T22:52:18.889 INFO:teuthology.orchestra.run.smithi062.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 15 MB/s | 4.1 MB 00:00 2024-03-08T22:52:18.931 INFO:teuthology.orchestra.run.smithi062.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-03-08T22:52:19.228 INFO:teuthology.orchestra.run.smithi062.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-03-08T22:52:19.298 INFO:teuthology.orchestra.run.smithi192.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 281 kB/s | 465 kB 00:01 2024-03-08T22:52:19.432 INFO:teuthology.orchestra.run.smithi192.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 233 kB/s | 31 kB 00:00 2024-03-08T22:52:19.452 INFO:teuthology.orchestra.run.smithi046.stdout:(10/174): ceph-radosgw-19.0.0-1965.g7d63482a.el 3.3 MB/s | 9.8 MB 00:02 2024-03-08T22:52:19.565 INFO:teuthology.orchestra.run.smithi062.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 811 kB/s | 548 kB 00:00 2024-03-08T22:52:19.591 INFO:teuthology.orchestra.run.smithi062.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 54 kB/s | 19 kB 00:00 2024-03-08T22:52:19.624 INFO:teuthology.orchestra.run.smithi062.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 446 kB/s | 309 kB 00:00 2024-03-08T22:52:19.716 INFO:teuthology.orchestra.run.smithi062.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 207 kB/s | 26 kB 00:00 2024-03-08T22:52:19.727 INFO:teuthology.orchestra.run.smithi046.stdout:(11/174): ceph-common-19.0.0-1965.g7d63482a.el9 2.4 MB/s | 20 MB 00:08 2024-03-08T22:52:19.741 INFO:teuthology.orchestra.run.smithi062.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2024-03-08T22:52:19.760 INFO:teuthology.orchestra.run.smithi046.stdout:(12/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 715 kB/s | 220 kB 00:00 2024-03-08T22:52:19.794 INFO:teuthology.orchestra.run.smithi046.stdout:(13/174): cephfs-shell-19.0.0-1965.g7d63482a.el 411 kB/s | 27 kB 00:00 2024-03-08T22:52:19.820 INFO:teuthology.orchestra.run.smithi046.stdout:(14/174): libcephfs-devel-19.0.0-1965.g7d63482a 537 kB/s | 31 kB 00:00 2024-03-08T22:52:19.867 INFO:teuthology.orchestra.run.smithi062.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-03-08T22:52:19.886 INFO:teuthology.orchestra.run.smithi046.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 2.4 MB/s | 163 kB 00:00 2024-03-08T22:52:19.900 INFO:teuthology.orchestra.run.smithi062.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 367 kB/s | 67 kB 00:00 2024-03-08T22:52:19.962 INFO:teuthology.orchestra.run.smithi046.stdout:(16/174): librados-devel-19.0.0-1965.g7d63482a. 1.7 MB/s | 127 kB 00:00 2024-03-08T22:52:19.967 INFO:teuthology.orchestra.run.smithi062.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-08T22:52:20.045 INFO:teuthology.orchestra.run.smithi046.stdout:(17/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 2.9 MB/s | 745 kB 00:00 2024-03-08T22:52:20.067 INFO:teuthology.orchestra.run.smithi062.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 435 kB/s | 43 kB 00:00 2024-03-08T22:52:20.126 INFO:teuthology.orchestra.run.smithi062.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.2 MB/s | 838 kB 00:00 2024-03-08T22:52:20.137 INFO:teuthology.orchestra.run.smithi046.stdout:(18/174): libradosstriper1-19.0.0-1965.g7d63482 2.8 MB/s | 501 kB 00:00 2024-03-08T22:52:20.168 INFO:teuthology.orchestra.run.smithi062.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 321 kB/s | 32 kB 00:00 2024-03-08T22:52:20.204 INFO:teuthology.orchestra.run.smithi046.stdout:(19/174): python3-ceph-argparse-19.0.0-1965.g7d 687 kB/s | 45 kB 00:00 2024-03-08T22:52:20.227 INFO:teuthology.orchestra.run.smithi062.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 150 kB/s | 15 kB 00:00 2024-03-08T22:52:20.260 INFO:teuthology.orchestra.run.smithi062.stdout:(119/174): python3-asyncssh-2.13.2-4.el9.noarch 1.5 MB/s | 548 kB 00:00 2024-03-08T22:52:20.271 INFO:teuthology.orchestra.run.smithi046.stdout:(20/174): python3-ceph-common-19.0.0-1965.g7d63 1.9 MB/s | 131 kB 00:00 2024-03-08T22:52:20.277 INFO:teuthology.orchestra.run.smithi062.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-08T22:52:20.346 INFO:teuthology.orchestra.run.smithi046.stdout:(21/174): python3-cephfs-19.0.0-1965.g7d63482a. 2.1 MB/s | 162 kB 00:00 2024-03-08T22:52:20.347 INFO:teuthology.orchestra.run.smithi062.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.9 MB/s | 358 kB 00:00 2024-03-08T22:52:20.372 INFO:teuthology.orchestra.run.smithi062.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 372 kB/s | 35 kB 00:00 2024-03-08T22:52:20.397 INFO:teuthology.orchestra.run.smithi062.stdout:(123/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-08T22:52:20.456 INFO:teuthology.orchestra.run.smithi062.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-08T22:52:20.463 INFO:teuthology.orchestra.run.smithi046.stdout:(22/174): python3-rados-19.0.0-1965.g7d63482a.e 2.7 MB/s | 320 kB 00:00 2024-03-08T22:52:20.515 INFO:teuthology.orchestra.run.smithi062.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-03-08T22:52:20.540 INFO:teuthology.orchestra.run.smithi062.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-03-08T22:52:20.557 INFO:teuthology.orchestra.run.smithi062.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-03-08T22:52:20.572 INFO:teuthology.orchestra.run.smithi046.stdout:(23/174): python3-rbd-19.0.0-1965.g7d63482a.el9 2.7 MB/s | 298 kB 00:00 2024-03-08T22:52:20.608 INFO:teuthology.orchestra.run.smithi062.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-03-08T22:52:20.631 INFO:teuthology.orchestra.run.smithi046.stdout:(24/174): python3-rgw-19.0.0-1965.g7d63482a.el9 1.7 MB/s | 99 kB 00:00 2024-03-08T22:52:20.649 INFO:teuthology.orchestra.run.smithi062.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-08T22:52:20.674 INFO:teuthology.orchestra.run.smithi062.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 166 kB/s | 19 kB 00:00 2024-03-08T22:52:20.697 INFO:teuthology.orchestra.run.smithi046.stdout:(25/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-08T22:52:20.700 INFO:teuthology.orchestra.run.smithi062.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-03-08T22:52:20.775 INFO:teuthology.orchestra.run.smithi062.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 460 kB/s | 46 kB 00:00 2024-03-08T22:52:20.792 INFO:teuthology.orchestra.run.smithi062.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-03-08T22:52:20.834 INFO:teuthology.orchestra.run.smithi062.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-03-08T22:52:20.876 INFO:teuthology.orchestra.run.smithi062.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-08T22:52:20.901 INFO:teuthology.orchestra.run.smithi062.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-08T22:52:20.935 INFO:teuthology.orchestra.run.smithi062.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-08T22:52:20.985 INFO:teuthology.orchestra.run.smithi062.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 826 kB/s | 90 kB 00:00 2024-03-08T22:52:21.010 INFO:teuthology.orchestra.run.smithi062.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 259 kB/s | 28 kB 00:00 2024-03-08T22:52:21.035 INFO:teuthology.orchestra.run.smithi062.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-08T22:52:21.103 INFO:teuthology.orchestra.run.smithi062.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-08T22:52:21.128 INFO:teuthology.orchestra.run.smithi062.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-08T22:52:21.153 INFO:teuthology.orchestra.run.smithi062.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2024-03-08T22:52:21.203 INFO:teuthology.orchestra.run.smithi062.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-08T22:52:21.229 INFO:teuthology.orchestra.run.smithi062.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-08T22:52:21.254 INFO:teuthology.orchestra.run.smithi062.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 891 kB/s | 90 kB 00:00 2024-03-08T22:52:21.329 INFO:teuthology.orchestra.run.smithi062.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-03-08T22:52:21.355 INFO:teuthology.orchestra.run.smithi062.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-08T22:52:21.380 INFO:teuthology.orchestra.run.smithi062.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-08T22:52:21.463 INFO:teuthology.orchestra.run.smithi062.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-08T22:52:21.505 INFO:teuthology.orchestra.run.smithi062.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-08T22:52:21.514 INFO:teuthology.orchestra.run.smithi046.stdout:(26/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 3.5 MB/s | 5.1 MB 00:01 2024-03-08T22:52:21.640 INFO:teuthology.orchestra.run.smithi046.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 1.3 MB/s | 171 kB 00:00 2024-03-08T22:52:21.674 INFO:teuthology.orchestra.run.smithi192.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 435 kB/s | 2.2 MB 00:05 2024-03-08T22:52:21.764 INFO:teuthology.orchestra.run.smithi062.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.9 MB/s | 2.3 MB 00:00 2024-03-08T22:52:21.790 INFO:teuthology.orchestra.run.smithi046.stdout:(28/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 2.8 MB/s | 3.1 MB 00:01 2024-03-08T22:52:21.807 INFO:teuthology.orchestra.run.smithi192.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 559 kB/s | 74 kB 00:00 2024-03-08T22:52:21.815 INFO:teuthology.orchestra.run.smithi046.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1965.g 195 kB/s | 34 kB 00:00 2024-03-08T22:52:21.882 INFO:teuthology.orchestra.run.smithi046.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 1.5 MB/s | 139 kB 00:00 2024-03-08T22:52:21.923 INFO:teuthology.orchestra.run.smithi062.stdout:(153/174): librados2-19.0.0-1965.g7d63482a.el9. 6.0 MB/s | 3.4 MB 00:00 2024-03-08T22:52:21.965 INFO:teuthology.orchestra.run.smithi062.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 9.1 MB/s | 1.8 MB 00:00 2024-03-08T22:52:22.023 INFO:teuthology.orchestra.run.smithi062.stdout:(155/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 4.9 MB/s | 3.1 MB 00:00 2024-03-08T22:52:22.048 INFO:teuthology.orchestra.run.smithi062.stdout:(156/174): NetworkManager-team-1.46.0-1.el9.x86 338 kB/s | 42 kB 00:00 2024-03-08T22:52:22.049 INFO:teuthology.orchestra.run.smithi192.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 434 kB/s | 2.3 MB 00:05 2024-03-08T22:52:22.074 INFO:teuthology.orchestra.run.smithi062.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-03-08T22:52:22.107 INFO:teuthology.orchestra.run.smithi062.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 851 kB/s | 71 kB 00:00 2024-03-08T22:52:22.141 INFO:teuthology.orchestra.run.smithi062.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-08T22:52:22.149 INFO:teuthology.orchestra.run.smithi192.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 404 kB/s | 1.1 MB 00:02 2024-03-08T22:52:22.154 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-03-08T22:52:22.154 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.6 MB/s | 223 MB 00:33 2024-03-08T22:52:22.166 INFO:teuthology.orchestra.run.smithi062.stdout:(160/174): elfutils-debuginfod-client-0.190-2.e 310 kB/s | 36 kB 00:00 2024-03-08T22:52:22.200 INFO:teuthology.orchestra.run.smithi062.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 391 kB/s | 23 kB 00:00 2024-03-08T22:52:22.242 INFO:teuthology.orchestra.run.smithi062.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.9 MB/s | 256 kB 00:00 2024-03-08T22:52:22.267 INFO:teuthology.orchestra.run.smithi062.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 155 kB 00:00 2024-03-08T22:52:22.284 INFO:teuthology.orchestra.run.smithi062.stdout:(164/174): libblkid-2.37.4-18.el9.x86_64.rpm 921 kB/s | 108 kB 00:00 2024-03-08T22:52:22.309 INFO:teuthology.orchestra.run.smithi062.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-08T22:52:22.326 INFO:teuthology.orchestra.run.smithi062.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 687 kB/s | 29 kB 00:00 2024-03-08T22:52:22.343 INFO:teuthology.orchestra.run.smithi062.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 843 kB/s | 64 kB 00:00 2024-03-08T22:52:22.357 INFO:teuthology.orchestra.run.smithi046.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 3.3 MB/s | 1.8 MB 00:00 2024-03-08T22:52:22.435 INFO:teuthology.orchestra.run.smithi062.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-08T22:52:22.494 INFO:teuthology.orchestra.run.smithi062.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 7.8 MB/s | 465 kB 00:00 2024-03-08T22:52:22.544 INFO:teuthology.orchestra.run.smithi062.stdout:(170/174): util-linux-2.37.4-18.el9.x86_64.rpm 11 MB/s | 2.3 MB 00:00 2024-03-08T22:52:22.569 INFO:teuthology.orchestra.run.smithi062.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 413 kB/s | 31 kB 00:00 2024-03-08T22:52:22.633 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-03-08T22:52:22.636 INFO:teuthology.orchestra.run.smithi062.stdout:(172/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.0 MB/s | 2.2 MB 00:00 2024-03-08T22:52:22.662 INFO:teuthology.orchestra.run.smithi062.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 810 kB/s | 74 kB 00:00 2024-03-08T22:52:22.695 INFO:teuthology.orchestra.run.smithi062.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 7.1 MB/s | 1.1 MB 00:00 2024-03-08T22:52:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-03-08T22:52:22.701 INFO:teuthology.orchestra.run.smithi062.stdout:Total 16 MB/s | 223 MB 00:13 2024-03-08T22:52:22.904 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-03-08T22:52:22.905 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-03-08T22:52:22.941 INFO:teuthology.orchestra.run.smithi046.stdout:(32/174): ceph-test-19.0.0-1965.g7d63482a.el9.x 8.8 MB/s | 47 MB 00:05 2024-03-08T22:52:22.966 INFO:teuthology.orchestra.run.smithi046.stdout:(33/174): ceph-mgr-modules-core-19.0.0-1965.g7d 408 kB/s | 248 kB 00:00 2024-03-08T22:52:22.991 INFO:teuthology.orchestra.run.smithi046.stdout:(34/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 1.0 MB/s | 49 kB 00:00 2024-03-08T22:52:23.033 INFO:teuthology.orchestra.run.smithi046.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1965.g7 249 kB/s | 16 kB 00:00 2024-03-08T22:52:23.084 INFO:teuthology.orchestra.run.smithi046.stdout:(36/174): ceph-volume-19.0.0-1965.g7d63482a.el9 2.8 MB/s | 264 kB 00:00 2024-03-08T22:52:23.134 INFO:teuthology.orchestra.run.smithi046.stdout:(37/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 372 kB/s | 18 kB 00:00 2024-03-08T22:52:23.170 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-03-08T22:52:23.326 INFO:teuthology.orchestra.run.smithi046.stdout:(38/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 2.5 MB/s | 748 kB 00:00 2024-03-08T22:52:23.409 INFO:teuthology.orchestra.run.smithi046.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-08T22:52:23.451 INFO:teuthology.orchestra.run.smithi046.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 920 kB 00:00 2024-03-08T22:52:23.467 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-03-08T22:52:23.467 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-03-08T22:52:23.476 INFO:teuthology.orchestra.run.smithi046.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-08T22:52:23.510 INFO:teuthology.orchestra.run.smithi046.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-08T22:52:23.535 INFO:teuthology.orchestra.run.smithi046.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 578 kB/s | 32 kB 00:00 2024-03-08T22:52:23.644 INFO:teuthology.orchestra.run.smithi046.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-03-08T22:52:23.677 INFO:teuthology.orchestra.run.smithi046.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 133 kB/s | 17 kB 00:00 2024-03-08T22:52:23.719 INFO:teuthology.orchestra.run.smithi046.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 249 kB/s | 10 kB 00:00 2024-03-08T22:52:23.745 INFO:teuthology.orchestra.run.smithi046.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.4 MB/s | 159 kB 00:00 2024-03-08T22:52:23.770 INFO:teuthology.orchestra.run.smithi046.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 920 kB/s | 44 kB 00:00 2024-03-08T22:52:23.812 INFO:teuthology.orchestra.run.smithi046.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.9 MB/s | 661 kB 00:00 2024-03-08T22:52:23.854 INFO:teuthology.orchestra.run.smithi046.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-03-08T22:52:23.879 INFO:teuthology.orchestra.run.smithi046.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 760 kB/s | 46 kB 00:00 2024-03-08T22:52:23.913 INFO:teuthology.orchestra.run.smithi046.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-08T22:52:23.946 INFO:teuthology.orchestra.run.smithi046.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-03-08T22:52:23.980 INFO:teuthology.orchestra.run.smithi046.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-08T22:52:24.005 INFO:teuthology.orchestra.run.smithi046.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 613 kB/s | 34 kB 00:00 2024-03-08T22:52:24.139 INFO:teuthology.orchestra.run.smithi046.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-03-08T22:52:24.198 INFO:teuthology.orchestra.run.smithi046.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.7 MB/s | 1.5 MB 00:00 2024-03-08T22:52:24.270 INFO:teuthology.orchestra.run.smithi046.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 6.7 MB/s | 811 kB 00:00 2024-03-08T22:52:24.298 INFO:teuthology.orchestra.run.smithi046.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.4 MB/s | 134 kB 00:00 2024-03-08T22:52:24.448 INFO:teuthology.orchestra.run.smithi046.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-08T22:52:24.498 INFO:teuthology.orchestra.run.smithi046.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-08T22:52:24.524 INFO:teuthology.orchestra.run.smithi046.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-08T22:52:24.683 INFO:teuthology.orchestra.run.smithi046.stdout:(63/174): ceph-mgr-diskprediction-local-19.0.0- 2.6 MB/s | 7.4 MB 00:02 2024-03-08T22:52:24.683 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-03-08T22:52:24.683 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-03-08T22:52:24.816 INFO:teuthology.orchestra.run.smithi046.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-08T22:52:24.833 INFO:teuthology.orchestra.run.smithi046.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 83 kB/s | 24 kB 00:00 2024-03-08T22:52:24.850 INFO:teuthology.orchestra.run.smithi046.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 118 kB/s | 18 kB 00:00 2024-03-08T22:52:24.908 INFO:teuthology.orchestra.run.smithi046.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-08T22:52:24.934 INFO:teuthology.orchestra.run.smithi046.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 319 kB/s | 32 kB 00:00 2024-03-08T22:52:25.109 INFO:teuthology.orchestra.run.smithi046.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.1 MB/s | 217 kB 00:00 2024-03-08T22:52:25.151 INFO:teuthology.orchestra.run.smithi046.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 809 kB/s | 174 kB 00:00 2024-03-08T22:52:25.268 INFO:teuthology.orchestra.run.smithi046.stdout:(71/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-08T22:52:25.283 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-03-08T22:52:25.284 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-03-08T22:52:25.293 INFO:teuthology.orchestra.run.smithi046.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 399 kB/s | 69 kB 00:00 2024-03-08T22:52:25.443 INFO:teuthology.orchestra.run.smithi046.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 146 kB/s | 41 kB 00:00 2024-03-08T22:52:25.477 INFO:teuthology.orchestra.run.smithi046.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 1.1 MB/s | 238 kB 00:00 2024-03-08T22:52:25.510 INFO:teuthology.orchestra.run.smithi046.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2024-03-08T22:52:25.544 INFO:teuthology.orchestra.run.smithi046.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.1 MB/s | 211 kB 00:00 2024-03-08T22:52:25.569 INFO:teuthology.orchestra.run.smithi046.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 535 kB/s | 47 kB 00:00 2024-03-08T22:52:25.603 INFO:teuthology.orchestra.run.smithi046.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 747 kB/s | 43 kB 00:00 2024-03-08T22:52:25.762 INFO:teuthology.orchestra.run.smithi046.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 202 kB/s | 39 kB 00:00 2024-03-08T22:52:25.937 INFO:teuthology.orchestra.run.smithi046.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 123 kB/s | 41 kB 00:00 2024-03-08T22:52:26.221 INFO:teuthology.orchestra.run.smithi046.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-03-08T22:52:26.362 INFO:teuthology.orchestra.run.smithi046.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 513 kB/s | 300 kB 00:00 2024-03-08T22:52:26.387 INFO:teuthology.orchestra.run.smithi046.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 129 kB/s | 56 kB 00:00 2024-03-08T22:52:26.413 INFO:teuthology.orchestra.run.smithi046.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 329 kB/s | 63 kB 00:00 2024-03-08T22:52:26.438 INFO:teuthology.orchestra.run.smithi046.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 907 kB/s | 44 kB 00:00 2024-03-08T22:52:26.455 INFO:teuthology.orchestra.run.smithi046.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-08T22:52:26.488 INFO:teuthology.orchestra.run.smithi046.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-08T22:52:26.514 INFO:teuthology.orchestra.run.smithi046.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-08T22:52:26.572 INFO:teuthology.orchestra.run.smithi046.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 10 MB/s | 1.1 MB 00:00 2024-03-08T22:52:26.606 INFO:teuthology.orchestra.run.smithi046.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-08T22:52:26.631 INFO:teuthology.orchestra.run.smithi046.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 187 kB 00:00 2024-03-08T22:52:26.823 INFO:teuthology.orchestra.run.smithi046.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 757 kB/s | 187 kB 00:00 2024-03-08T22:52:26.848 INFO:teuthology.orchestra.run.smithi046.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 906 kB/s | 195 kB 00:00 2024-03-08T22:52:26.882 INFO:teuthology.orchestra.run.smithi046.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-08T22:52:26.907 INFO:teuthology.orchestra.run.smithi046.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 373 kB/s | 21 kB 00:00 2024-03-08T22:52:26.932 INFO:teuthology.orchestra.run.smithi046.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 515 kB/s | 25 kB 00:00 2024-03-08T22:52:26.982 INFO:teuthology.orchestra.run.smithi046.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-08T22:52:27.016 INFO:teuthology.orchestra.run.smithi046.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-08T22:52:27.050 INFO:teuthology.orchestra.run.smithi046.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.7 MB/s | 248 kB 00:00 2024-03-08T22:52:27.092 INFO:teuthology.orchestra.run.smithi046.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 216 kB 00:00 2024-03-08T22:52:27.109 INFO:teuthology.orchestra.run.smithi046.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 604 kB/s | 35 kB 00:00 2024-03-08T22:52:27.134 INFO:teuthology.orchestra.run.smithi046.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-08T22:52:27.167 INFO:teuthology.orchestra.run.smithi046.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-03-08T22:52:27.218 INFO:teuthology.orchestra.run.smithi046.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 270 kB/s | 22 kB 00:00 2024-03-08T22:52:27.385 INFO:teuthology.orchestra.run.smithi046.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 8.7 MB/s | 4.1 MB 00:00 2024-03-08T22:52:27.660 INFO:teuthology.orchestra.run.smithi046.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-03-08T22:52:29.293 INFO:teuthology.orchestra.run.smithi046.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 162 kB/s | 309 kB 00:01 2024-03-08T22:52:29.643 INFO:teuthology.orchestra.run.smithi046.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 9.8 kB/s | 19 kB 00:01 2024-03-08T22:52:30.261 INFO:teuthology.orchestra.run.smithi046.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 180 kB/s | 548 kB 00:03 2024-03-08T22:52:31.502 INFO:teuthology.orchestra.run.smithi046.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 14 kB/s | 26 kB 00:01 2024-03-08T22:52:31.511 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-03-08T22:52:31.619 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T22:52:31.653 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T22:52:31.692 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:52:31.702 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:52:31.750 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T22:52:31.803 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T22:52:31.836 INFO:teuthology.orchestra.run.smithi046.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 31 kB/s | 49 kB 00:01 2024-03-08T22:52:31.837 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T22:52:31.871 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T22:52:31.905 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T22:52:32.236 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T22:52:32.289 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:52:32.387 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-03-08T22:52:32.410 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:52:32.440 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:52:32.533 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:52:32.560 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:52:32.594 INFO:teuthology.orchestra.run.smithi046.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 62 kB/s | 67 kB 00:01 2024-03-08T22:52:32.603 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T22:52:32.632 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T22:52:32.666 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T22:52:32.706 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:52:32.715 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:52:32.757 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T22:52:32.778 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T22:52:32.810 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T22:52:32.825 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T22:52:32.843 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T22:52:32.853 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:52:32.868 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T22:52:32.911 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T22:52:32.948 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:52:32.971 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:52:33.018 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:52:33.058 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T22:52:33.081 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:52:33.097 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:52:33.247 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:52:33.255 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T22:52:33.278 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:52:33.310 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T22:52:33.311 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:52:33.336 INFO:teuthology.orchestra.run.smithi046.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 101 kB/s | 151 kB 00:01 2024-03-08T22:52:33.407 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T22:52:33.418 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:52:33.445 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:52:33.495 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:52:33.520 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:52:33.547 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:52:33.566 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:52:33.612 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T22:52:33.614 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:33.665 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:33.791 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T22:52:33.840 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T22:52:33.867 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:52:33.871 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:33.956 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:52:33.977 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T22:52:33.985 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:52:34.033 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:52:34.064 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T22:52:34.072 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T22:52:34.086 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:52:34.103 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:52:34.103 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T22:52:34.130 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T22:52:34.165 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T22:52:34.239 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T22:52:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:52:34.301 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T22:52:34.326 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:52:34.340 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T22:52:34.358 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T22:52:34.403 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T22:52:34.439 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T22:52:34.464 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T22:52:34.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T22:52:34.515 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T22:52:34.544 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T22:52:34.570 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:52:34.612 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T22:52:34.614 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:52:34.648 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T22:52:34.681 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:34.683 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T22:52:34.729 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T22:52:34.797 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T22:52:34.843 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T22:52:34.891 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:34.915 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T22:52:35.047 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T22:52:35.103 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T22:52:35.124 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:52:35.235 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T22:52:35.337 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T22:52:35.496 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T22:52:35.522 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T22:52:35.558 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T22:52:35.615 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T22:52:35.652 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T22:52:35.699 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T22:52:35.712 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T22:52:35.750 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T22:52:35.768 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T22:52:35.820 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:52:35.846 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T22:52:35.847 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:52:35.859 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T22:52:35.866 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T22:52:35.909 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T22:52:35.909 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T22:52:35.942 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T22:52:35.946 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T22:52:35.969 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T22:52:35.981 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T22:52:36.042 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T22:52:36.082 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T22:52:36.127 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T22:52:36.155 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T22:52:36.193 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T22:52:36.232 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T22:52:36.278 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T22:52:36.354 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T22:52:36.369 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T22:52:36.452 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T22:52:36.497 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T22:52:36.503 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T22:52:36.556 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T22:52:36.630 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T22:52:36.666 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T22:52:36.721 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T22:52:36.784 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T22:52:36.818 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T22:52:36.854 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T22:52:36.900 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T22:52:36.936 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T22:52:36.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T22:52:37.021 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T22:52:37.111 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T22:52:37.162 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T22:52:37.201 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T22:52:37.202 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T22:52:37.232 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T22:52:37.273 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T22:52:37.284 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T22:52:37.328 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:52:37.337 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:52:37.365 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T22:52:37.368 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T22:52:37.407 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T22:52:37.409 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T22:52:37.446 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T22:52:37.447 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T22:52:37.473 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T22:52:37.476 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T22:52:37.497 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T22:52:37.568 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T22:52:37.687 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T22:52:37.699 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T22:52:37.790 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T22:52:37.987 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T22:52:38.107 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T22:52:38.149 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T22:52:38.164 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T22:52:38.209 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T22:52:38.284 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T22:52:38.355 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T22:52:38.409 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T22:52:38.465 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T22:52:38.466 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T22:52:38.509 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T22:52:38.538 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T22:52:38.568 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T22:52:38.604 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T22:52:38.636 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T22:52:38.652 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T22:52:38.682 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T22:52:38.687 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T22:52:38.714 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T22:52:38.731 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T22:52:38.745 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T22:52:38.773 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T22:52:38.773 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T22:52:38.816 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T22:52:38.855 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T22:52:38.886 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T22:52:38.911 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T22:52:38.916 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T22:52:38.962 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T22:52:38.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T22:52:38.991 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T22:52:39.027 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T22:52:39.034 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T22:52:39.052 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T22:52:39.092 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T22:52:39.119 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T22:52:39.131 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T22:52:39.167 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T22:52:39.207 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T22:52:39.208 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T22:52:39.232 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T22:52:39.257 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T22:52:39.289 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T22:52:39.332 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T22:52:39.333 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T22:52:39.365 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T22:52:39.411 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T22:52:39.471 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T22:52:39.508 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T22:52:39.558 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T22:52:39.576 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T22:52:39.624 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T22:52:39.681 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T22:52:39.737 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T22:52:39.927 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T22:52:39.970 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T22:52:40.112 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T22:52:40.151 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T22:52:40.182 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T22:52:40.213 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T22:52:40.241 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T22:52:40.272 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T22:52:40.275 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T22:52:40.315 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T22:52:40.321 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T22:52:40.346 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T22:52:40.355 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T22:52:40.377 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T22:52:40.378 INFO:teuthology.orchestra.run.smithi046.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 78 kB/s | 548 kB 00:07 2024-03-08T22:52:40.403 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T22:52:40.455 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T22:52:40.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T22:52:40.495 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T22:52:40.523 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T22:52:40.528 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T22:52:40.570 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T22:52:40.604 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T22:52:40.655 INFO:teuthology.orchestra.run.smithi046.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 104 kB/s | 838 kB 00:08 2024-03-08T22:52:40.692 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T22:52:40.696 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T22:52:40.721 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T22:52:40.797 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T22:52:40.803 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T22:52:40.845 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T22:52:40.893 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T22:52:41.040 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T22:52:41.295 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T22:52:41.412 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T22:52:41.429 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T22:52:41.534 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T22:52:41.813 INFO:teuthology.orchestra.run.smithi046.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 30 kB/s | 43 kB 00:01 2024-03-08T22:52:41.863 INFO:teuthology.orchestra.run.smithi046.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 27 kB/s | 32 kB 00:01 2024-03-08T22:52:42.180 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T22:52:42.241 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T22:52:42.274 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T22:52:42.324 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T22:52:42.414 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T22:52:42.450 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T22:52:42.636 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T22:52:42.672 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T22:52:42.739 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T22:52:42.753 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T22:52:42.829 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T22:52:42.860 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T22:52:42.892 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T22:52:42.919 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T22:52:43.232 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T22:52:43.257 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:52:43.347 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:52:43.379 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T22:52:43.415 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T22:52:43.425 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T22:52:43.539 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T22:52:43.680 INFO:teuthology.orchestra.run.smithi046.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 8.0 kB/s | 15 kB 00:01 2024-03-08T22:52:43.870 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T22:52:43.902 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:52:43.964 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:52:43.987 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T22:52:44.605 INFO:teuthology.orchestra.run.smithi046.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 63 kB/s | 172 kB 00:02 2024-03-08T22:52:44.825 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T22:52:44.907 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T22:52:44.937 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T22:52:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T22:52:44.997 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T22:52:45.325 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T22:52:45.359 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:52:45.451 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:52:45.480 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T22:52:45.517 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T22:52:46.011 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T22:52:46.045 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:52:46.087 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:46.117 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:52:46.117 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:46.147 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T22:52:46.658 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:46.695 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:52:46.776 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:52:46.777 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T22:52:46.777 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T22:52:46.777 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:52:46.804 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:52:47.080 INFO:teuthology.orchestra.run.smithi046.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 105 kB/s | 358 kB 00:03 2024-03-08T22:52:47.347 INFO:teuthology.orchestra.run.smithi046.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 111 kB/s | 303 kB 00:02 2024-03-08T22:52:47.897 INFO:teuthology.orchestra.run.smithi046.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 43 kB/s | 35 kB 00:00 2024-03-08T22:52:48.322 INFO:teuthology.orchestra.run.smithi046.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 208 kB/s | 203 kB 00:00 2024-03-08T22:52:48.372 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:48.402 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:48.969 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:52:49.006 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:52:49.088 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:52:49.088 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T22:52:49.088 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T22:52:49.088 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:52:49.115 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:52:50.339 INFO:teuthology.orchestra.run.smithi046.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 72 kB/s | 144 kB 00:02 2024-03-08T22:52:50.923 INFO:teuthology.orchestra.run.smithi046.stdout:(125/174): libarrow-9.0.0-10.el9.x86_64.rpm 209 kB/s | 4.4 MB 00:21 2024-03-08T22:52:50.990 INFO:teuthology.orchestra.run.smithi046.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 16 kB/s | 11 kB 00:00 2024-03-08T22:52:51.390 INFO:teuthology.orchestra.run.smithi046.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 38 kB/s | 18 kB 00:00 2024-03-08T22:52:51.415 INFO:teuthology.orchestra.run.smithi046.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 54 kB/s | 23 kB 00:00 2024-03-08T22:52:51.666 INFO:teuthology.orchestra.run.smithi046.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-03-08T22:52:51.682 INFO:teuthology.orchestra.run.smithi046.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 74 kB/s | 20 kB 00:00 2024-03-08T22:52:51.991 INFO:teuthology.orchestra.run.smithi046.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 151 kB/s | 46 kB 00:00 2024-03-08T22:52:52.533 INFO:teuthology.orchestra.run.smithi046.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 146 kB/s | 79 kB 00:00 2024-03-08T22:52:53.192 INFO:teuthology.orchestra.run.smithi046.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 88 kB/s | 58 kB 00:00 2024-03-08T22:52:53.464 INFO:teuthology.orchestra.run.smithi046.stdout:(134/174): python3-grpcio-1.46.7-10.el9.x86_64. 390 kB/s | 2.0 MB 00:05 2024-03-08T22:52:53.983 INFO:teuthology.orchestra.run.smithi046.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 22 kB/s | 16 kB 00:00 2024-03-08T22:52:54.492 INFO:teuthology.orchestra.run.smithi046.stdout:(136/174): python3-kubernetes-26.1.0-2.el9.noar 370 kB/s | 1.0 MB 00:02 2024-03-08T22:52:54.802 INFO:teuthology.orchestra.run.smithi046.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 169 kB/s | 272 kB 00:01 2024-03-08T22:52:54.926 INFO:teuthology.orchestra.run.smithi046.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 95 kB/s | 90 kB 00:00 2024-03-08T22:52:54.951 INFO:teuthology.orchestra.run.smithi046.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 62 kB/s | 28 kB 00:00 2024-03-08T22:52:54.993 INFO:teuthology.orchestra.run.smithi046.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 161 kB/s | 31 kB 00:00 2024-03-08T22:52:55.285 INFO:teuthology.orchestra.run.smithi046.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 527 kB/s | 188 kB 00:00 2024-03-08T22:52:55.594 INFO:teuthology.orchestra.run.smithi046.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 92 kB/s | 59 kB 00:00 2024-03-08T22:52:55.869 INFO:teuthology.orchestra.run.smithi046.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 41 kB/s | 36 kB 00:00 2024-03-08T22:52:56.136 INFO:teuthology.orchestra.run.smithi046.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 60 kB/s | 51 kB 00:00 2024-03-08T22:52:57.194 INFO:teuthology.orchestra.run.smithi046.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 143 kB/s | 230 kB 00:01 2024-03-08T22:52:57.370 INFO:teuthology.orchestra.run.smithi046.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 60 kB/s | 90 kB 00:01 2024-03-08T22:52:57.887 INFO:teuthology.orchestra.run.smithi046.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 29 kB/s | 20 kB 00:00 2024-03-08T22:52:58.053 INFO:teuthology.orchestra.run.smithi046.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 223 kB/s | 427 kB 00:01 2024-03-08T22:52:58.179 INFO:teuthology.orchestra.run.smithi046.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 237 kB/s | 192 kB 00:00 2024-03-08T22:52:58.439 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:52:58.439 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-03-08T22:52:58.439 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-03-08T22:52:58.439 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-03-08T22:52:58.440 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-03-08T22:52:58.440 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-03-08T22:52:58.440 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-03-08T22:52:58.440 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-03-08T22:52:58.440 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:52:58.537 INFO:teuthology.orchestra.run.smithi046.stdout:(150/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-03-08T22:52:58.596 INFO:teuthology.orchestra.run.smithi046.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 5.5 MB/s | 321 kB 00:00 2024-03-08T22:52:58.610 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:52:58.626 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:52:58.627 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:52:58.627 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T22:52:58.627 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:52:58.947 INFO:teuthology.orchestra.run.smithi046.stdout:(152/174): librados2-19.0.0-1965.g7d63482a.el9. 3.8 MB/s | 3.4 MB 00:00 2024-03-08T22:53:00.105 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:00.131 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:00.339 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:00.362 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:00.806 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:00.829 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:00.924 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /sys 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /proc 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /mnt 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /var/tmp 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /home 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /root 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout:skipping the directory /tmp 2024-03-08T22:53:01.233 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:01.267 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:01.298 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:01.355 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T22:53:01.396 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:53:01.414 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:53:01.415 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:01.415 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T22:53:01.415 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:01.416 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T22:53:01.446 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:01.683 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:01.714 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:53:01.938 INFO:teuthology.orchestra.run.smithi046.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 400 kB/s | 1.6 MB 00:04 2024-03-08T22:53:02.046 INFO:teuthology.orchestra.run.smithi046.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 394 kB/s | 42 kB 00:00 2024-03-08T22:53:02.355 INFO:teuthology.orchestra.run.smithi046.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 546 kB/s | 1.8 MB 00:03 2024-03-08T22:53:02.538 INFO:teuthology.orchestra.run.smithi046.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 505 kB/s | 248 kB 00:00 2024-03-08T22:53:02.597 INFO:teuthology.orchestra.run.smithi046.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 296 kB/s | 71 kB 00:00 2024-03-08T22:53:02.622 INFO:teuthology.orchestra.run.smithi046.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 440 kB/s | 36 kB 00:00 2024-03-08T22:53:02.682 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:53:02.714 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:53:02.797 INFO:teuthology.orchestra.run.smithi046.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.4 MB/s | 256 kB 00:00 2024-03-08T22:53:02.814 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:53:02.856 INFO:teuthology.orchestra.run.smithi046.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 401 kB/s | 23 kB 00:00 2024-03-08T22:53:02.923 INFO:teuthology.orchestra.run.smithi046.stdout:(161/174): elfutils-libelf-0.190-2.el9.x86_64.r 592 kB/s | 192 kB 00:00 2024-03-08T22:53:02.948 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:02.974 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:02.978 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T22:53:02.981 INFO:teuthology.orchestra.run.smithi046.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 871 kB/s | 108 kB 00:00 2024-03-08T22:53:03.007 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:53:03.031 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:53:03.031 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:03.031 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T22:53:03.031 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:53:03.031 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:53:03.032 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:03.075 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:53:03.086 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:53:03.098 INFO:teuthology.orchestra.run.smithi046.stdout:(163/174): libmount-2.37.4-18.el9.x86_64.rpm 1.1 MB/s | 136 kB 00:00 2024-03-08T22:53:03.131 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T22:53:03.165 INFO:teuthology.orchestra.run.smithi046.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 642 kB/s | 155 kB 00:00 2024-03-08T22:53:03.190 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:03.199 INFO:teuthology.orchestra.run.smithi046.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 641 kB/s | 64 kB 00:00 2024-03-08T22:53:03.213 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:03.213 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:03.213 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T22:53:03.214 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:03.214 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:03.214 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:03.258 INFO:teuthology.orchestra.run.smithi046.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 315 kB/s | 29 kB 00:00 2024-03-08T22:53:03.483 INFO:teuthology.orchestra.run.smithi046.stdout:(167/174): NetworkManager-1.46.0-1.el9.x86_64.r 480 kB/s | 2.3 MB 00:04 2024-03-08T22:53:03.682 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:03.706 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:03.809 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T22:53:03.998 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T22:53:04.036 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:53:04.059 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:04.103 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:53:04.125 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:53:04.125 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:04.125 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T22:53:04.125 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:04.170 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:04.192 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:04.193 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:04.193 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T22:53:04.193 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:53:04.193 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:53:04.193 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:04.194 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:04.224 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:04.256 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T22:53:04.326 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T22:53:04.355 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:04.504 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:53:04.508 INFO:teuthology.orchestra.run.smithi046.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 947 kB/s | 1.2 MB 00:01 2024-03-08T22:53:04.526 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:53:04.527 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:04.527 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T22:53:04.527 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:53:04.527 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:53:04.527 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:04.602 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:04.631 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:53:05.646 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:53:05.673 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:53:05.780 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:53:05.841 INFO:teuthology.orchestra.run.smithi046.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 349 kB/s | 465 kB 00:01 2024-03-08T22:53:05.925 INFO:teuthology.orchestra.run.smithi046.stdout:(170/174): util-linux-user-2.37.4-18.el9.x86_64 378 kB/s | 31 kB 00:00 2024-03-08T22:53:05.952 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T22:53:05.984 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:53:06.008 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:06.059 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:53:06.070 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:53:06.117 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T22:53:06.683 INFO:teuthology.orchestra.run.smithi046.stdout:(171/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 645 kB/s | 2.2 MB 00:03 2024-03-08T22:53:07.025 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T22:53:07.053 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:53:07.077 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:07.092 INFO:teuthology.orchestra.run.smithi046.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 182 kB/s | 74 kB 00:00 2024-03-08T22:53:07.120 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:53:07.143 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:53:07.143 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:07.143 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T22:53:07.143 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:07.188 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:53:07.211 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:07.530 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T22:53:07.553 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:53:07.554 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:08.000 INFO:teuthology.orchestra.run.smithi046.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 530 kB/s | 1.1 MB 00:02 2024-03-08T22:53:08.251 INFO:teuthology.orchestra.run.smithi046.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 490 kB/s | 2.3 MB 00:04 2024-03-08T22:53:08.261 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-08T22:53:08.261 INFO:teuthology.orchestra.run.smithi046.stdout:Total 3.9 MB/s | 223 MB 00:57 2024-03-08T22:53:08.941 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:53:08.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T22:53:09.030 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T22:53:09.057 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T22:53:09.238 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T22:53:09.348 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T22:53:09.388 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T22:53:09.427 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T22:53:09.488 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T22:53:09.556 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T22:53:09.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T22:53:09.630 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T22:53:09.660 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T22:53:09.705 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T22:53:09.762 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction check 2024-03-08T22:53:09.775 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T22:53:09.816 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T22:53:10.161 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction check succeeded. 2024-03-08T22:53:10.162 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction test 2024-03-08T22:53:10.385 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:10.422 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:10.442 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:11.261 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T22:53:11.334 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T22:53:11.374 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T22:53:11.455 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T22:53:11.496 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T22:53:11.521 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T22:53:11.546 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T22:53:11.572 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T22:53:11.572 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:53:11.660 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:53:11.690 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T22:53:11.690 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:53:11.774 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:53:11.800 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T22:53:11.834 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T22:53:11.859 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T22:53:11.884 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T22:53:11.917 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T22:53:11.957 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T22:53:11.989 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T22:53:12.014 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T22:53:12.039 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T22:53:12.065 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:53:12.090 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T22:53:12.097 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:12.097 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:12.234 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:53:12.276 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:12.276 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:53:12.277 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T22:53:12.306 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T22:53:12.333 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T22:53:12.367 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:53:12.392 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:53:12.417 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T22:53:12.450 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T22:53:12.477 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T22:53:12.523 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T22:53:12.541 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:53:12.645 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T22:53:12.671 INFO:teuthology.orchestra.run.smithi046.stdout:Transaction test succeeded. 2024-03-08T22:53:12.672 INFO:teuthology.orchestra.run.smithi046.stdout:Running transaction 2024-03-08T22:53:12.682 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T22:53:12.720 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T22:53:12.781 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T22:53:12.816 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T22:53:12.848 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T22:53:12.882 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T22:53:12.913 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T22:53:12.958 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T22:53:13.010 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T22:53:13.059 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T22:53:13.603 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:13.642 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:13.662 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:53:14.556 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T22:53:14.637 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T22:53:14.679 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T22:53:14.762 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T22:53:14.790 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T22:53:14.816 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T22:53:14.841 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T22:53:14.867 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T22:53:14.868 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:53:14.957 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:53:14.993 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T22:53:14.994 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:53:15.078 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:53:15.104 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T22:53:15.137 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T22:53:15.163 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T22:53:15.187 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T22:53:15.231 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T22:53:15.269 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T22:53:15.301 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T22:53:15.326 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T22:53:15.351 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T22:53:15.377 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:53:15.402 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T22:53:15.408 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:15.408 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:53:15.597 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:53:15.671 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:53:15.696 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:53:15.721 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T22:53:15.754 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T22:53:15.780 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T22:53:15.847 INFO:teuthology.orchestra.run.smithi062.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:53:17.408 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:53:17.408 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T22:53:17.408 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T22:53:17.409 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T22:53:17.410 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T22:53:17.411 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T22:53:17.412 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T22:53:17.413 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T22:53:17.414 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T22:53:17.415 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T22:53:17.416 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T22:53:17.417 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T22:53:17.418 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T22:53:17.419 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T22:53:17.420 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:53:17.421 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T22:53:17.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T22:53:17.422 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T22:53:19.956 INFO:teuthology.orchestra.run.smithi046.stdout: Preparing : 1/1 2024-03-08T22:53:20.100 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T22:53:20.140 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T22:53:20.169 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:53:20.214 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T22:53:20.298 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T22:53:20.403 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T22:53:20.464 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T22:53:20.502 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T22:53:20.568 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.569 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.570 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.571 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T22:53:20.572 INFO:teuthology.orchestra.run.smithi192.stdout: jq-1.6-16.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.573 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:20.574 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.575 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T22:53:20.576 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T22:53:20.577 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T22:53:20.578 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T22:53:20.579 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T22:53:20.580 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T22:53:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T22:53:20.581 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:53:20.581 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T22:53:20.592 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T22:53:20.593 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T22:53:20.594 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T22:53:20.594 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T22:53:20.594 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T22:53:20.608 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T22:53:20.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T22:53:20.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T22:53:20.610 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T22:53:20.611 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T22:53:20.612 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T22:53:20.613 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T22:53:20.614 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T22:53:20.615 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T22:53:20.616 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T22:53:20.617 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T22:53:20.618 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T22:53:20.619 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T22:53:20.620 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T22:53:20.621 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T22:53:20.622 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T22:53:21.034 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T22:53:21.059 DEBUG:teuthology.parallel:result is None 2024-03-08T22:53:21.082 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:53:21.279 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:53:21.308 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:53:21.617 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:53:21.664 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:53:21.746 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T22:53:22.070 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T22:53:22.131 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T22:53:22.155 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:53:22.339 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:53:22.373 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:53:22.447 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T22:53:22.500 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T22:53:22.525 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:53:22.590 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T22:53:22.903 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:53:22.975 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T22:53:23.025 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T22:53:23.381 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T22:53:23.482 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout:Upgraded: 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T22:53:23.548 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.549 INFO:teuthology.orchestra.run.smithi062.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.550 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.551 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: jq-1.6-16.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T22:53:23.552 INFO:teuthology.orchestra.run.smithi062.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T22:53:23.553 INFO:teuthology.orchestra.run.smithi062.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T22:53:23.554 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.555 INFO:teuthology.orchestra.run.smithi062.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T22:53:23.556 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T22:53:23.557 INFO:teuthology.orchestra.run.smithi062.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.558 INFO:teuthology.orchestra.run.smithi062.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T22:53:23.559 INFO:teuthology.orchestra.run.smithi062.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T22:53:23.560 INFO:teuthology.orchestra.run.smithi062.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T22:53:23.561 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T22:53:23.561 INFO:teuthology.orchestra.run.smithi062.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T22:53:23.561 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-03-08T22:53:23.561 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-03-08T22:53:23.762 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:53:23.809 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:53:24.087 DEBUG:teuthology.parallel:result is None 2024-03-08T22:53:24.239 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T22:53:24.472 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T22:53:24.563 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T22:53:24.599 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T22:53:24.626 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T22:53:24.682 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T22:53:24.799 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T22:53:24.875 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T22:53:24.933 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T22:53:25.029 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T22:53:25.080 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T22:53:25.144 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T22:53:25.185 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T22:53:25.222 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T22:53:25.341 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T22:53:25.381 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T22:53:25.429 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T22:53:25.462 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T22:53:25.584 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T22:53:25.651 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T22:53:25.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T22:53:25.940 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T22:53:26.002 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T22:53:26.727 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T22:53:26.847 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T22:53:26.952 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T22:53:27.015 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:53:27.041 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T22:53:27.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T22:53:27.166 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T22:53:27.227 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T22:53:27.275 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T22:53:27.704 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T22:53:27.890 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T22:53:27.978 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T22:53:28.035 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T22:53:28.239 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T22:53:28.291 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T22:53:28.373 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T22:53:28.474 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T22:53:28.550 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T22:53:28.593 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T22:53:28.676 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T22:53:28.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T22:53:28.785 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T22:53:28.850 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T22:53:29.049 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T22:53:29.103 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T22:53:29.142 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T22:53:29.201 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T22:53:29.348 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T22:53:29.381 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T22:53:29.426 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T22:53:29.463 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T22:53:29.488 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T22:53:29.644 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T22:53:29.835 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T22:53:29.975 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T22:53:30.024 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T22:53:30.088 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T22:53:30.164 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T22:53:30.417 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T22:53:30.480 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T22:53:30.657 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T22:53:30.702 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T22:53:30.741 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T22:53:30.780 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T22:53:30.821 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T22:53:30.868 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T22:53:30.936 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T22:53:30.995 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T22:53:31.042 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T22:53:31.176 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T22:53:31.214 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T22:53:31.256 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T22:53:31.321 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T22:53:31.360 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T22:53:31.513 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T22:53:31.639 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T22:53:31.698 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T22:53:31.734 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T22:53:31.778 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T22:53:31.922 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T22:53:31.992 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T22:53:32.994 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T22:53:33.057 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T22:53:33.091 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T22:53:33.148 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T22:53:33.265 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T22:53:33.316 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T22:53:33.556 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T22:53:33.587 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T22:53:33.683 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T22:53:34.520 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T22:53:34.655 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T22:53:36.261 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T22:53:36.359 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T22:53:36.413 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T22:53:36.444 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T22:53:36.485 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T22:53:36.882 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T22:53:36.907 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:53:37.029 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T22:53:37.063 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T22:53:37.118 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T22:53:37.640 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T22:53:37.677 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:53:37.777 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T22:53:37.804 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T22:53:40.484 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:53:40.517 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:53:41.173 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T22:53:41.212 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:53:41.390 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T22:53:41.390 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T22:53:41.390 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T22:53:41.391 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:41.438 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:53:54.720 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T22:53:54.720 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /sys 2024-03-08T22:53:54.720 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /proc 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /mnt 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /var/tmp 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /home 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /root 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout:skipping the directory /tmp 2024-03-08T22:53:54.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:54.962 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:53:55.015 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T22:53:55.015 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:55.015 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T22:53:55.016 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:56.689 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:56.780 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T22:53:56.780 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:56.780 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T22:53:56.780 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:56.780 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T22:53:56.781 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:57.031 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:57.113 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T22:53:57.113 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:57.113 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T22:53:57.113 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:57.113 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T22:53:57.114 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:57.632 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T22:53:57.713 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:53:57.834 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T22:53:58.467 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:58.493 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T22:53:58.536 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T22:53:58.638 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T22:53:58.663 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:59.006 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T22:53:59.056 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:54:00.240 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T22:54:00.281 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:54:00.432 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T22:54:00.612 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T22:54:00.642 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T22:54:00.721 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:00.776 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:54:00.814 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T22:54:00.870 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T22:54:01.864 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T22:54:01.894 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:54:01.977 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T22:54:01.977 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:54:01.977 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T22:54:01.977 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:54:01.977 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T22:54:01.978 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:02.028 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:54:02.123 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T22:54:02.123 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:54:02.123 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T22:54:02.123 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:02.205 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:54:02.309 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:54:02.309 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:54:02.310 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T22:54:02.310 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:54:02.310 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T22:54:02.310 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:02.697 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T22:54:02.784 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:07.820 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:54:07.872 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T22:54:07.909 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T22:54:07.946 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T22:54:08.237 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T22:54:08.368 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T22:54:08.421 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T22:54:08.481 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T22:54:08.561 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T22:54:08.607 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T22:54:08.675 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T22:54:08.719 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T22:54:08.771 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T22:54:08.861 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T22:54:08.972 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T22:54:09.003 INFO:teuthology.orchestra.run.smithi046.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T22:54:09.924 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:54:09.983 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:54:10.049 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T22:54:11.479 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T22:54:11.588 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T22:54:11.624 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T22:54:11.764 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T22:54:11.809 INFO:teuthology.orchestra.run.smithi046.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T22:54:11.840 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T22:54:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T22:54:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T22:54:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:54:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T22:54:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T22:54:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:54:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T22:54:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T22:54:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T22:54:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T22:54:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T22:54:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T22:54:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T22:54:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T22:54:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T22:54:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T22:54:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:54:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T22:54:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:54:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:54:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T22:54:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:54:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T22:54:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:54:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T22:54:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T22:54:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T22:54:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:54:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T22:54:13.855 INFO:teuthology.orchestra.run.smithi046.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-08T22:54:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T22:54:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T22:54:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T22:54:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T22:54:18.710 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T22:54:18.711 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T22:54:18.712 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T22:54:18.729 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T22:54:18.729 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T22:54:18.729 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T22:54:18.730 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T22:54:18.731 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T22:54:18.732 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T22:54:18.733 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T22:54:18.734 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T22:54:18.735 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T22:54:18.736 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T22:54:18.737 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T22:54:18.738 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T22:54:18.739 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T22:54:18.740 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T22:54:18.741 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T22:54:18.742 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T22:54:18.743 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout:Upgraded: 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T22:54:21.219 INFO:teuthology.orchestra.run.smithi046.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.220 INFO:teuthology.orchestra.run.smithi046.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.221 INFO:teuthology.orchestra.run.smithi046.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout:Installed: 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T22:54:21.222 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.223 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T22:54:21.224 INFO:teuthology.orchestra.run.smithi046.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: jq-1.6-16.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.225 INFO:teuthology.orchestra.run.smithi046.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.226 INFO:teuthology.orchestra.run.smithi046.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T22:54:21.227 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T22:54:21.228 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T22:54:21.229 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T22:54:21.230 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T22:54:21.231 INFO:teuthology.orchestra.run.smithi046.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T22:54:21.232 INFO:teuthology.orchestra.run.smithi046.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T22:54:21.233 INFO:teuthology.orchestra.run.smithi046.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T22:54:21.234 INFO:teuthology.orchestra.run.smithi046.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T22:54:21.234 INFO:teuthology.orchestra.run.smithi046.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T22:54:21.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:54:21.234 INFO:teuthology.orchestra.run.smithi046.stdout:Complete! 2024-03-08T22:54:21.922 DEBUG:teuthology.parallel:result is None 2024-03-08T22:54:21.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:22.145 DEBUG:teuthology.orchestra.run.smithi046:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T22:54:22.206 INFO:teuthology.orchestra.run.smithi046.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.206 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.206 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T22:54:22.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:22.345 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T22:54:22.372 INFO:teuthology.orchestra.run.smithi062.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.372 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.372 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T22:54:22.374 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:22.512 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T22:54:22.538 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.538 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T22:54:22.538 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T22:54:22.540 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-08T22:54:22.540 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:54:22.540 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T22:54:22.620 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:54:22.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T22:54:22.660 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:54:22.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T22:54:22.695 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-08T22:54:22.695 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:54:22.696 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T22:54:22.773 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T22:54:22.892 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:54:22.892 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T22:54:22.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T22:54:23.002 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:54:23.002 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T22:54:23.034 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T22:54:23.102 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-08T22:54:23.103 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:54:23.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T22:54:23.186 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T22:54:23.303 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:54:23.303 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T22:54:23.334 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T22:54:23.406 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:54:23.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T22:54:23.433 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T22:54:23.501 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-08T22:54:23.502 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:54:23.502 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T22:54:23.581 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T22:54:23.703 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:54:23.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T22:54:23.734 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T22:54:23.804 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:54:23.804 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T22:54:23.830 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T22:54:23.896 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-08T22:54:23.988 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-08T22:54:23.988 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:23.988 INFO:tasks.cephadm:Cluster fsid is ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:54:23.989 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-08T22:54:23.989 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.46', 'mon.b': '172.21.15.62', 'mon.c': '172.21.15.192'} 2024-03-08T22:54:23.989 INFO:tasks.cephadm:First mon is mon.a on smithi046 2024-03-08T22:54:23.989 INFO:tasks.cephadm:First mgr is x 2024-03-08T22:54:23.989 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-08T22:54:23.989 DEBUG:teuthology.orchestra.run.smithi046:> sudo hostname $(hostname -s) 2024-03-08T22:54:24.069 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2024-03-08T22:54:24.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-03-08T22:54:24.127 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-08T22:54:24.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:24.270 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77443/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1965-g7d63482a', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1965.g7d63482a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 17:09:11.137784', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'rishabh-2024mar8', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-08T22:54:24.490 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref rishabh-2024mar8, sha1 7d63482aa89628d733e298984a762389b625c761 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:54:24.492 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm 2024-03-08T22:54:24.492 DEBUG:teuthology.orchestra.run.smithi046:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:24.987 INFO:teuthology.orchestra.run.smithi046.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 22:54 /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:24.987 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:25.510 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 22:54 /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:25.510 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:25.981 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 22:54 /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:25.982 DEBUG:teuthology.orchestra.run.smithi046:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:26.026 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:26.047 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T22:54:26.088 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 on all hosts... 2024-03-08T22:54:26.089 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T22:54:26.093 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T22:54:26.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T22:54:26.846 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T22:54:26.939 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T22:54:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T22:54:59.709 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-03-08T22:54:59.710 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-08T22:55:01.177 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-08T22:55:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T22:55:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T22:55:01.177 INFO:teuthology.orchestra.run.smithi062.stdout: "repo_digests": [ 2024-03-08T22:55:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T22:55:01.178 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-03-08T22:55:01.178 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-08T22:55:19.150 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T22:55:19.150 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T22:55:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T22:55:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "repo_digests": [ 2024-03-08T22:55:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T22:55:19.151 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T22:55:19.151 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T22:55:19.218 DEBUG:teuthology.orchestra.run.smithi046:> sudo mkdir -p /etc/ceph 2024-03-08T22:55:19.324 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2024-03-08T22:55:19.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-03-08T22:55:19.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 777 /etc/ceph 2024-03-08T22:55:19.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2024-03-08T22:55:19.540 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-03-08T22:55:19.572 INFO:tasks.cephadm:Writing seed config... 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-08T22:55:19.573 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-08T22:55:19.574 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-08T22:55:19.575 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-08T22:55:19.576 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-08T22:55:19.577 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:55:19.577 DEBUG:teuthology.orchestra.run.smithi046:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-08T22:55:19.636 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ced7dd96-dd9e-11ee-95c6-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-08T22:55:19.637 DEBUG:teuthology.orchestra.run.smithi046:mon.a> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service 2024-03-08T22:55:19.684 DEBUG:teuthology.orchestra.run.smithi046:mgr.x> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x.service 2024-03-08T22:55:19.729 INFO:tasks.cephadm:Bootstrapping... 2024-03-08T22:55:19.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 -v bootstrap --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.46 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-08T22:55:20.027 INFO:teuthology.orchestra.run.smithi046.stdout:-------------------------------------------------------------------------------- 2024-03-08T22:55:20.027 INFO:teuthology.orchestra.run.smithi046.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761', '-v', 'bootstrap', '--fsid', 'ced7dd96-dd9e-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.46', '--skip-admin-label'] 2024-03-08T22:55:20.085 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T22:55:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-08T22:55:20.086 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying podman|docker is present... 2024-03-08T22:55:20.143 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T22:55:20.143 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying lvm2 is present... 2024-03-08T22:55:20.143 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying time synchronization is in place... 2024-03-08T22:55:20.166 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-08T22:55:20.166 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-08T22:55:20.189 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-08T22:55:20.189 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:55:20.215 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-03-08T22:55:20.238 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-03-08T22:55:20.238 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-03-08T22:55:20.238 INFO:teuthology.orchestra.run.smithi046.stdout:Repeating the final host check... 2024-03-08T22:55:20.304 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T22:55:20.305 INFO:teuthology.orchestra.run.smithi046.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-08T22:55:20.306 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl is present 2024-03-08T22:55:20.306 INFO:teuthology.orchestra.run.smithi046.stdout:lvcreate is present 2024-03-08T22:55:20.329 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-08T22:55:20.329 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-08T22:55:20.352 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-08T22:55:20.352 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:55:20.378 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout enabled 2024-03-08T22:55:20.401 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout active 2024-03-08T22:55:20.402 INFO:teuthology.orchestra.run.smithi046.stdout:Unit chronyd.service is enabled and running 2024-03-08T22:55:20.402 INFO:teuthology.orchestra.run.smithi046.stdout:Host looks OK 2024-03-08T22:55:20.402 INFO:teuthology.orchestra.run.smithi046.stdout:Cluster fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:20.403 INFO:teuthology.orchestra.run.smithi046.stdout:Acquiring lock 140023169478080 on /run/cephadm/ced7dd96-dd9e-11ee-95c6-87774f69a715.lock 2024-03-08T22:55:20.403 INFO:teuthology.orchestra.run.smithi046.stdout:Lock 140023169478080 acquired on /run/cephadm/ced7dd96-dd9e-11ee-95c6-87774f69a715.lock 2024-03-08T22:55:20.404 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 3300 ... 2024-03-08T22:55:20.405 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying IP 172.21.15.46 port 6789 ... 2024-03-08T22:55:20.405 INFO:teuthology.orchestra.run.smithi046.stdout:Base mon IP(s) is [172.21.15.46:3300, 172.21.15.46:6789], mon addrv is [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-03-08T22:55:20.422 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.46 metric 100 2024-03-08T22:55:20.422 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.46 metric 100 2024-03-08T22:55:20.434 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-08T22:55:20.434 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-08T22:55:20.435 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-08T22:55:20.447 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-08T22:55:20.447 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-08T22:55:20.447 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-08T22:55:20.448 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-08T22:55:20.448 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-08T22:55:20.448 INFO:teuthology.orchestra.run.smithi046.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-08T22:55:20.450 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-03-08T22:55:20.450 INFO:teuthology.orchestra.run.smithi046.stdout:Mon IP `172.21.15.46` is in CIDR network `172.21.0.0/20` 2024-03-08T22:55:20.450 INFO:teuthology.orchestra.run.smithi046.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-08T22:55:20.453 INFO:teuthology.orchestra.run.smithi046.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-08T22:55:20.456 INFO:teuthology.orchestra.run.smithi046.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T22:55:21.069 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stdout a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 2024-03-08T22:55:21.069 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T22:55:21.070 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Getting image source signatures 2024-03-08T22:55:21.070 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:7a785ebef89f89c1f7e1e5df571ab58da14c499794e7964e586765517f59740d 2024-03-08T22:55:21.070 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-08T22:55:21.071 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Copying config sha256:a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 2024-03-08T22:55:21.071 INFO:teuthology.orchestra.run.smithi046.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-08T22:55:22.340 INFO:teuthology.orchestra.run.smithi046.stdout:ceph: stdout ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev) 2024-03-08T22:55:22.340 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph version: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev) 2024-03-08T22:55:22.340 INFO:teuthology.orchestra.run.smithi046.stdout:Extracting ceph user uid/gid from container image... 2024-03-08T22:55:23.399 INFO:teuthology.orchestra.run.smithi046.stdout:stat: stdout 167 167 2024-03-08T22:55:23.400 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial keys... 2024-03-08T22:55:25.433 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQBcl+tlWu2sLRAArBs/8LNvpqHdu7Su68IT2Q== 2024-03-08T22:55:26.490 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQBdl+tlQF9yMxAAF3Gma8aBzbnkO9vG0mNwhw== 2024-03-08T22:55:27.571 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph-authtool: stdout AQBel+tlIpapNhAAbmTFhXG5OdzEAA4/WB8pkg== 2024-03-08T22:55:27.573 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial monmap... 2024-03-08T22:55:28.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-08T22:55:28.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-08T22:55:28.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:28.608 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:monmaptool for a [v2:172.21.15.46:3300,v1:172.21.15.46:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:setting min_mon_release = quincy 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: set fsid to ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:55:28.609 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mon... 2024-03-08T22:55:29.726 INFO:teuthology.orchestra.run.smithi046.stdout:create mon.a on 2024-03-08T22:55:30.126 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-08T22:55:30.368 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-08T22:55:30.670 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target. 2024-03-08T22:55:30.670 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target. 2024-03-08T22:55:31.082 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a 2024-03-08T22:55:31.082 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-08T22:55:31.362 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target.wants/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@.service. 2024-03-08T22:55:32.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:31 smithi046 podman[31474]: 2024-03-08 22:55:31.814554586 +0000 UTC m=+0.182338546 container create 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T22:55:32.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 podman[31474]: 2024-03-08 22:55:32.050717293 +0000 UTC m=+0.418501260 container init 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-08T22:55:32.164 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T22:55:32.165 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-08T22:55:32.180 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T22:55:32.180 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:55:32.180 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-08T22:55:32.181 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-03-08T22:55:32.181 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon to start... 2024-03-08T22:55:32.182 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mon... 2024-03-08T22:55:32.331 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 podman[31474]: 2024-03-08 22:55:32.062574233 +0000 UTC m=+0.430358200 container start 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2) 2024-03-08T22:55:32.331 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 bash[31474]: 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb 2024-03-08T22:55:32.331 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 systemd[1]: Started Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:55:32.332 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 ceph-mon[31492]: mkfs ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:32.332 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:32 smithi046 ceph-mon[31492]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-08T22:55:33.426 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-08T22:55:33.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: monmap e1: 1 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:55:33.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: fsmap 2024-03-08T22:55:33.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: osdmap e1: 0 total, 0 up, 0 in 2024-03-08T22:55:33.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: mgrmap e1: no daemons active 2024-03-08T22:55:33.427 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:33 smithi046 ceph-mon[31492]: from='client.? 172.21.15.46:0/990402417' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-08T22:55:33.439 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout cluster: 2024-03-08T22:55:33.445 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout id: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:33.445 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout services: 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.729933s) 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout data: 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout pgs: 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.775+0000 7f03a6a00700 1 Processor -- start 2024-03-08T22:55:33.446 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.775+0000 7f03a6a00700 1 -- start start 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.776+0000 7f03a6a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.776+0000 7f03a6a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a010eb30 con 0x7f03a010e140 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.777+0000 7f039fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.777+0000 7f039fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57122/0 (socket says 172.21.15.46:57122) 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.777+0000 7f039fe00700 1 -- 172.21.15.46:0/356689977 learned_addr learned my addr 172.21.15.46:0/356689977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.777+0000 7f039fe00700 1 -- 172.21.15.46:0/356689977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a010ef70 con 0x7f03a010e140 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f039fe00700 1 --2- 172.21.15.46:0/356689977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03900098a0 tx=0x7f0390009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4c20dfacd5e985 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f039ea00700 1 -- 172.21.15.46:0/356689977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0390004160 con 0x7f03a010e140 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f039ea00700 1 -- 172.21.15.46:0/356689977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0390004300 con 0x7f03a010e140 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f039ea00700 1 -- 172.21.15.46:0/356689977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f03900036a0 con 0x7f03a010e140 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f03a6a00700 1 -- 172.21.15.46:0/356689977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 msgr2=0x7f03a010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.778+0000 7f03a6a00700 1 --2- 172.21.15.46:0/356689977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03900098a0 tx=0x7f0390009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.779+0000 7f03a6a00700 1 -- 172.21.15.46:0/356689977 shutdown_connections 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.779+0000 7f03a6a00700 1 --2- 172.21.15.46:0/356689977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a010e560 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:33.447 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.779+0000 7f03a6a00700 1 -- 172.21.15.46:0/356689977 >> 172.21.15.46:0/356689977 conn(0x7f03a007ceb0 msgr2=0x7f03a007f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.779+0000 7f03a6a00700 1 -- 172.21.15.46:0/356689977 shutdown_connections 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.779+0000 7f03a6a00700 1 -- 172.21.15.46:0/356689977 wait complete. 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.780+0000 7f03a6a00700 1 Processor -- start 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.780+0000 7f03a6a00700 1 -- start start 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.780+0000 7f03a6a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.780+0000 7f03a6a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a010f4a0 con 0x7f03a010e140 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.781+0000 7f039fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.781+0000 7f039fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57130/0 (socket says 172.21.15.46:57130) 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.781+0000 7f039fe00700 1 -- 172.21.15.46:0/990402417 learned_addr learned my addr 172.21.15.46:0/990402417 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.781+0000 7f039fe00700 1 -- 172.21.15.46:0/990402417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a01a1950 con 0x7f03a010e140 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.781+0000 7f039fe00700 1 --2- 172.21.15.46:0/990402417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0390000c00 tx=0x7f0390004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0390003db0 con 0x7f03a010e140 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0390011630 con 0x7f03a010e140 2024-03-08T22:55:33.448 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a01a1be0 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0390011950 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a01a48d0 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f039001a460 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.782+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f039001b780 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.783+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a0063410 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.786+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f039001b980 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.868+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f03a002da80 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.868+0000 7f039cc00700 1 -- 172.21.15.46:0/990402417 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f039001bb60 con 0x7f03a010e140 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 msgr2=0x7f03a01a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 --2- 172.21.15.46:0/990402417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0390000c00 tx=0x7f0390004750 comp rx=0 tx=0).stop 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 shutdown_connections 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 --2- 172.21.15.46:0/990402417 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03a010e140 0x7f03a01a1410 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 >> 172.21.15.46:0/990402417 conn(0x7f03a007ceb0 msgr2=0x7f03a0198120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:33.449 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 shutdown_connections 2024-03-08T22:55:33.450 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:32.870+0000 7f03a6a00700 1 -- 172.21.15.46:0/990402417 wait complete. 2024-03-08T22:55:33.450 INFO:teuthology.orchestra.run.smithi046.stdout:mon is available 2024-03-08T22:55:33.450 INFO:teuthology.orchestra.run.smithi046.stdout:Assimilating anything we can from ceph.conf... 2024-03-08T22:55:34.443 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:34 smithi046 ceph-mon[31492]: from='client.? 172.21.15.46:0/3658789926' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-08T22:55:34.443 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:34 smithi046 ceph-mon[31492]: from='client.? 172.21.15.46:0/3658789926' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.46:3300,v1:172.21.15.46:6789] 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-08T22:55:34.725 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.018+0000 7f89a6400700 1 Processor -- start 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.019+0000 7f89a6400700 1 -- start start 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.019+0000 7f89a6400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.019+0000 7f89a6400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a007b890 con 0x7f89a007cf10 2024-03-08T22:55:34.726 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.020+0000 7f899f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.020+0000 7f899f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50668/0 (socket says 172.21.15.46:50668) 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.020+0000 7f899f400700 1 -- 172.21.15.46:0/244455007 learned_addr learned my addr 172.21.15.46:0/244455007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.020+0000 7f899f400700 1 -- 172.21.15.46:0/244455007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a007ba10 con 0x7f89a007cf10 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.021+0000 7f899f400700 1 --2- 172.21.15.46:0/244455007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8990009e80 tx=0x7f899000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40833d56ce544746 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.021+0000 7f899ea00700 1 -- 172.21.15.46:0/244455007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8990017070 con 0x7f89a007cf10 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.021+0000 7f899ea00700 1 -- 172.21.15.46:0/244455007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990003a20 con 0x7f89a007cf10 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.021+0000 7f899ea00700 1 -- 172.21.15.46:0/244455007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8990003d20 con 0x7f89a007cf10 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 -- 172.21.15.46:0/244455007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 msgr2=0x7f89a007b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 --2- 172.21.15.46:0/244455007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f8990009e80 tx=0x7f899000b330 comp rx=0 tx=0).stop 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 -- 172.21.15.46:0/244455007 shutdown_connections 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 --2- 172.21.15.46:0/244455007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a007b2c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 -- 172.21.15.46:0/244455007 >> 172.21.15.46:0/244455007 conn(0x7f89a0102d80 msgr2=0x7f89a01051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:34.727 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 -- 172.21.15.46:0/244455007 shutdown_connections 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.022+0000 7f89a6400700 1 -- 172.21.15.46:0/244455007 wait complete. 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.023+0000 7f89a6400700 1 Processor -- start 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.023+0000 7f89a6400700 1 -- start start 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f89a6400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f89a6400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a007bf40 con 0x7f89a007cf10 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f899f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f899f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50672/0 (socket says 172.21.15.46:50672) 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f899f400700 1 -- 172.21.15.46:0/3658789926 learned_addr learned my addr 172.21.15.46:0/3658789926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.024+0000 7f899f400700 1 -- 172.21.15.46:0/3658789926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a019d720 con 0x7f89a007cf10 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f899f400700 1 --2- 172.21.15.46:0/3658789926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f89a007c330 tx=0x7f8990004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8990022020 con 0x7f89a007cf10 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990018040 con 0x7f89a007cf10 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a019d9b0 con 0x7f89a007cf10 2024-03-08T22:55:34.728 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8990017070 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a01a06a0 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.025+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8990025020 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.026+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f899000f050 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.026+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a0063410 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.029+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f899002c030 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.108+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f89a002d8d0 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.114+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f899001e480 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f899cc00700 1 -- 172.21.15.46:0/3658789926 <== mon.0 v2:172.21.15.46:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8990035070 con 0x7f89a007cf10 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 msgr2=0x7f89a019d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 --2- 172.21.15.46:0/3658789926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f89a007c330 tx=0x7f8990004ad0 comp rx=0 tx=0).stop 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 shutdown_connections 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 --2- 172.21.15.46:0/3658789926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f89a007cf10 0x7f89a019d1e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 >> 172.21.15.46:0/3658789926 conn(0x7f89a0102d80 msgr2=0x7f89a01031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:34.729 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 shutdown_connections 2024-03-08T22:55:34.730 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:34.116+0000 7f89a6400700 1 -- 172.21.15.46:0/3658789926 wait complete. 2024-03-08T22:55:34.730 INFO:teuthology.orchestra.run.smithi046.stdout:Generating new minimal ceph.conf... 2024-03-08T22:55:35.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:35 smithi046 ceph-mon[31492]: from='client.? 172.21.15.46:0/2464823840' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:55:36.055 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.301+0000 7f9a91200700 1 Processor -- start 2024-03-08T22:55:36.055 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.301+0000 7f9a91200700 1 -- start start 2024-03-08T22:55:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.302+0000 7f9a91200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.302+0000 7f9a91200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c07adf0 con 0x7f9a8c108310 2024-03-08T22:55:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.302+0000 7f9a89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:36.056 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.302+0000 7f9a89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50682/0 (socket says 172.21.15.46:50682) 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.302+0000 7f9a89e00700 1 -- 172.21.15.46:0/98303507 learned_addr learned my addr 172.21.15.46:0/98303507 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.303+0000 7f9a89e00700 1 -- 172.21.15.46:0/98303507 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c10ac40 con 0x7f9a8c108310 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.303+0000 7f9a89e00700 1 --2- 172.21.15.46:0/98303507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098a0 tx=0x7f9a7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b1cc7383043edb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.303+0000 7f9a89400700 1 -- 172.21.15.46:0/98303507 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a7c004160 con 0x7f9a8c108310 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.303+0000 7f9a89400700 1 -- 172.21.15.46:0/98303507 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a7c004300 con 0x7f9a8c108310 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.303+0000 7f9a89400700 1 -- 172.21.15.46:0/98303507 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a7c011b50 con 0x7f9a8c108310 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 -- 172.21.15.46:0/98303507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 msgr2=0x7f9a8c10a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 --2- 172.21.15.46:0/98303507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0098a0 tx=0x7f9a7c009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 -- 172.21.15.46:0/98303507 shutdown_connections 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 --2- 172.21.15.46:0/98303507 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c10a700 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:36.061 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 -- 172.21.15.46:0/98303507 >> 172.21.15.46:0/98303507 conn(0x7f9a8c101ce0 msgr2=0x7f9a8c104100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.304+0000 7f9a91200700 1 -- 172.21.15.46:0/98303507 shutdown_connections 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.305+0000 7f9a91200700 1 -- 172.21.15.46:0/98303507 wait complete. 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a91200700 1 Processor -- start 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a91200700 1 -- start start 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a91200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a91200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c10bb40 con 0x7f9a8c108310 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50684/0 (socket says 172.21.15.46:50684) 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.306+0000 7f9a89e00700 1 -- 172.21.15.46:0/2464823840 learned_addr learned my addr 172.21.15.46:0/2464823840 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.307+0000 7f9a89e00700 1 -- 172.21.15.46:0/2464823840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c19de50 con 0x7f9a8c108310 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.307+0000 7f9a89e00700 1 --2- 172.21.15.46:0/2464823840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011670 tx=0x7f9a7c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:36.062 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.307+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a7c0039c0 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8c19e0e0 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a7c011e50 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9a7c01a5c0 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8c1a0dd0 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9a7c01ac00 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.308+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9a7c019700 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.309+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8c063410 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.312+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9a7c003b60 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.391+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9a8c1a0fa0 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.392+0000 7f9a82a00700 1 -- 172.21.15.46:0/2464823840 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9a7c019900 con 0x7f9a8c108310 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 msgr2=0x7f9a8c19d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 --2- 172.21.15.46:0/2464823840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c011670 tx=0x7f9a7c011b60 comp rx=0 tx=0).stop 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 shutdown_connections 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 --2- 172.21.15.46:0/2464823840 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a8c108310 0x7f9a8c19d910 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:36.063 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 >> 172.21.15.46:0/2464823840 conn(0x7f9a8c101ce0 msgr2=0x7f9a8c102730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:36.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 shutdown_connections 2024-03-08T22:55:36.064 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:35.393+0000 7f9a91200700 1 -- 172.21.15.46:0/2464823840 wait complete. 2024-03-08T22:55:36.064 INFO:teuthology.orchestra.run.smithi046.stdout:Restarting the monitor... 2024-03-08T22:55:36.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:36 smithi046 systemd[1]: Stopping Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:55:36.877 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:36 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a[31488]: 2024-03-08T22:55:36.603+0000 7fc106600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T22:55:36.877 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:36 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a[31488]: 2024-03-08T22:55:36.603+0000 7fc106600700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-08T22:55:37.190 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:36 smithi046 podman[31735]: 2024-03-08 22:55:36.872880897 +0000 UTC m=+0.444750811 container died 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-08T22:55:37.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:37 smithi046 podman[31735]: 2024-03-08 22:55:37.296394364 +0000 UTC m=+0.868264305 container cleanup 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-08T22:55:37.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:37 smithi046 bash[31735]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a 2024-03-08T22:55:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:37 smithi046 podman[31749]: 2024-03-08 22:55:37.555654562 +0000 UTC m=+0.660682691 container remove 789429cb78e52621cdb3092617dca24743008e0fc9534ac6b2b890e653f866eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T22:55:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-08T22:55:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 systemd[1]: Stopped Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:55:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service: Consumed 2.065s CPU time. 2024-03-08T22:55:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 systemd[1]: Starting Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:55:38.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 podman[31841]: 2024-03-08 22:55:38.447591879 +0000 UTC m=+0.033381890 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:55:38.799 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 podman[31841]: 2024-03-08 22:55:38.5649686 +0000 UTC m=+0.150758641 container create 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-08T22:55:38.799 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 podman[31841]: 2024-03-08 22:55:38.785210368 +0000 UTC m=+0.371000416 container init 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-08T22:55:38.910 INFO:teuthology.orchestra.run.smithi046.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 podman[31841]: 2024-03-08 22:55:38.805327899 +0000 UTC m=+0.391117947 container start 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: set uid:gid to 167:167 (ceph:ceph) 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev), process ceph-mon, pid 2 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: pidfile_write: ignore empty --pid-file 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: load: jerasure load: lrc 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: RocksDB version: 7.9.2 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Git sha 0 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Compile date 2024-03-08 15:06:51 2024-03-08T22:55:39.058 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: DB SUMMARY 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: DB Session ID: 6AMPEO5ILGG86ZTNJEIY 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: CURRENT file: CURRENT 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: IDENTITY file: IDENTITY 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100298 ; 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.error_if_exists: 0 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.create_if_missing: 0 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.paranoid_checks: 1 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-08T22:55:39.059 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.env: 0x55fe8ac17b20 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.fs: PosixFileSystem 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.info_log: 0x55fe8ceed920 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_file_opening_threads: 16 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.statistics: (nil) 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.use_fsync: 0 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_log_file_size: 0 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.keep_log_file_num: 1000 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.recycle_log_file_num: 0 2024-03-08T22:55:39.060 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_fallocate: 1 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_mmap_reads: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_mmap_writes: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.use_direct_reads: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.create_missing_column_families: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.db_log_dir: 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.wal_dir: 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-08T22:55:39.061 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.advise_random_on_open: 1 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.db_write_buffer_size: 0 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.write_buffer_manager: 0x55fe8cec1400 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-08T22:55:39.062 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.rate_limiter: (nil) 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.wal_recovery_mode: 2 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enable_thread_tracking: 0 2024-03-08T22:55:39.063 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enable_pipelined_write: 0 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.unordered_write: 0 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-08T22:55:39.064 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.row_cache: None 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.wal_filter: None 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_ingest_behind: 0 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.two_write_queues: 0 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.manual_wal_flush: 0 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.wal_compression: 0 2024-03-08T22:55:39.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.atomic_flush: 0 2024-03-08T22:55:39.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-08T22:55:39.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-08T22:55:39.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-08T22:55:39.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.log_readahead_size: 0 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.best_efforts_recovery: 0 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.allow_data_in_errors: 0 2024-03-08T22:55:39.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.db_host_id: __hostname__ 2024-03-08T22:55:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-08T22:55:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_background_jobs: 2 2024-03-08T22:55:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_background_compactions: -1 2024-03-08T22:55:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_subcompactions: 1 2024-03-08T22:55:39.069 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_total_wal_size: 0 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-08T22:55:39.070 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_open_files: -1 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bytes_per_sync: 0 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-08T22:55:39.071 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_readahead_size: 0 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_background_flushes: -1 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Compression algorithms supported: 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kZSTD supported: 0 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kXpressCompression supported: 0 2024-03-08T22:55:39.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kLZ4HCCompression supported: 1 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kZlibCompression supported: 1 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kSnappyCompression supported: 1 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kLZ4Compression supported: 1 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: kBZip2Compression supported: 0 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-08T22:55:39.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-08T22:55:39.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-08T22:55:39.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-08T22:55:39.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-08T22:55:39.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.merge_operator: 2024-03-08T22:55:39.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_filter: None 2024-03-08T22:55:39.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_filter_factory: None 2024-03-08T22:55:39.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.sst_partitioner_factory: None 2024-03-08T22:55:39.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-08T22:55:39.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-08T22:55:39.075 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fe8ceeda20) 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks: 1 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: pin_top_level_index_and_filter: 1 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_type: 0 2024-03-08T22:55:39.076 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_index_type: 0 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_shortening: 1 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: checksum: 4 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: no_block_cache: 0 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache: 0x55fe8cef3090 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_name: BinnedLRUCache 2024-03-08T22:55:39.077 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_options: 2024-03-08T22:55:39.078 INFO:journalctl@ceph.mon.a.smithi046.stdout: capacity : 536870912 2024-03-08T22:55:39.078 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_shard_bits : 4 2024-03-08T22:55:39.078 INFO:journalctl@ceph.mon.a.smithi046.stdout: strict_capacity_limit : 0 2024-03-08T22:55:39.078 INFO:journalctl@ceph.mon.a.smithi046.stdout: high_pri_pool_ratio: 0.000 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_cache_compressed: (nil) 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: persistent_cache: (nil) 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size: 4096 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_size_deviation: 10 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_restart_interval: 16 2024-03-08T22:55:39.082 INFO:journalctl@ceph.mon.a.smithi046.stdout: index_block_restart_interval: 1 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: metadata_block_size: 4096 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: partition_filters: 0 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: use_delta_encoding: 1 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: filter_policy: bloomfilter 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: whole_key_filtering: 1 2024-03-08T22:55:39.083 INFO:journalctl@ceph.mon.a.smithi046.stdout: verify_compression: 0 2024-03-08T22:55:39.084 INFO:journalctl@ceph.mon.a.smithi046.stdout: read_amp_bytes_per_bit: 0 2024-03-08T22:55:39.084 INFO:journalctl@ceph.mon.a.smithi046.stdout: format_version: 5 2024-03-08T22:55:39.084 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_index_compression: 1 2024-03-08T22:55:39.084 INFO:journalctl@ceph.mon.a.smithi046.stdout: block_align: 0 2024-03-08T22:55:39.084 INFO:journalctl@ceph.mon.a.smithi046.stdout: max_auto_readahead_size: 262144 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout: prepopulate_block_cache: 0 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout: initial_auto_readahead_size: 8192 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout: num_file_reads_for_auto_readahead: 2 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.write_buffer_size: 33554432 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_write_buffer_number: 2 2024-03-08T22:55:39.085 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression: NoCompression 2024-03-08T22:55:39.086 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression: Disabled 2024-03-08T22:55:39.086 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.prefix_extractor: nullptr 2024-03-08T22:55:39.086 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-08T22:55:39.086 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.num_levels: 7 2024-03-08T22:55:39.086 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-08T22:55:39.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-08T22:55:39.088 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.level: 32767 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.strategy: 0 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-08T22:55:39.089 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.enabled: false 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-08T22:55:39.090 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.target_file_size_base: 67108864 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-08T22:55:39.091 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-08T22:55:39.092 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.arena_block_size: 1048576 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-08T22:55:39.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.disable_auto_compactions: 0 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-08T22:55:39.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.table_properties_collectors: 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.inplace_update_support: 0 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-08T22:55:39.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.bloom_locality: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.max_successive_merges: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.paranoid_file_checks: 0 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.force_consistency_checks: 1 2024-03-08T22:55:39.096 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.report_bg_io_stats: 0 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.ttl: 2592000 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enable_blob_files: false 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.min_blob_size: 0 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_file_size: 268435456 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-08T22:55:39.097 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-08T22:55:39.098 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-08T22:55:39.098 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-08T22:55:39.098 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.blob_file_starting_level: 0 2024-03-08T22:55:39.098 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 71a7f490-d4ad-452d-adb2-b97ad9db468e 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938538864714, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938538866343, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95419, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93405, "index_size": 296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13904, "raw_average_key_size": 50, "raw_value_size": 85865, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709938538, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "71a7f490-d4ad-452d-adb2-b97ad9db468e", "db_session_id": "6AMPEO5ILGG86ZTNJEIY", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938538866542, "job": 1, "event": "recovery_finished"} 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fe8cfe6000 2024-03-08T22:55:39.099 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: DB pointer 0x55fe8cfd0000 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** DB Stats ** 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-03-08T22:55:39.100 INFO:journalctl@ceph.mon.a.smithi046.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: L0 2/0 95.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Sum 2/0 95.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 85.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** Compaction Stats [default] ** 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 85.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T22:55:39.101 INFO:journalctl@ceph.mon.a.smithi046.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: Cumulative compaction: 0.00 GB write, 13.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: Interval compaction: 0.00 GB write, 13.34 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache BinnedLRUCache@0x55fe8cef3090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.102 INFO:journalctl@ceph.mon.a.smithi046.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: starting mon.a rank 0 at public addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] at bind addrs [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???) e1 preinit fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).mds e1 new map 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).mds e1 print_map 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: e1 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: legacy client fscid: -1 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout: No filesystems configured 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:55:39.103 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 bash[31841]: 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 systemd[1]: Started Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: monmap e1: 1 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: fsmap 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: osdmap e1: 0 total, 0 up, 0 in 2024-03-08T22:55:39.104 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:38 smithi046 ceph-mon[31859]: mgrmap e1: no daemons active 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.557+0000 7f9bfc200700 1 Processor -- start 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.558+0000 7f9bfc200700 1 -- start start 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.558+0000 7f9bfc200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.558+0000 7f9bfc200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf41062d0 con 0x7f9bf4107d90 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.559+0000 7f9bf9800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.559+0000 7f9bf9800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50688/0 (socket says 172.21.15.46:50688) 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.559+0000 7f9bf9800700 1 -- 172.21.15.46:0/489215416 learned_addr learned my addr 172.21.15.46:0/489215416 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.560+0000 7f9bf9800700 1 -- 172.21.15.46:0/489215416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4106450 con 0x7f9bf4107d90 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.560+0000 7f9bf9800700 1 --2- 172.21.15.46:0/489215416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098a0 tx=0x7f9be8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb590521f5973ab server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:40.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.560+0000 7f9bf3e00700 1 -- 172.21.15.46:0/489215416 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9be8004160 con 0x7f9bf4107d90 2024-03-08T22:55:40.330 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.560+0000 7f9bf3e00700 1 -- 172.21.15.46:0/489215416 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be8004300 con 0x7f9bf4107d90 2024-03-08T22:55:40.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.560+0000 7f9bf3e00700 1 -- 172.21.15.46:0/489215416 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9be8011b50 con 0x7f9bf4107d90 2024-03-08T22:55:40.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.561+0000 7f9bfc200700 1 -- 172.21.15.46:0/489215416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 msgr2=0x7f9bf41081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:40.333 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.561+0000 7f9bfc200700 1 --2- 172.21.15.46:0/489215416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f9be80098a0 tx=0x7f9be8009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.562+0000 7f9bfc200700 1 -- 172.21.15.46:0/489215416 shutdown_connections 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.562+0000 7f9bfc200700 1 --2- 172.21.15.46:0/489215416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf41081b0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.562+0000 7f9bfc200700 1 -- 172.21.15.46:0/489215416 >> 172.21.15.46:0/489215416 conn(0x7f9bf4101c70 msgr2=0x7f9bf4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.562+0000 7f9bfc200700 1 -- 172.21.15.46:0/489215416 shutdown_connections 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.562+0000 7f9bfc200700 1 -- 172.21.15.46:0/489215416 wait complete. 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.563+0000 7f9bfc200700 1 Processor -- start 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bfc200700 1 -- start start 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bfc200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bfc200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf410ac60 con 0x7f9bf4107d90 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bf9800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bf9800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50700/0 (socket says 172.21.15.46:50700) 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.564+0000 7f9bf9800700 1 -- 172.21.15.46:0/1041876828 learned_addr learned my addr 172.21.15.46:0/1041876828 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:40.334 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.565+0000 7f9bf9800700 1 -- 172.21.15.46:0/1041876828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf419be30 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.565+0000 7f9bf9800700 1 --2- 172.21.15.46:0/1041876828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009fb0 tx=0x7f9be8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9be80038b0 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9be8011e50 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf419c0c0 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9be8019550 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf419ed60 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9be8019be0 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.566+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9be80243f0 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.567+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf410ade0 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.570+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9be8003a50 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.651+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f9bf402da60 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.654+0000 7f9bf2000700 1 -- 172.21.15.46:0/1041876828 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9be801a770 con 0x7f9bf4107d90 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.655+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 msgr2=0x7f9bf419b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:40.335 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.655+0000 7f9bfc200700 1 --2- 172.21.15.46:0/1041876828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9be8009fb0 tx=0x7f9be8011bf0 comp rx=0 tx=0).stop 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.655+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 shutdown_connections 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.655+0000 7f9bfc200700 1 --2- 172.21.15.46:0/1041876828 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9bf4107d90 0x7f9bf419b8f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.655+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 >> 172.21.15.46:0/1041876828 conn(0x7f9bf4101c70 msgr2=0x7f9bf4102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.656+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 shutdown_connections 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:39.656+0000 7f9bfc200700 1 -- 172.21.15.46:0/1041876828 wait complete. 2024-03-08T22:55:40.336 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-08T22:55:40.369 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-08T22:55:40.369 INFO:teuthology.orchestra.run.smithi046.stdout:Creating mgr... 2024-03-08T22:55:40.371 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-08T22:55:40.372 INFO:teuthology.orchestra.run.smithi046.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-08T22:55:40.785 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x 2024-03-08T22:55:40.785 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x.service: Unit ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x.service not loaded. 2024-03-08T22:55:40.952 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1041876828' entity='client.admin' 2024-03-08T22:55:41.062 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715.target.wants/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@.service. 2024-03-08T22:55:41.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 systemd[1]: Starting Ceph mgr.x for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:55:41.727 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 podman[32087]: 2024-03-08 22:55:41.352425139 +0000 UTC m=+0.040413353 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T22:55:41.727 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 podman[32087]: 2024-03-08 22:55:41.459232447 +0000 UTC m=+0.147220675 container create 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-08T22:55:41.727 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 podman[32087]: 2024-03-08 22:55:41.679760669 +0000 UTC m=+0.367748952 container init 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-08T22:55:41.727 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 podman[32087]: 2024-03-08 22:55:41.690591419 +0000 UTC m=+0.378579672 container start 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-08T22:55:41.806 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T22:55:41.806 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-08T22:55:41.822 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T22:55:41.822 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:55:41.822 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-08T22:55:41.822 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to enable service . firewalld.service is not available 2024-03-08T22:55:41.843 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T22:55:41.843 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-08T22:55:41.862 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T22:55:41.863 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:55:41.863 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-08T22:55:41.863 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-08T22:55:41.864 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr to start... 2024-03-08T22:55:41.864 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr... 2024-03-08T22:55:42.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 bash[32087]: 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 2024-03-08T22:55:42.024 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:41 smithi046 systemd[1]: Started Ceph mgr.x for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:55:42.685 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:42.473+0000 7f1339c35200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T22:55:42.685 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:42.600+0000 7f1339c35200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T22:55:42.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:42 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1047034126' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T22:55:43.411 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:43.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:55:43.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T22:55:43.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T22:55:43.413 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:43.414 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T22:55:43.416 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T22:55:43.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T22:55:43.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T22:55:43.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T22:55:43.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T22:55:43.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T22:55:43.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T22:55:43.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T22:55:43.423 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T22:55:43.423 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T22:55:43.423 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:43.423 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T22:55:32.141133+0000", 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.588+0000 7f4534a00700 1 Processor -- start 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.588+0000 7f4534a00700 1 -- start start 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.589+0000 7f4534a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.589+0000 7f4534a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45301080b0 con 0x7f45301076c0 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.589+0000 7f452f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:43.424 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.589+0000 7f452f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50716/0 (socket says 172.21.15.46:50716) 2024-03-08T22:55:43.425 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.589+0000 7f452f400700 1 -- 172.21.15.46:0/104655540 learned_addr learned my addr 172.21.15.46:0/104655540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:43.425 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.590+0000 7f452f400700 1 -- 172.21.15.46:0/104655540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530108870 con 0x7f45301076c0 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.590+0000 7f452f400700 1 --2- 172.21.15.46:0/104655540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f45200098a0 tx=0x7f4520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=432c78842c87ada8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.590+0000 7f452e000700 1 -- 172.21.15.46:0/104655540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4520004160 con 0x7f45301076c0 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f452e000700 1 -- 172.21.15.46:0/104655540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4520004300 con 0x7f45301076c0 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f452e000700 1 -- 172.21.15.46:0/104655540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4520011b50 con 0x7f45301076c0 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 -- 172.21.15.46:0/104655540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 msgr2=0x7f4530107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 --2- 172.21.15.46:0/104655540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f45200098a0 tx=0x7f4520009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 -- 172.21.15.46:0/104655540 shutdown_connections 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 --2- 172.21.15.46:0/104655540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f4530107ae0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 -- 172.21.15.46:0/104655540 >> 172.21.15.46:0/104655540 conn(0x7f4530102ce0 msgr2=0x7f4530105100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:43.429 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 -- 172.21.15.46:0/104655540 shutdown_connections 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.591+0000 7f4534a00700 1 -- 172.21.15.46:0/104655540 wait complete. 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.592+0000 7f4534a00700 1 Processor -- start 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f4534a00700 1 -- start start 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f4534a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f4534a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453010baa0 con 0x7f45301076c0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f452f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f452f400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50722/0 (socket says 172.21.15.46:50722) 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.593+0000 7f452f400700 1 -- 172.21.15.46:0/1047034126 learned_addr learned my addr 172.21.15.46:0/1047034126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f452f400700 1 -- 172.21.15.46:0/1047034126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453007d0b0 con 0x7f45301076c0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f452f400700 1 --2- 172.21.15.46:0/1047034126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4520011850 tx=0x7f4520011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4520003af0 con 0x7f45301076c0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4520003c90 con 0x7f45301076c0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f45200236b0 con 0x7f45301076c0 2024-03-08T22:55:43.430 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.594+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453007b2a0 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.595+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f453007b7e0 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.595+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f4520022030 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.595+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4520011e00 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.596+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530063410 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.598+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f45200042a0 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.679+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f453007bcb0 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.680+0000 7f4527e00700 1 -- 172.21.15.46:0/1047034126 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f452001a980 con 0x7f45301076c0 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 msgr2=0x7f453007cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 --2- 172.21.15.46:0/1047034126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4520011850 tx=0x7f4520011c10 comp rx=0 tx=0).stop 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 shutdown_connections 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 --2- 172.21.15.46:0/1047034126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45301076c0 0x7f453007cb70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 >> 172.21.15.46:0/1047034126 conn(0x7f4530102ce0 msgr2=0x7f4530103810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 shutdown_connections 2024-03-08T22:55:43.431 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:42.683+0000 7f4534a00700 1 -- 172.21.15.46:0/1047034126 wait complete. 2024-03-08T22:55:43.432 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (1/15)... 2024-03-08T22:55:43.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:43.453+0000 7f1339c35200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T22:55:43.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:43.576+0000 7f1339c35200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T22:55:44.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:43.758+0000 7f1339c35200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T22:55:44.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:43.884+0000 7f1339c35200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T22:55:44.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:43.997+0000 7f1339c35200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T22:55:44.602 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:44.336+0000 7f1339c35200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T22:55:44.602 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:44.480+0000 7f1339c35200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T22:55:44.949 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:44.597+0000 7f1339c35200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T22:55:44.949 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:44.723+0000 7f1339c35200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T22:55:44.950 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:44.945+0000 7f1339c35200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T22:55:46.480 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:46 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3326329821' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T22:55:46.480 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:46.333+0000 7f1339c35200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T22:55:46.786 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:46.510+0000 7f1339c35200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T22:55:46.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:46.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:55:46.798 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T22:55:46.799 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T22:55:46.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T22:55:46.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T22:55:46.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T22:55:32.141133+0000", 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:46.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.030+0000 7fb5c7200700 1 Processor -- start 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.030+0000 7fb5c7200700 1 -- start start 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.031+0000 7fb5c7200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.031+0000 7fb5c7200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c0107400 con 0x7fb5c0108e70 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.031+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.031+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50182/0 (socket says 172.21.15.46:50182) 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.031+0000 7fb5bfe00700 1 -- 172.21.15.46:0/3903037813 learned_addr learned my addr 172.21.15.46:0/3903037813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.032+0000 7fb5bfe00700 1 -- 172.21.15.46:0/3903037813 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0107580 con 0x7fb5c0108e70 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.032+0000 7fb5bfe00700 1 --2- 172.21.15.46:0/3903037813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00098a0 tx=0x7fb5b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52f0328c634dddc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:46.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.032+0000 7fb5bea00700 1 -- 172.21.15.46:0/3903037813 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0004160 con 0x7fb5c0108e70 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.032+0000 7fb5bea00700 1 -- 172.21.15.46:0/3903037813 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5b0004300 con 0x7fb5c0108e70 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.032+0000 7fb5bea00700 1 -- 172.21.15.46:0/3903037813 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0011b50 con 0x7fb5c0108e70 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 -- 172.21.15.46:0/3903037813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 msgr2=0x7fb5c0109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 --2- 172.21.15.46:0/3903037813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb5b00098a0 tx=0x7fb5b0009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 -- 172.21.15.46:0/3903037813 shutdown_connections 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 --2- 172.21.15.46:0/3903037813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c0109290 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 -- 172.21.15.46:0/3903037813 >> 172.21.15.46:0/3903037813 conn(0x7fb5c0102d80 msgr2=0x7fb5c01051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 -- 172.21.15.46:0/3903037813 shutdown_connections 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.033+0000 7fb5c7200700 1 -- 172.21.15.46:0/3903037813 wait complete. 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.034+0000 7fb5c7200700 1 Processor -- start 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5c7200700 1 -- start start 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5c7200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5c7200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c010bd40 con 0x7fb5c0108e70 2024-03-08T22:55:46.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5bfe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50198/0 (socket says 172.21.15.46:50198) 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.035+0000 7fb5bfe00700 1 -- 172.21.15.46:0/3326329821 learned_addr learned my addr 172.21.15.46:0/3326329821 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5bfe00700 1 -- 172.21.15.46:0/3326329821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c019e010 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5bfe00700 1 --2- 172.21.15.46:0/3326329821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0011850 tx=0x7fb5b0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5b0003af0 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb5b0020070 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c019e2a0 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5b00195a0 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.036+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c01a0f90 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.037+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fb5b0019c30 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.037+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb5b0011e00 con 0x7fb5c0108e70 2024-03-08T22:55:46.814 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.038+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c010bec0 con 0x7fb5c0108e70 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.042+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fb5b0003c90 con 0x7fb5c0108e70 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.119+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb5c01a14d0 con 0x7fb5c0108e70 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.120+0000 7fb5bcc00700 1 -- 172.21.15.46:0/3326329821 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb5b001a430 con 0x7fb5c0108e70 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 msgr2=0x7fb5c019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 --2- 172.21.15.46:0/3326329821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb5b0011850 tx=0x7fb5b0011c10 comp rx=0 tx=0).stop 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 shutdown_connections 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 --2- 172.21.15.46:0/3326329821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5c0108e70 0x7fb5c019dad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 >> 172.21.15.46:0/3326329821 conn(0x7fb5c0102d80 msgr2=0x7fb5c01038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 shutdown_connections 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:46.121+0000 7fb5c7200700 1 -- 172.21.15.46:0/3326329821 wait complete. 2024-03-08T22:55:46.815 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (2/15)... 2024-03-08T22:55:47.065 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:46.938+0000 7f1339c35200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T22:55:47.065 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:47.060+0000 7f1339c35200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T22:55:47.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:47.320+0000 7f1339c35200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:55:47.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:47.568+0000 7f1339c35200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T22:55:48.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:47.948+0000 7f1339c35200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:55:48.584 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:48 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:48.269+0000 7f1339c35200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T22:55:48.961 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:48 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:48.579+0000 7f1339c35200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T22:55:49.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:49 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1350712027' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T22:55:49.910 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:49 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:49.662+0000 7f1339c35200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T22:55:50.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:50.239 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-08T22:55:50.240 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T22:55:50.241 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T22:55:50.250 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T22:55:50.251 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T22:55:32.141133+0000", 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:50.252 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.396+0000 7f93aaa00700 1 Processor -- start 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.397+0000 7f93aaa00700 1 -- start start 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.397+0000 7f93aaa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.397+0000 7f93aaa00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a41062d0 con 0x7f93a4107d90 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.397+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.398+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50200/0 (socket says 172.21.15.46:50200) 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.398+0000 7f93a3e00700 1 -- 172.21.15.46:0/3293997486 learned_addr learned my addr 172.21.15.46:0/3293997486 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.398+0000 7f93a3e00700 1 -- 172.21.15.46:0/3293997486 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4106450 con 0x7f93a4107d90 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.398+0000 7f93a3e00700 1 --2- 172.21.15.46:0/3293997486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f939400bb40 tx=0x7f939400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a789c069edb625 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93a2a00700 1 -- 172.21.15.46:0/3293997486 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f939400dbe0 con 0x7f93a4107d90 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93a2a00700 1 -- 172.21.15.46:0/3293997486 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f93940044f0 con 0x7f93a4107d90 2024-03-08T22:55:50.253 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93a2a00700 1 -- 172.21.15.46:0/3293997486 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9394010470 con 0x7f93a4107d90 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 -- 172.21.15.46:0/3293997486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 msgr2=0x7f93a41081b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 --2- 172.21.15.46:0/3293997486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f939400bb40 tx=0x7f939400be50 comp rx=0 tx=0).stop 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 -- 172.21.15.46:0/3293997486 shutdown_connections 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 --2- 172.21.15.46:0/3293997486 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a41081b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 -- 172.21.15.46:0/3293997486 >> 172.21.15.46:0/3293997486 conn(0x7f93a4101c70 msgr2=0x7f93a4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.399+0000 7f93aaa00700 1 -- 172.21.15.46:0/3293997486 shutdown_connections 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.400+0000 7f93aaa00700 1 -- 172.21.15.46:0/3293997486 wait complete. 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93aaa00700 1 Processor -- start 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93aaa00700 1 -- start start 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93aaa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93aaa00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a407a150 con 0x7f93a4107d90 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:50.254 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93a3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50208/0 (socket says 172.21.15.46:50208) 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.401+0000 7f93a3e00700 1 -- 172.21.15.46:0/1350712027 learned_addr learned my addr 172.21.15.46:0/1350712027 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.402+0000 7f93a3e00700 1 -- 172.21.15.46:0/1350712027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a419e020 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.402+0000 7f93a3e00700 1 --2- 172.21.15.46:0/1350712027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f939400b340 tx=0x7f939400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.402+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9394010680 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.402+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f939401d070 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.402+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9394021af0 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.403+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a419e2b0 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.403+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a41a0fa0 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.403+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f939401f030 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.403+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f939402a7b0 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.404+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a4063410 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.406+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9394015070 con 0x7f93a4107d90 2024-03-08T22:55:50.255 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.486+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f93a4197040 con 0x7f93a4107d90 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.487+0000 7f93a0c00700 1 -- 172.21.15.46:0/1350712027 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f9394021410 con 0x7f93a4107d90 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.488+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 msgr2=0x7f93a419dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.488+0000 7f93aaa00700 1 --2- 172.21.15.46:0/1350712027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f939400b340 tx=0x7f939400dd90 comp rx=0 tx=0).stop 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.488+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 shutdown_connections 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.488+0000 7f93aaa00700 1 --2- 172.21.15.46:0/1350712027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93a4107d90 0x7f93a419dae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.488+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 >> 172.21.15.46:0/1350712027 conn(0x7f93a4101c70 msgr2=0x7f93a4194100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.489+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 shutdown_connections 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:49.489+0000 7f93aaa00700 1 -- 172.21.15.46:0/1350712027 wait complete. 2024-03-08T22:55:50.256 INFO:teuthology.orchestra.run.smithi046.stdout:mgr not available, waiting (3/15)... 2024-03-08T22:55:50.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:50.381+0000 7f1339c35200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T22:55:50.808 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:50.514+0000 7f1339c35200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T22:55:51.072 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:50.803+0000 7f1339c35200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: Activating manager daemon x 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: mgrmap e2: x(active, starting, since 0.00517236s) 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: Manager daemon x is now available 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T22:55:51.073 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' 2024-03-08T22:55:51.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T22:55:51.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' 2024-03-08T22:55:51.074 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:50 smithi046 ceph-mon[31859]: from='mgr.14100 172.21.15.46:0/1186163418' entity='mgr.x' 2024-03-08T22:55:52.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:52 smithi046 ceph-mon[31859]: mgrmap e3: x(active, since 1.02064s) 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 0 2024-03-08T22:55:53.800 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T22:55:53.801 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T22:55:53.808 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T22:55:53.809 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ], 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T22:55:32.141133+0000", 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout }, 2024-03-08T22:55:53.810 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.824+0000 7f007de00700 1 Processor -- start 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.824+0000 7f007de00700 1 -- start start 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.825+0000 7f007de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.825+0000 7f007de00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007807cfe0 con 0x7f007807b300 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.825+0000 7f0076a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.825+0000 7f0076a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50280/0 (socket says 172.21.15.46:50280) 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.825+0000 7f0076a00700 1 -- 172.21.15.46:0/4085442389 learned_addr learned my addr 172.21.15.46:0/4085442389 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.826+0000 7f0076a00700 1 -- 172.21.15.46:0/4085442389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007807bc60 con 0x7f007807b300 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.826+0000 7f0076a00700 1 --2- 172.21.15.46:0/4085442389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00680098a0 tx=0x7f0068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfd2689c1e73742 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.830+0000 7f0075600700 1 -- 172.21.15.46:0/4085442389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0068004160 con 0x7f007807b300 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.830+0000 7f0075600700 1 -- 172.21.15.46:0/4085442389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0068004300 con 0x7f007807b300 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.830+0000 7f0075600700 1 -- 172.21.15.46:0/4085442389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0068011b50 con 0x7f007807b300 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.830+0000 7f007de00700 1 -- 172.21.15.46:0/4085442389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 msgr2=0x7f007807b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:53.811 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.830+0000 7f007de00700 1 --2- 172.21.15.46:0/4085442389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00680098a0 tx=0x7f0068009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.831+0000 7f007de00700 1 -- 172.21.15.46:0/4085442389 shutdown_connections 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.831+0000 7f007de00700 1 --2- 172.21.15.46:0/4085442389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007807b720 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.831+0000 7f007de00700 1 -- 172.21.15.46:0/4085442389 >> 172.21.15.46:0/4085442389 conn(0x7f0078102d80 msgr2=0x7f00781051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.831+0000 7f007de00700 1 -- 172.21.15.46:0/4085442389 shutdown_connections 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.831+0000 7f007de00700 1 -- 172.21.15.46:0/4085442389 wait complete. 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.832+0000 7f007de00700 1 Processor -- start 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.832+0000 7f007de00700 1 -- start start 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.832+0000 7f007de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.832+0000 7f007de00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007810ddf0 con 0x7f007807b300 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.833+0000 7f0076a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:53.812 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.833+0000 7f0076a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50288/0 (socket says 172.21.15.46:50288) 2024-03-08T22:55:53.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.833+0000 7f0076a00700 1 -- 172.21.15.46:0/2873188655 learned_addr learned my addr 172.21.15.46:0/2873188655 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:53.813 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.833+0000 7f0076a00700 1 -- 172.21.15.46:0/2873188655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007819df90 con 0x7f007807b300 2024-03-08T22:55:53.815 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.833+0000 7f0076a00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0068011b60 tx=0x7f0068011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.834+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0068003970 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.834+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0068003b10 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.834+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0068024610 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.834+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007819e220 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.834+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00781a0f10 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.835+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f006801a9f0 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.835+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078063410 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.836+0000 7f007ca00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 0x7f005c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.836+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0068011e00 con 0x7f007807b300 2024-03-08T22:55:53.816 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.836+0000 7f0076000700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 0x7f005c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.837+0000 7f0076000700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 0x7f005c03b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:52.842+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0068019e10 con 0x7f007807b300 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.154+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f00781a1440 con 0x7f007807b300 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.155+0000 7f007ca00700 1 -- 172.21.15.46:0/2873188655 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f006804c9f0 con 0x7f007807b300 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 msgr2=0x7f005c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 0x7f005c03b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0064006fd0 tx=0x7f0064006e40 comp rx=0 tx=0).stop 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 msgr2=0x7f007819da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0068011b60 tx=0x7f0068011c40 comp rx=0 tx=0).stop 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 shutdown_connections 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f005c038f20 0x7f005c03b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:53.817 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 --2- 172.21.15.46:0/2873188655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f007807b300 0x7f007819da50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:53.818 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 >> 172.21.15.46:0/2873188655 conn(0x7f0078102d80 msgr2=0x7f0078103970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:53.818 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 shutdown_connections 2024-03-08T22:55:53.818 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:53.158+0000 7f007de00700 1 -- 172.21.15.46:0/2873188655 wait complete. 2024-03-08T22:55:53.818 INFO:teuthology.orchestra.run.smithi046.stdout:mgr is available 2024-03-08T22:55:54.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:53 smithi046 ceph-mon[31859]: mgrmap e4: x(active, since 2s) 2024-03-08T22:55:54.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:53 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2873188655' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T22:55:54.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2858380033' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [global] 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout fsid = ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-08T22:55:55.403 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout [osd] 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.403+0000 7f8a17200700 1 Processor -- start 2024-03-08T22:55:55.404 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.403+0000 7f8a17200700 1 -- start start 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.403+0000 7f8a17200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.403+0000 7f8a17200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1010eb30 con 0x7f8a1010e140 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.403+0000 7f8a0fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.404+0000 7f8a0fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48566/0 (socket says 172.21.15.46:48566) 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.404+0000 7f8a0fe00700 1 -- 172.21.15.46:0/1764736361 learned_addr learned my addr 172.21.15.46:0/1764736361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.404+0000 7f8a0fe00700 1 -- 172.21.15.46:0/1764736361 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1010ef70 con 0x7f8a1010e140 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.404+0000 7f8a0fe00700 1 --2- 172.21.15.46:0/1764736361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098a0 tx=0x7f8a00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2645e358d762a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.405+0000 7f8a0ea00700 1 -- 172.21.15.46:0/1764736361 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a00004160 con 0x7f8a1010e140 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.405+0000 7f8a0ea00700 1 -- 172.21.15.46:0/1764736361 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a00004300 con 0x7f8a1010e140 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.405+0000 7f8a0ea00700 1 -- 172.21.15.46:0/1764736361 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a00011b50 con 0x7f8a1010e140 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 -- 172.21.15.46:0/1764736361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 msgr2=0x7f8a1010e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 --2- 172.21.15.46:0/1764736361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8a000098a0 tx=0x7f8a00009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:55.405 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 -- 172.21.15.46:0/1764736361 shutdown_connections 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 --2- 172.21.15.46:0/1764736361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a1010e560 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 -- 172.21.15.46:0/1764736361 >> 172.21.15.46:0/1764736361 conn(0x7f8a1007ceb0 msgr2=0x7f8a1007f2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 -- 172.21.15.46:0/1764736361 shutdown_connections 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.406+0000 7f8a17200700 1 -- 172.21.15.46:0/1764736361 wait complete. 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.407+0000 7f8a17200700 1 Processor -- start 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.407+0000 7f8a17200700 1 -- start start 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.407+0000 7f8a17200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.407+0000 7f8a17200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10112320 con 0x7f8a1010e140 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.408+0000 7f8a0fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.408+0000 7f8a0fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48582/0 (socket says 172.21.15.46:48582) 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.408+0000 7f8a0fe00700 1 -- 172.21.15.46:0/2858380033 learned_addr learned my addr 172.21.15.46:0/2858380033 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.408+0000 7f8a0fe00700 1 -- 172.21.15.46:0/2858380033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a101a2230 con 0x7f8a1010e140 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.408+0000 7f8a0fe00700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009fb0 tx=0x7f8a00011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:55.406 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.409+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a000038b0 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.409+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a00011e50 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.409+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8a00019420 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.409+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a101a24c0 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.409+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a101a51a0 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.410+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a10063410 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.410+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f8a00003a50 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.411+0000 7f8a0cc00700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 0x7f89f403b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.411+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8a0004d0b0 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.411+0000 7f8a0f400700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 0x7f89f403b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.412+0000 7f8a0f400700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 0x7f89f403b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f89fc006fd0 tx=0x7f89fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.416+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8a0004cad0 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.638+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8a101a5370 con 0x7f8a1010e140 2024-03-08T22:55:55.407 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.639+0000 7f8a0cc00700 1 -- 172.21.15.46:0/2858380033 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f8a101a5370 con 0x7f8a1010e140 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 msgr2=0x7f89f403b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 0x7f89f403b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f89fc006fd0 tx=0x7f89fc006e40 comp rx=0 tx=0).stop 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 msgr2=0x7f8a101a1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a00009fb0 tx=0x7f8a00011bf0 comp rx=0 tx=0).stop 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 shutdown_connections 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f89f4038e80 0x7f89f403b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 --2- 172.21.15.46:0/2858380033 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a1010e140 0x7f8a101a1cf0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 >> 172.21.15.46:0/2858380033 conn(0x7f8a1007ceb0 msgr2=0x7f8a1007eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 shutdown_connections 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:54.642+0000 7f8a17200700 1 -- 172.21.15.46:0/2858380033 wait complete. 2024-03-08T22:55:55.408 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling cephadm module... 2024-03-08T22:55:56.697 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:56 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/498570216' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-08T22:55:56.697 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setuser ceph since I am not root 2024-03-08T22:55:56.697 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setgroup ceph since I am not root 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.030+0000 7f12b9000700 1 Processor -- start 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.031+0000 7f12b9000700 1 -- start start 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.031+0000 7f12b9000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.031+0000 7f12b9000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b410a230 con 0x7f12b4109840 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.032+0000 7f12b3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.032+0000 7f12b3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48590/0 (socket says 172.21.15.46:48590) 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.032+0000 7f12b3400700 1 -- 172.21.15.46:0/3939903227 learned_addr learned my addr 172.21.15.46:0/3939903227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.032+0000 7f12b3400700 1 -- 172.21.15.46:0/3939903227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b410a9f0 con 0x7f12b4109840 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.032+0000 7f12b3400700 1 --2- 172.21.15.46:0/3939903227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f12a40098a0 tx=0x7f12a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fb5e4265e6f89a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.033+0000 7f12b2000700 1 -- 172.21.15.46:0/3939903227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a4004160 con 0x7f12b4109840 2024-03-08T22:55:56.974 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.033+0000 7f12b2000700 1 -- 172.21.15.46:0/3939903227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a4004300 con 0x7f12b4109840 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.033+0000 7f12b2000700 1 -- 172.21.15.46:0/3939903227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a4011b50 con 0x7f12b4109840 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.033+0000 7f12b9000700 1 -- 172.21.15.46:0/3939903227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 msgr2=0x7f12b4109c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.033+0000 7f12b9000700 1 --2- 172.21.15.46:0/3939903227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f12a40098a0 tx=0x7f12a4009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.034+0000 7f12b9000700 1 -- 172.21.15.46:0/3939903227 shutdown_connections 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.034+0000 7f12b9000700 1 --2- 172.21.15.46:0/3939903227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b4109c60 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.034+0000 7f12b9000700 1 -- 172.21.15.46:0/3939903227 >> 172.21.15.46:0/3939903227 conn(0x7f12b407cdd0 msgr2=0x7f12b41074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.034+0000 7f12b9000700 1 -- 172.21.15.46:0/3939903227 shutdown_connections 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.034+0000 7f12b9000700 1 -- 172.21.15.46:0/3939903227 wait complete. 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.035+0000 7f12b9000700 1 Processor -- start 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.035+0000 7f12b9000700 1 -- start start 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.035+0000 7f12b9000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.035+0000 7f12b9000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b410dda0 con 0x7f12b4109840 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.036+0000 7f12b3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:56.975 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.036+0000 7f12b3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48602/0 (socket says 172.21.15.46:48602) 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.036+0000 7f12b3400700 1 -- 172.21.15.46:0/498570216 learned_addr learned my addr 172.21.15.46:0/498570216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.036+0000 7f12b3400700 1 -- 172.21.15.46:0/498570216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b419dce0 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.036+0000 7f12b3400700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f12a4011850 tx=0x7f12a4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.037+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a4003af0 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.037+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12a4020070 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.037+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b419df70 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.037+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12a40195a0 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.037+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b41a0c60 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.038+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f12a401a430 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.038+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4063410 con 0x7f12b4109840 2024-03-08T22:55:56.976 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.039+0000 7f12abe00700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 0x7f129803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.039+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f12a4011e00 con 0x7f12b4109840 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.039+0000 7f12b2a00700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 0x7f129803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.040+0000 7f12b2a00700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 0x7f129803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.044+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f12a404cbf0 con 0x7f12b4109840 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.337+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f12b41a0e30 con 0x7f12b4109840 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.391+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f12b41a0e30 con 0x7f12b4109840 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12abe00700 1 -- 172.21.15.46:0/498570216 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f12a402bd00 con 0x7f12b4109840 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 msgr2=0x7f129803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 0x7f129803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).stop 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 msgr2=0x7f12b419d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f12a4011850 tx=0x7f12a4011c10 comp rx=0 tx=0).stop 2024-03-08T22:55:56.977 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 shutdown_connections 2024-03-08T22:55:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f1298038f20 0x7f129803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 --2- 172.21.15.46:0/498570216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12b4109840 0x7f12b419d7a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.397+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 >> 172.21.15.46:0/498570216 conn(0x7f12b407cdd0 msgr2=0x7f12b4106c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.398+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 shutdown_connections 2024-03-08T22:55:56.978 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:56.398+0000 7f12b9000700 1 -- 172.21.15.46:0/498570216 wait complete. 2024-03-08T22:55:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/498570216' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-08T22:55:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:57 smithi046 ceph-mon[31859]: mgrmap e5: x(active, since 5s) 2024-03-08T22:55:57.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:57.206+0000 7fc39924b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T22:55:57.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:57.331+0000 7fc39924b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T22:55:58.405 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:58.160+0000 7fc39924b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T22:55:58.405 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:58.286+0000 7fc39924b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:55:58.472 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.566+0000 7f148ca00700 1 Processor -- start 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f148ca00700 1 -- start start 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f148ca00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f148ca00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148807b910 con 0x7f148807ce90 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f1485e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f1485e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48614/0 (socket says 172.21.15.46:48614) 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.567+0000 7f1485e00700 1 -- 172.21.15.46:0/3418913056 learned_addr learned my addr 172.21.15.46:0/3418913056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.568+0000 7f1485e00700 1 -- 172.21.15.46:0/3418913056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148807ba90 con 0x7f148807ce90 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.568+0000 7f1485e00700 1 --2- 172.21.15.46:0/3418913056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14780098a0 tx=0x7f1478009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f510711840a40818 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.568+0000 7f1484a00700 1 -- 172.21.15.46:0/3418913056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1478004160 con 0x7f148807ce90 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.568+0000 7f1484a00700 1 -- 172.21.15.46:0/3418913056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1478004300 con 0x7f148807ce90 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.568+0000 7f1484a00700 1 -- 172.21.15.46:0/3418913056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1478011b50 con 0x7f148807ce90 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 -- 172.21.15.46:0/3418913056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 msgr2=0x7f148807b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 --2- 172.21.15.46:0/3418913056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f14780098a0 tx=0x7f1478009bb0 comp rx=0 tx=0).stop 2024-03-08T22:55:58.473 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 -- 172.21.15.46:0/3418913056 shutdown_connections 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 --2- 172.21.15.46:0/3418913056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f148807b340 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 -- 172.21.15.46:0/3418913056 >> 172.21.15.46:0/3418913056 conn(0x7f1488102e20 msgr2=0x7f1488105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 -- 172.21.15.46:0/3418913056 shutdown_connections 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.569+0000 7f148ca00700 1 -- 172.21.15.46:0/3418913056 wait complete. 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.570+0000 7f148ca00700 1 Processor -- start 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.570+0000 7f148ca00700 1 -- start start 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f148ca00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f148ca00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1488109940 con 0x7f148807ce90 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f1485e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f1485e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48620/0 (socket says 172.21.15.46:48620) 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f1485e00700 1 -- 172.21.15.46:0/1507648943 learned_addr learned my addr 172.21.15.46:0/1507648943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.571+0000 7f1485e00700 1 -- 172.21.15.46:0/1507648943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14881a21d0 con 0x7f148807ce90 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f1485e00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1478011670 tx=0x7f1478011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:55:58.474 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14780039c0 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1478011e50 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14881a2460 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f147801a5c0 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.572+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14881a5150 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.573+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f147801a870 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.573+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1488063410 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.574+0000 7f147ea00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 0x7f146c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.574+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f147804d200 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.574+0000 7f1485400700 1 -- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 msgr2=0x7f146c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.574+0000 7f1485400700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 0x7f146c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.579+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f147800b3e0 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.775+0000 7f1485400700 1 -- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 msgr2=0x7f146c03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.775+0000 7f1485400700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 0x7f146c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.861+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f14880008d0 con 0x7f148807ce90 2024-03-08T22:55:58.475 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.862+0000 7f147ea00700 1 -- 172.21.15.46:0/1507648943 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1478020240 con 0x7f148807ce90 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 msgr2=0x7f146c03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 0x7f146c03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 msgr2=0x7f14881a1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1478011670 tx=0x7f1478011b60 comp rx=0 tx=0).stop 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 shutdown_connections 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f146c038f20 0x7f146c03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 --2- 172.21.15.46:0/1507648943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f148807ce90 0x7f14881a1c90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.864+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 >> 172.21.15.46:0/1507648943 conn(0x7f1488102e20 msgr2=0x7f1488105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.865+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 shutdown_connections 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:57.865+0000 7f148ca00700 1 -- 172.21.15.46:0/1507648943 wait complete. 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-03-08T22:55:58.476 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 5... 2024-03-08T22:55:58.719 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1507648943' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-08T22:55:58.719 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:58.470+0000 7fc39924b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T22:55:58.719 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:58.600+0000 7fc39924b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T22:55:58.719 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:58.714+0000 7fc39924b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T22:55:59.329 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:59.064+0000 7fc39924b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T22:55:59.330 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:59.207+0000 7fc39924b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T22:55:59.330 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:59.325+0000 7fc39924b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T22:55:59.640 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:59.452+0000 7fc39924b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T22:55:59.976 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:55:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:55:59.674+0000 7fc39924b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T22:56:01.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:00.989+0000 7fc39924b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T22:56:01.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:01.167+0000 7fc39924b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T22:56:01.976 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:01.592+0000 7fc39924b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T22:56:01.976 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:01.715+0000 7fc39924b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T22:56:02.235 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:01.978+0000 7fc39924b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:56:02.618 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:02.230+0000 7fc39924b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T22:56:02.941 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:02.614+0000 7fc39924b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:56:03.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:02.937+0000 7fc39924b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T22:56:03.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:03.251+0000 7fc39924b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T22:56:04.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:04.310+0000 7fc39924b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T22:56:05.455 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:05.025+0000 7fc39924b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T22:56:05.455 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:05.160+0000 7fc39924b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T22:56:05.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:05.450+0000 7fc39924b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T22:56:05.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: Active manager daemon x restarted 2024-03-08T22:56:05.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: Activating manager daemon x 2024-03-08T22:56:05.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: osdmap e2: 0 total, 0 up, 0 in 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: mgrmap e6: x(active, starting, since 0.00924381s) 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: Manager daemon x is now available 2024-03-08T22:56:05.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:05 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:06.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: Found migration_current of "None". Setting to last migration. 2024-03-08T22:56:06.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: mgrmap e7: x(active, since 1.01605s) 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:06.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:06 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:56:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-08T22:56:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-08T22:56:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:56:07.161 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.026+0000 7f4d95400700 1 Processor -- start 2024-03-08T22:56:07.162 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.026+0000 7f4d95400700 1 -- start start 2024-03-08T22:56:07.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.026+0000 7f4d95400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:07.170 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.027+0000 7f4d95400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9010a390 con 0x7f4d901099a0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.027+0000 7f4d8e800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.027+0000 7f4d8e800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48632/0 (socket says 172.21.15.46:48632) 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.027+0000 7f4d8e800700 1 -- 172.21.15.46:0/3735622129 learned_addr learned my addr 172.21.15.46:0/3735622129 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.027+0000 7f4d8e800700 1 -- 172.21.15.46:0/3735622129 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9010aba0 con 0x7f4d901099a0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.028+0000 7f4d8e800700 1 --2- 172.21.15.46:0/3735622129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098a0 tx=0x7f4d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1834368db9b6eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.028+0000 7f4d8d400700 1 -- 172.21.15.46:0/3735622129 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d80004160 con 0x7f4d901099a0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.028+0000 7f4d8d400700 1 -- 172.21.15.46:0/3735622129 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d80004300 con 0x7f4d901099a0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.028+0000 7f4d8d400700 1 -- 172.21.15.46:0/3735622129 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d80011b50 con 0x7f4d901099a0 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 -- 172.21.15.46:0/3735622129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 msgr2=0x7f4d90109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:07.171 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 --2- 172.21.15.46:0/3735622129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098a0 tx=0x7f4d80009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 -- 172.21.15.46:0/3735622129 shutdown_connections 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 --2- 172.21.15.46:0/3735622129 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d90109dc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 -- 172.21.15.46:0/3735622129 >> 172.21.15.46:0/3735622129 conn(0x7f4d9007ceb0 msgr2=0x7f4d901074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 -- 172.21.15.46:0/3735622129 shutdown_connections 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.029+0000 7f4d95400700 1 -- 172.21.15.46:0/3735622129 wait complete. 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.030+0000 7f4d95400700 1 Processor -- start 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.030+0000 7f4d95400700 1 -- start start 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.030+0000 7f4d95400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.030+0000 7f4d95400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9010df50 con 0x7f4d901099a0 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.031+0000 7f4d8e800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.031+0000 7f4d8e800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:48634/0 (socket says 172.21.15.46:48634) 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.031+0000 7f4d8e800700 1 -- 172.21.15.46:0/2823247600 learned_addr learned my addr 172.21.15.46:0/2823247600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:07.172 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.031+0000 7f4d8e800700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9019dea0 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.031+0000 7f4d8e800700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d80011850 tx=0x7f4d80011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.032+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d80003af0 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.032+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d80020070 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.032+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9019e130 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.032+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d800195a0 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.032+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d901a0e20 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.033+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f4d80019850 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.034+0000 7f4d87400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.034+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d7403bba0 con 0x7f4d74038fd0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.034+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4d80011e00 con 0x7f4d901099a0 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.034+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.034+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:56:07.173 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.235+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.235+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.636+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:55:59.636+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:00.437+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:00.437+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:02.039+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:02.039+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:05.243+0000 7f4d8de00700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/2941246107 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:05.243+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:05.460+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f4d80019b40 con 0x7f4d901099a0 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:05.460+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 msgr2=0x7f4d7403b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:05.460+0000 7f4d87400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.174 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.463+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f4d8004e7d0 con 0x7f4d901099a0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.464+0000 7f4d87400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 0x7f4d7403ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.464+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d80001d30 con 0x7f4d7403c8f0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.465+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 0x7f4d7403ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.466+0000 7f4d8de00700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 0x7f4d7403ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c003a10 tx=0x7f4d7c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.467+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f4d80001d30 con 0x7f4d7403c8f0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.477+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f4d90063410 con 0x7f4d7403c8f0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.478+0000 7f4d87400700 1 -- 172.21.15.46:0/2823247600 <== mgr.14120 v2:172.21.15.46:6800/4209254516 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f4d90063410 con 0x7f4d7403c8f0 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.478+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 msgr2=0x7f4d7403ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.478+0000 7f4d95400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 0x7f4d7403ece0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c003a10 tx=0x7f4d7c0092b0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.478+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 msgr2=0x7f4d9019d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.478+0000 7f4d95400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4d80011850 tx=0x7f4d80011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 shutdown_connections 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f4d7403c8f0 0x7f4d7403ece0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.175 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d901099a0 0x7f4d9019d960 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 --2- 172.21.15.46:0/2823247600 >> [v2:172.21.15.46:6800/2941246107,v1:172.21.15.46:6801/2941246107] conn(0x7f4d74038fd0 0x7f4d7403b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 >> 172.21.15.46:0/2823247600 conn(0x7f4d9007ceb0 msgr2=0x7f4d90106d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 shutdown_connections 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:06.479+0000 7f4d95400700 1 -- 172.21.15.46:0/2823247600 wait complete. 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 5 is available 2024-03-08T22:56:07.176 INFO:teuthology.orchestra.run.smithi046.stdout:Setting orchestrator backend to cephadm... 2024-03-08T22:56:07.765 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:07] ENGINE Bus STARTING 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:07] ENGINE Serving on http://172.21.15.46:8765 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:07] ENGINE Serving on https://172.21.15.46:7150 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:07] ENGINE Bus STARTED 2024-03-08T22:56:07.766 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:07 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:08.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:08 smithi046 ceph-mon[31859]: mgrmap e8: x(active, since 2s) 2024-03-08T22:56:08.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:08 smithi046 ceph-mon[31859]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:08.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:08 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:08.638 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:08 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:08.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.761+0000 7f0d97000700 1 Processor -- start 2024-03-08T22:56:08.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.761+0000 7f0d97000700 1 -- start start 2024-03-08T22:56:08.658 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.762+0000 7f0d97000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.762+0000 7f0d97000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9007ae50 con 0x7f0d90106070 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.762+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.762+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51978/0 (socket says 172.21.15.46:51978) 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.762+0000 7f0d8fe00700 1 -- 172.21.15.46:0/1695280906 learned_addr learned my addr 172.21.15.46:0/1695280906 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.763+0000 7f0d8fe00700 1 -- 172.21.15.46:0/1695280906 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d901069d0 con 0x7f0d90106070 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.763+0000 7f0d8fe00700 1 --2- 172.21.15.46:0/1695280906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098a0 tx=0x7f0d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc6d1f15ee76a08a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.763+0000 7f0d8f400700 1 -- 172.21.15.46:0/1695280906 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80004160 con 0x7f0d90106070 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.763+0000 7f0d8f400700 1 -- 172.21.15.46:0/1695280906 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d80004300 con 0x7f0d90106070 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.763+0000 7f0d8f400700 1 -- 172.21.15.46:0/1695280906 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80011b50 con 0x7f0d90106070 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.764+0000 7f0d97000700 1 -- 172.21.15.46:0/1695280906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 msgr2=0x7f0d90106490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.764+0000 7f0d97000700 1 --2- 172.21.15.46:0/1695280906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098a0 tx=0x7f0d80009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:08.659 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.764+0000 7f0d97000700 1 -- 172.21.15.46:0/1695280906 shutdown_connections 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.764+0000 7f0d97000700 1 --2- 172.21.15.46:0/1695280906 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d90106490 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.765+0000 7f0d97000700 1 -- 172.21.15.46:0/1695280906 >> 172.21.15.46:0/1695280906 conn(0x7f0d90101d30 msgr2=0x7f0d90104150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.765+0000 7f0d97000700 1 -- 172.21.15.46:0/1695280906 shutdown_connections 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.765+0000 7f0d97000700 1 -- 172.21.15.46:0/1695280906 wait complete. 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.766+0000 7f0d97000700 1 Processor -- start 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.766+0000 7f0d97000700 1 -- start start 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.766+0000 7f0d97000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.766+0000 7f0d97000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d901079f0 con 0x7f0d90106070 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.767+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.767+0000 7f0d8fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51984/0 (socket says 172.21.15.46:51984) 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.767+0000 7f0d8fe00700 1 -- 172.21.15.46:0/3459044969 learned_addr learned my addr 172.21.15.46:0/3459044969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.767+0000 7f0d8fe00700 1 -- 172.21.15.46:0/3459044969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9019de50 con 0x7f0d90106070 2024-03-08T22:56:08.660 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.767+0000 7f0d8fe00700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d80011b60 tx=0x7f0d80011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.768+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80003970 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.768+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d80003b10 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.768+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9019e0e0 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.768+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d80024610 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.768+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d901a0dd0 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.769+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0d8001a9f0 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.769+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d90063410 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.769+0000 7f0d8d600700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 0x7f0d7403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.769+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d80011e00 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.770+0000 7f0d87e00700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 0x7f0d7403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.770+0000 7f0d87e00700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 0x7f0d7403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d78006fd0 tx=0x7f0d78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:07.775+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0d80019e10 con 0x7f0d90106070 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.017+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d900008d0 con 0x7f0d74038f70 2024-03-08T22:56:08.661 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.027+0000 7f0d8d600700 1 -- 172.21.15.46:0/3459044969 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0d900008d0 con 0x7f0d74038f70 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 msgr2=0x7f0d7403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 0x7f0d7403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0d78006fd0 tx=0x7f0d78006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 msgr2=0x7f0d9019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0d80011b60 tx=0x7f0d80011c40 comp rx=0 tx=0).stop 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 shutdown_connections 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0d74038f70 0x7f0d7403b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 --2- 172.21.15.46:0/3459044969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d90106070 0x7f0d9019d910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.030+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 >> 172.21.15.46:0/3459044969 conn(0x7f0d90101d30 msgr2=0x7f0d90194580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.031+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 shutdown_connections 2024-03-08T22:56:08.662 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:08.031+0000 7f0d97000700 1 -- 172.21.15.46:0/3459044969 wait complete. 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.267+0000 7f3487800700 1 Processor -- start 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.267+0000 7f3487800700 1 -- start start 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.268+0000 7f3487800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.268+0000 7f3487800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480108220 con 0x7f3480107830 2024-03-08T22:56:10.144 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.269+0000 7f3484e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.269+0000 7f3484e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51998/0 (socket says 172.21.15.46:51998) 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.269+0000 7f3484e00700 1 -- 172.21.15.46:0/2895972064 learned_addr learned my addr 172.21.15.46:0/2895972064 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.269+0000 7f3484e00700 1 -- 172.21.15.46:0/2895972064 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480108a30 con 0x7f3480107830 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.269+0000 7f3484e00700 1 --2- 172.21.15.46:0/2895972064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34740098a0 tx=0x7f3474009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a197065dfdeca41d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f347f400700 1 -- 172.21.15.46:0/2895972064 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474004160 con 0x7f3480107830 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f347f400700 1 -- 172.21.15.46:0/2895972064 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3474004300 con 0x7f3480107830 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f347f400700 1 -- 172.21.15.46:0/2895972064 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474011b50 con 0x7f3480107830 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f3487800700 1 -- 172.21.15.46:0/2895972064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 msgr2=0x7f3480107c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f3487800700 1 --2- 172.21.15.46:0/2895972064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f34740098a0 tx=0x7f3474009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f3487800700 1 -- 172.21.15.46:0/2895972064 shutdown_connections 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f3487800700 1 --2- 172.21.15.46:0/2895972064 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f3480107c50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:10.145 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.270+0000 7f3487800700 1 -- 172.21.15.46:0/2895972064 >> 172.21.15.46:0/2895972064 conn(0x7f3480102e50 msgr2=0x7f3480105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.271+0000 7f3487800700 1 -- 172.21.15.46:0/2895972064 shutdown_connections 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.271+0000 7f3487800700 1 -- 172.21.15.46:0/2895972064 wait complete. 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3487800700 1 Processor -- start 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3487800700 1 -- start start 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3487800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3487800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348010bc60 con 0x7f3480107830 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3484e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3484e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52006/0 (socket says 172.21.15.46:52006) 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.272+0000 7f3484e00700 1 -- 172.21.15.46:0/471843874 learned_addr learned my addr 172.21.15.46:0/471843874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f3484e00700 1 -- 172.21.15.46:0/471843874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348019de90 con 0x7f3480107830 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f3484e00700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fb0 tx=0x7f3474011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34740039e0 con 0x7f3480107830 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3474011e50 con 0x7f3480107830 2024-03-08T22:56:10.146 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348019e120 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.273+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3474023550 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.274+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34801a0e10 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.275+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f3474022030 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.275+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348010bde0 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.275+0000 7f347d600700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 0x7f346803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.275+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f347404cb70 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.276+0000 7f347fe00700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 0x7f346803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.276+0000 7f347fe00700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 0x7f346803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f346c006fd0 tx=0x7f346c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.281+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f347404c4c0 con 0x7f3480107830 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.507+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3480104030 con 0x7f3468038f20 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.508+0000 7f347d600700 1 -- 172.21.15.46:0/471843874 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3480104030 con 0x7f3468038f20 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.511+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 msgr2=0x7f346803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.511+0000 7f3487800700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 0x7f346803b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f346c006fd0 tx=0x7f346c006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:10.147 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.511+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 msgr2=0x7f348019d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.511+0000 7f3487800700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3474009fb0 tx=0x7f3474011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 shutdown_connections 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f3468038f20 0x7f346803b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 --2- 172.21.15.46:0/471843874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3480107830 0x7f348019d950 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 >> 172.21.15.46:0/471843874 conn(0x7f3480102e50 msgr2=0x7f3480103920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 shutdown_connections 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:09.512+0000 7f3487800700 1 -- 172.21.15.46:0/471843874 wait complete. 2024-03-08T22:56:10.148 INFO:teuthology.orchestra.run.smithi046.stdout:Generating ssh key... 2024-03-08T22:56:10.669 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:10 smithi046 ceph-mon[31859]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: Generating public/private rsa key pair. 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: Your identification has been saved in /tmp/tmprnz7y9dm/key. 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: Your public key has been saved in /tmp/tmprnz7y9dm/key.pub. 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: The key fingerprint is: 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: SHA256:GQRI6rcv2S08UR9S7XSVOAUjpXJBCokYGtCgGanR5xQ ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:56:11.228 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: The key's randomart image is: 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: +---[RSA 3072]----+ 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: |=* oEoooo .=.++oo| 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: |+o=o+. o. o *oo. | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: |+o.+ .+ = .. | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: |.. . oo+ . | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: | . . .So . | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: | . .. . | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: | .+ o | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: | o.= . | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: | ..o | 2024-03-08T22:56:11.229 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: +----[SHA256]-----+ 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.714+0000 7f9683000700 1 Processor -- start 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.715+0000 7f9683000700 1 -- start start 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.715+0000 7f9683000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.715+0000 7f9683000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c07b890 con 0x7f967c07cf10 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.716+0000 7f967be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.716+0000 7f967be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52016/0 (socket says 172.21.15.46:52016) 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.716+0000 7f967be00700 1 -- 172.21.15.46:0/2961386206 learned_addr learned my addr 172.21.15.46:0/2961386206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.716+0000 7f967be00700 1 -- 172.21.15.46:0/2961386206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c07ba10 con 0x7f967c07cf10 2024-03-08T22:56:11.747 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.716+0000 7f967be00700 1 --2- 172.21.15.46:0/2961386206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f966c0098a0 tx=0x7f966c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3116e1bae809f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.717+0000 7f967aa00700 1 -- 172.21.15.46:0/2961386206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f966c004160 con 0x7f967c07cf10 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.717+0000 7f967aa00700 1 -- 172.21.15.46:0/2961386206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f966c004300 con 0x7f967c07cf10 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.717+0000 7f967aa00700 1 -- 172.21.15.46:0/2961386206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f966c011b50 con 0x7f967c07cf10 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 -- 172.21.15.46:0/2961386206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 msgr2=0x7f967c07b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 --2- 172.21.15.46:0/2961386206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f966c0098a0 tx=0x7f966c009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 -- 172.21.15.46:0/2961386206 shutdown_connections 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 --2- 172.21.15.46:0/2961386206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c07b2c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 -- 172.21.15.46:0/2961386206 >> 172.21.15.46:0/2961386206 conn(0x7f967c102d80 msgr2=0x7f967c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 -- 172.21.15.46:0/2961386206 shutdown_connections 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.718+0000 7f9683000700 1 -- 172.21.15.46:0/2961386206 wait complete. 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.719+0000 7f9683000700 1 Processor -- start 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.719+0000 7f9683000700 1 -- start start 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f9683000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f9683000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f967c10de20 con 0x7f967c07cf10 2024-03-08T22:56:11.748 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f967be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f967be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52020/0 (socket says 172.21.15.46:52020) 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f967be00700 1 -- 172.21.15.46:0/2135623552 learned_addr learned my addr 172.21.15.46:0/2135623552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.720+0000 7f967be00700 1 -- 172.21.15.46:0/2135623552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967c19df90 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f967be00700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f966c011b60 tx=0x7f966c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f966c003970 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f966c003b10 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f967c19e220 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f966c024610 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.721+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f967c1a0f10 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.722+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f966c024910 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.722+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f967c063410 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.723+0000 7f9678c00700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 0x7f966003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.723+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f966c011e00 con 0x7f967c07cf10 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.723+0000 7f967b400700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 0x7f966003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:11.749 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.724+0000 7f967b400700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 0x7f966003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.728+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f966c00ba00 con 0x7f967c07cf10 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:10.956+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f967c104090 con 0x7f9660038f20 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.079+0000 7f9678c00700 1 -- 172.21.15.46:0/2135623552 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f967c104090 con 0x7f9660038f20 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 msgr2=0x7f966003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 0x7f966003b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9668006fd0 tx=0x7f9668006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 msgr2=0x7f967c19da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f966c011b60 tx=0x7f966c011c40 comp rx=0 tx=0).stop 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 shutdown_connections 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f9660038f20 0x7f966003b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 --2- 172.21.15.46:0/2135623552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f967c07cf10 0x7f967c19da50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 >> 172.21.15.46:0/2135623552 conn(0x7f967c102d80 msgr2=0x7f967c103980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 shutdown_connections 2024-03-08T22:56:11.750 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:11.082+0000 7f9683000700 1 -- 172.21.15.46:0/2135623552 wait complete. 2024-03-08T22:56:12.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:12 smithi046 ceph-mon[31859]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:12.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:12 smithi046 ceph-mon[31859]: Generating ssh key... 2024-03-08T22:56:12.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:12 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:12.349 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:12 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:13.315 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.343+0000 7ff894c00700 1 Processor -- start 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.343+0000 7ff894c00700 1 -- start start 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.344+0000 7ff894c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.344+0000 7ff894c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890107400 con 0x7ff890108e70 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.344+0000 7ff88de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:13.317 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.344+0000 7ff88de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52028/0 (socket says 172.21.15.46:52028) 2024-03-08T22:56:13.318 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.345+0000 7ff88de00700 1 -- 172.21.15.46:0/3790110395 learned_addr learned my addr 172.21.15.46:0/3790110395 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:13.318 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.345+0000 7ff88de00700 1 -- 172.21.15.46:0/3790110395 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890107580 con 0x7ff890108e70 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.345+0000 7ff88de00700 1 --2- 172.21.15.46:0/3790110395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8800098a0 tx=0x7ff880009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8071c15ed6d7901c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.346+0000 7ff88ca00700 1 -- 172.21.15.46:0/3790110395 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff880004160 con 0x7ff890108e70 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.346+0000 7ff88ca00700 1 -- 172.21.15.46:0/3790110395 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff880004300 con 0x7ff890108e70 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.346+0000 7ff88ca00700 1 -- 172.21.15.46:0/3790110395 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff880011b50 con 0x7ff890108e70 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 -- 172.21.15.46:0/3790110395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 msgr2=0x7ff890109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:13.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 --2- 172.21.15.46:0/3790110395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8800098a0 tx=0x7ff880009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 -- 172.21.15.46:0/3790110395 shutdown_connections 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 --2- 172.21.15.46:0/3790110395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff890109290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 -- 172.21.15.46:0/3790110395 >> 172.21.15.46:0/3790110395 conn(0x7ff890102d80 msgr2=0x7ff8901051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 -- 172.21.15.46:0/3790110395 shutdown_connections 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.347+0000 7ff894c00700 1 -- 172.21.15.46:0/3790110395 wait complete. 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.348+0000 7ff894c00700 1 Processor -- start 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.348+0000 7ff894c00700 1 -- start start 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.348+0000 7ff894c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:13.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.348+0000 7ff894c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff89010bd40 con 0x7ff890108e70 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.349+0000 7ff88de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.349+0000 7ff88de00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52040/0 (socket says 172.21.15.46:52040) 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.349+0000 7ff88de00700 1 -- 172.21.15.46:0/1936823418 learned_addr learned my addr 172.21.15.46:0/1936823418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.349+0000 7ff88de00700 1 -- 172.21.15.46:0/1936823418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89019bdc0 con 0x7ff890108e70 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.349+0000 7ff88de00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff880011850 tx=0x7ff880011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.350+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff880003af0 con 0x7ff890108e70 2024-03-08T22:56:13.322 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.350+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff880020070 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.350+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8800195a0 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.350+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89019c050 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.350+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff89019ed40 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.351+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff89010bec0 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.351+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff88001a430 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.352+0000 7ff886a00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 0x7ff87403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.352+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff880011e00 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.352+0000 7ff88d400700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 0x7ff87403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.353+0000 7ff88d400700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 0x7ff87403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.357+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff880021270 con 0x7ff890108e70 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.574+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8901040c0 con 0x7ff874038f20 2024-03-08T22:56:13.325 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.576+0000 7ff886a00700 1 -- 172.21.15.46:0/1936823418 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7ff8901040c0 con 0x7ff874038f20 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.578+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 msgr2=0x7ff87403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.578+0000 7ff894c00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 0x7ff87403b3e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff87c006fd0 tx=0x7ff87c006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.578+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 msgr2=0x7ff89019b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.578+0000 7ff894c00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff880011850 tx=0x7ff880011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 shutdown_connections 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7ff874038f20 0x7ff87403b3e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 --2- 172.21.15.46:0/1936823418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff890108e70 0x7ff89019b880 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 >> 172.21.15.46:0/1936823418 conn(0x7ff890102d80 msgr2=0x7ff8901039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 shutdown_connections 2024-03-08T22:56:13.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:12.579+0000 7ff894c00700 1 -- 172.21.15.46:0/1936823418 wait complete. 2024-03-08T22:56:13.327 INFO:teuthology.orchestra.run.smithi046.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-08T22:56:13.327 INFO:teuthology.orchestra.run.smithi046.stdout:Adding key to root@localhost authorized_keys... 2024-03-08T22:56:13.328 INFO:teuthology.orchestra.run.smithi046.stdout:Adding host smithi046... 2024-03-08T22:56:13.857 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:13 smithi046 ceph-mon[31859]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:14.681 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:14 smithi046 ceph-mon[31859]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:16.713 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:16 smithi046 ceph-mon[31859]: Deploying cephadm binary to smithi046 2024-03-08T22:56:21.450 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Added host 'smithi046' with addr '172.21.15.46' 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.931+0000 7efecb800700 1 Processor -- start 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efecb800700 1 -- start start 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efecb800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efecb800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec4107400 con 0x7efec4108e70 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efec8e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efec8e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51288/0 (socket says 172.21.15.46:51288) 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.932+0000 7efec8e00700 1 -- 172.21.15.46:0/1317056056 learned_addr learned my addr 172.21.15.46:0/1317056056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.933+0000 7efec8e00700 1 -- 172.21.15.46:0/1317056056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec4107580 con 0x7efec4108e70 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.933+0000 7efec8e00700 1 --2- 172.21.15.46:0/1317056056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efeb40098a0 tx=0x7efeb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fae92c2fd92a4b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efec3e00700 1 -- 172.21.15.46:0/1317056056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efeb4004160 con 0x7efec4108e70 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efec3e00700 1 -- 172.21.15.46:0/1317056056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efeb4004300 con 0x7efec4108e70 2024-03-08T22:56:21.451 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efec3e00700 1 -- 172.21.15.46:0/1317056056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efeb4011b50 con 0x7efec4108e70 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efecb800700 1 -- 172.21.15.46:0/1317056056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 msgr2=0x7efec4109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efecb800700 1 --2- 172.21.15.46:0/1317056056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efeb40098a0 tx=0x7efeb4009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efecb800700 1 -- 172.21.15.46:0/1317056056 shutdown_connections 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efecb800700 1 --2- 172.21.15.46:0/1317056056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec4109290 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.934+0000 7efecb800700 1 -- 172.21.15.46:0/1317056056 >> 172.21.15.46:0/1317056056 conn(0x7efec4102d80 msgr2=0x7efec41051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.935+0000 7efecb800700 1 -- 172.21.15.46:0/1317056056 shutdown_connections 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.935+0000 7efecb800700 1 -- 172.21.15.46:0/1317056056 wait complete. 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.936+0000 7efecb800700 1 Processor -- start 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.936+0000 7efecb800700 1 -- start start 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.936+0000 7efecb800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:21.452 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.936+0000 7efecb800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec407b260 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.936+0000 7efec8e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.937+0000 7efec8e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51292/0 (socket says 172.21.15.46:51292) 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.937+0000 7efec8e00700 1 -- 172.21.15.46:0/1642337757 learned_addr learned my addr 172.21.15.46:0/1642337757 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.937+0000 7efec8e00700 1 -- 172.21.15.46:0/1642337757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec419dfd0 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.937+0000 7efec8e00700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efeb4011850 tx=0x7efeb4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.938+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efeb4003af0 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.938+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efeb4020070 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.938+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efeb40195a0 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.938+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec419e260 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.938+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec41a0f50 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.939+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7efeb40198a0 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.939+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4063410 con 0x7efec4108e70 2024-03-08T22:56:21.453 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.939+0000 7efec2000700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 0x7efea803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.939+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efeb4011e00 con 0x7efec4108e70 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.940+0000 7efebbe00700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 0x7efea803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.940+0000 7efebbe00700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 0x7efea803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:13.945+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efeb4019b90 con 0x7efec4108e70 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:14.174+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi046", "addr": "172.21.15.46", "target": ["mon-mgr", ""]}) v1 -- 0x7efec402da60 con 0x7efea8038f20 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.764+0000 7efec2000700 1 -- 172.21.15.46:0/1642337757 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7efec402da60 con 0x7efea8038f20 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 msgr2=0x7efea803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 0x7efea803b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efeac006fd0 tx=0x7efeac006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 msgr2=0x7efec419da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:21.454 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efeb4011850 tx=0x7efeb4011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 shutdown_connections 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7efea8038f20 0x7efea803b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 --2- 172.21.15.46:0/1642337757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efec4108e70 0x7efec419da90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 >> 172.21.15.46:0/1642337757 conn(0x7efec4102d80 msgr2=0x7efec4194ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.768+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 shutdown_connections 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:20.769+0000 7efecb800700 1 -- 172.21.15.46:0/1642337757 wait complete. 2024-03-08T22:56:21.455 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mon service... 2024-03-08T22:56:22.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:21 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:22.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:21 smithi046 ceph-mon[31859]: Added host smithi046 2024-03-08T22:56:22.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:21 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-08T22:56:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.066+0000 7f0aae600700 1 Processor -- start 2024-03-08T22:56:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.066+0000 7f0aae600700 1 -- start start 2024-03-08T22:56:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.067+0000 7f0aae600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:23.069 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.067+0000 7f0aae600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa8109680 con 0x7f0aa810b140 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.067+0000 7f0aa7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.068+0000 7f0aa7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51306/0 (socket says 172.21.15.46:51306) 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.068+0000 7f0aa7400700 1 -- 172.21.15.46:0/3388452184 learned_addr learned my addr 172.21.15.46:0/3388452184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.068+0000 7f0aa7400700 1 -- 172.21.15.46:0/3388452184 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8109800 con 0x7f0aa810b140 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.068+0000 7f0aa7400700 1 --2- 172.21.15.46:0/3388452184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a980098a0 tx=0x7f0a98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc99e8196645863b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.069+0000 7f0aa6a00700 1 -- 172.21.15.46:0/3388452184 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a98004160 con 0x7f0aa810b140 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.069+0000 7f0aa6a00700 1 -- 172.21.15.46:0/3388452184 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a98004300 con 0x7f0aa810b140 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.069+0000 7f0aa6a00700 1 -- 172.21.15.46:0/3388452184 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a98011b50 con 0x7f0aa810b140 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 -- 172.21.15.46:0/3388452184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 msgr2=0x7f0aa810b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 --2- 172.21.15.46:0/3388452184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0a980098a0 tx=0x7f0a98009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 -- 172.21.15.46:0/3388452184 shutdown_connections 2024-03-08T22:56:23.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 --2- 172.21.15.46:0/3388452184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa810b560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 -- 172.21.15.46:0/3388452184 >> 172.21.15.46:0/3388452184 conn(0x7f0aa807ce90 msgr2=0x7f0aa81074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 -- 172.21.15.46:0/3388452184 shutdown_connections 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.070+0000 7f0aae600700 1 -- 172.21.15.46:0/3388452184 wait complete. 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.071+0000 7f0aae600700 1 Processor -- start 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.071+0000 7f0aae600700 1 -- start start 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aae600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aae600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa810e010 con 0x7f0aa810b140 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aa7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aa7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51316/0 (socket says 172.21.15.46:51316) 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aa7400700 1 -- 172.21.15.46:0/4224031721 learned_addr learned my addr 172.21.15.46:0/4224031721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aa7400700 1 -- 172.21.15.46:0/4224031721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa819dfa0 con 0x7f0aa810b140 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.072+0000 7f0aa7400700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a98011850 tx=0x7f0a98011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.073+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a98003af0 con 0x7f0aa810b140 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.073+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0a98020070 con 0x7f0aa810b140 2024-03-08T22:56:23.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.073+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0a98019550 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.073+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa819e230 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.073+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa81a0f20 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.074+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0a9801a430 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.074+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa8063410 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.074+0000 7f0aa4c00700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 0x7f0a8c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.075+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0a98011e00 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.075+0000 7f0a9fe00700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 0x7f0a8c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.076+0000 7f0a9fe00700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 0x7f0a8c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a90006fd0 tx=0x7f0a90006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.080+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0a9801a720 con 0x7f0aa810b140 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.321+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f0aa81062b0 con 0x7f0a8c038f70 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.329+0000 7f0aa4c00700 1 -- 172.21.15.46:0/4224031721 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0aa81062b0 con 0x7f0a8c038f70 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 msgr2=0x7f0a8c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 0x7f0a8c03b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0a90006fd0 tx=0x7f0a90006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:23.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 msgr2=0x7f0aa819da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a98011850 tx=0x7f0a98011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 shutdown_connections 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f0a8c038f70 0x7f0a8c03b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 --2- 172.21.15.46:0/4224031721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0aa810b140 0x7f0aa819da60 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 >> 172.21.15.46:0/4224031721 conn(0x7f0aa807ce90 msgr2=0x7f0aa8105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 shutdown_connections 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:22.332+0000 7f0aae600700 1 -- 172.21.15.46:0/4224031721 wait complete. 2024-03-08T22:56:23.073 INFO:teuthology.orchestra.run.smithi046.stdout:Deploying unmanaged mgr service... 2024-03-08T22:56:23.586 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:23 smithi046 ceph-mon[31859]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:23.586 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:23 smithi046 ceph-mon[31859]: Saving service mon spec with placement count:5 2024-03-08T22:56:23.586 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:23 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:25.319 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.225+0000 7fca0e000700 1 Processor -- start 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.225+0000 7fca0e000700 1 -- start start 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.226+0000 7fca0e000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.226+0000 7fca0e000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0807af30 con 0x7fca08106090 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.226+0000 7fca07400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.226+0000 7fca07400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36522/0 (socket says 172.21.15.46:36522) 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.226+0000 7fca07400700 1 -- 172.21.15.46:0/1429121397 learned_addr learned my addr 172.21.15.46:0/1429121397 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:25.320 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.227+0000 7fca07400700 1 -- 172.21.15.46:0/1429121397 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca081069f0 con 0x7fca08106090 2024-03-08T22:56:25.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.227+0000 7fca07400700 1 --2- 172.21.15.46:0/1429121397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80098a0 tx=0x7fc9f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c64725cbb8eb6b32 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:25.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.227+0000 7fca06000700 1 -- 172.21.15.46:0/1429121397 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8004160 con 0x7fca08106090 2024-03-08T22:56:25.321 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.227+0000 7fca06000700 1 -- 172.21.15.46:0/1429121397 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9f8004300 con 0x7fca08106090 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.227+0000 7fca06000700 1 -- 172.21.15.46:0/1429121397 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8011b50 con 0x7fca08106090 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 -- 172.21.15.46:0/1429121397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 msgr2=0x7fca081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 --2- 172.21.15.46:0/1429121397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80098a0 tx=0x7fc9f8009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 -- 172.21.15.46:0/1429121397 shutdown_connections 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 --2- 172.21.15.46:0/1429121397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca081064b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 -- 172.21.15.46:0/1429121397 >> 172.21.15.46:0/1429121397 conn(0x7fca08101c70 msgr2=0x7fca08104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 -- 172.21.15.46:0/1429121397 shutdown_connections 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.228+0000 7fca0e000700 1 -- 172.21.15.46:0/1429121397 wait complete. 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.229+0000 7fca0e000700 1 Processor -- start 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca0e000700 1 -- start start 2024-03-08T22:56:25.326 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca0e000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca0e000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca08107a10 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca07400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca07400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36538/0 (socket says 172.21.15.46:36538) 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.230+0000 7fca07400700 1 -- 172.21.15.46:0/744669456 learned_addr learned my addr 172.21.15.46:0/744669456 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.231+0000 7fca07400700 1 -- 172.21.15.46:0/744669456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca0819e070 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.231+0000 7fca07400700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8011b60 tx=0x7fc9f8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.232+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8003970 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.232+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc9f8003b10 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.232+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9f8024610 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.232+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0819e300 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.232+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0819ee20 con 0x7fca08106090 2024-03-08T22:56:25.327 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.233+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fc9f801a9f0 con 0x7fca08106090 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.233+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca08063410 con 0x7fca08106090 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.233+0000 7fc9ffe00700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 0x7fc9ec03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.233+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc9f8011e00 con 0x7fca08106090 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.234+0000 7fca06a00700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 0x7fc9ec03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.235+0000 7fca06a00700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 0x7fc9ec03b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4006fd0 tx=0x7fc9f4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.239+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc9f8019de0 con 0x7fca08106090 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.497+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fca0819fa60 con 0x7fc9ec038f70 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.505+0000 7fc9ffe00700 1 -- 172.21.15.46:0/744669456 <== mgr.14120 v2:172.21.15.46:6800/4209254516 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fca0819fa60 con 0x7fc9ec038f70 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 msgr2=0x7fc9ec03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 0x7fc9ec03b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4006fd0 tx=0x7fc9f4006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 msgr2=0x7fca0819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:25.328 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9f8011b60 tx=0x7fc9f8011c40 comp rx=0 tx=0).stop 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 shutdown_connections 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7fc9ec038f70 0x7fc9ec03b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 --2- 172.21.15.46:0/744669456 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fca08106090 0x7fca0819db30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.508+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 >> 172.21.15.46:0/744669456 conn(0x7fca08101c70 msgr2=0x7fca08194690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.509+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 shutdown_connections 2024-03-08T22:56:25.329 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:24.509+0000 7fca0e000700 1 -- 172.21.15.46:0/744669456 wait complete. 2024-03-08T22:56:25.610 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:25 smithi046 ceph-mon[31859]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:25.610 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:25 smithi046 ceph-mon[31859]: Saving service mgr spec with placement count:2 2024-03-08T22:56:25.610 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:25 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:25.611 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:25 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:26.822 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:26 smithi046 ceph-mon[31859]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:56:26.822 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:26 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:26.822 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:26 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1697489282' entity='client.admin' 2024-03-08T22:56:26.832 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.922+0000 7f6943000700 1 Processor -- start 2024-03-08T22:56:26.833 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.922+0000 7f6943000700 1 -- start start 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.922+0000 7f6943000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.922+0000 7f6943000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c1081d0 con 0x7f693c1077e0 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.923+0000 7f693be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.923+0000 7f693be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.923+0000 7f693be00700 1 -- 172.21.15.46:0/3728682218 learned_addr learned my addr 172.21.15.46:0/3728682218 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:26.834 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.924+0000 7f693be00700 1 -- 172.21.15.46:0/3728682218 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c108990 con 0x7f693c1077e0 2024-03-08T22:56:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.924+0000 7f693be00700 1 --2- 172.21.15.46:0/3728682218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f692c0098a0 tx=0x7f692c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6177ec148b0ea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.924+0000 7f693aa00700 1 -- 172.21.15.46:0/3728682218 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f692c004160 con 0x7f693c1077e0 2024-03-08T22:56:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.925+0000 7f693aa00700 1 -- 172.21.15.46:0/3728682218 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692c004300 con 0x7f693c1077e0 2024-03-08T22:56:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.925+0000 7f693aa00700 1 -- 172.21.15.46:0/3728682218 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f692c011b20 con 0x7f693c1077e0 2024-03-08T22:56:26.835 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.925+0000 7f6943000700 1 -- 172.21.15.46:0/3728682218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 msgr2=0x7f693c107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:26.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.925+0000 7f6943000700 1 --2- 172.21.15.46:0/3728682218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f692c0098a0 tx=0x7f692c009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:26.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.926+0000 7f6943000700 1 -- 172.21.15.46:0/3728682218 shutdown_connections 2024-03-08T22:56:26.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.926+0000 7f6943000700 1 --2- 172.21.15.46:0/3728682218 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c107c00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:26.836 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.926+0000 7f6943000700 1 -- 172.21.15.46:0/3728682218 >> 172.21.15.46:0/3728682218 conn(0x7f693c102e20 msgr2=0x7f693c105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:26.840 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.926+0000 7f6943000700 1 -- 172.21.15.46:0/3728682218 shutdown_connections 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.926+0000 7f6943000700 1 -- 172.21.15.46:0/3728682218 wait complete. 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.927+0000 7f6943000700 1 Processor -- start 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.927+0000 7f6943000700 1 -- start start 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.927+0000 7f6943000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.927+0000 7f6943000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c10bd40 con 0x7f693c1077e0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.928+0000 7f693be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.928+0000 7f693be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36556/0 (socket says 172.21.15.46:36556) 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.928+0000 7f693be00700 1 -- 172.21.15.46:0/1697489282 learned_addr learned my addr 172.21.15.46:0/1697489282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.928+0000 7f693be00700 1 -- 172.21.15.46:0/1697489282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c19ddc0 con 0x7f693c1077e0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.928+0000 7f693be00700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f692c0117e0 tx=0x7f692c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.929+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f692c003bf0 con 0x7f693c1077e0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.929+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c19e050 con 0x7f693c1077e0 2024-03-08T22:56:26.841 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.929+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f692c01a910 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.929+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f692c024970 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.929+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c19e4b0 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.930+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f692c022020 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.930+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c063410 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.930+0000 7f6938c00700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 0x7f692003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.931+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f692c04d5d0 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.931+0000 7f693b400700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 0x7f692003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.931+0000 7f693b400700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 0x7f692003b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6928006fd0 tx=0x7f6928006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:25.937+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f692c017ca0 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.167+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f693c1a0e00 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.172+0000 7f6938c00700 1 -- 172.21.15.46:0/1697489282 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f692c01ab70 con 0x7f693c1077e0 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 msgr2=0x7f692003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:26.842 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 0x7f692003b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6928006fd0 tx=0x7f6928006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 msgr2=0x7f693c19d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f692c0117e0 tx=0x7f692c011ba0 comp rx=0 tx=0).stop 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 shutdown_connections 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f6920038f20 0x7f692003b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 --2- 172.21.15.46:0/1697489282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f693c1077e0 0x7f693c19d880 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.175+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 >> 172.21.15.46:0/1697489282 conn(0x7f693c102e20 msgr2=0x7f693c1037f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.176+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 shutdown_connections 2024-03-08T22:56:26.843 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:26.176+0000 7f6943000700 1 -- 172.21.15.46:0/1697489282 wait complete. 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.418+0000 7f026d600700 1 Processor -- start 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.419+0000 7f026d600700 1 -- start start 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.419+0000 7f026d600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.419+0000 7f026d600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0268107410 con 0x7f0268108e80 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.419+0000 7f0266a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:28.704 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.419+0000 7f0266a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36558/0 (socket says 172.21.15.46:36558) 2024-03-08T22:56:28.705 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.420+0000 7f0266a00700 1 -- 172.21.15.46:0/1764840339 learned_addr learned my addr 172.21.15.46:0/1764840339 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.420+0000 7f0266a00700 1 -- 172.21.15.46:0/1764840339 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0268107590 con 0x7f0268108e80 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.420+0000 7f0266a00700 1 --2- 172.21.15.46:0/1764840339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02580098a0 tx=0x7f0258009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f40f3863d6454563 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.421+0000 7f0265600700 1 -- 172.21.15.46:0/1764840339 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0258004160 con 0x7f0268108e80 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.421+0000 7f0265600700 1 -- 172.21.15.46:0/1764840339 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0258004300 con 0x7f0268108e80 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.421+0000 7f0265600700 1 -- 172.21.15.46:0/1764840339 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0258011b20 con 0x7f0268108e80 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.421+0000 7f026d600700 1 -- 172.21.15.46:0/1764840339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 msgr2=0x7f02681092a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.421+0000 7f026d600700 1 --2- 172.21.15.46:0/1764840339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f02580098a0 tx=0x7f0258009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.422+0000 7f026d600700 1 -- 172.21.15.46:0/1764840339 shutdown_connections 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.422+0000 7f026d600700 1 --2- 172.21.15.46:0/1764840339 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0268108e80 0x7f02681092a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:28.774 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.422+0000 7f026d600700 1 -- 172.21.15.46:0/1764840339 >> 172.21.15.46:0/1764840339 conn(0x7f0268102db0 msgr2=0x7f02681051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.422+0000 7f026d600700 1 -- 172.21.15.46:0/1764840339 shutdown_connections 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.422+0000 7f026d600700 1 -- 172.21.15.46:0/1764840339 wait complete. 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.423+0000 7f026d600700 1 Processor -- start 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.423+0000 7f026d600700 1 -- start start 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.423+0000 7f026d600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.423+0000 7f026d600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026810bd50 con 0x7f026819dc60 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.424+0000 7f0266a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.424+0000 7f0266a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36564/0 (socket says 172.21.15.46:36564) 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.424+0000 7f0266a00700 1 -- 172.21.15.46:0/3339152041 learned_addr learned my addr 172.21.15.46:0/3339152041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.424+0000 7f0266a00700 1 -- 172.21.15.46:0/3339152041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026819e5c0 con 0x7f026819dc60 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.424+0000 7f0266a00700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0258009fb0 tx=0x7f0258011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.425+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f02580036a0 con 0x7f026819dc60 2024-03-08T22:56:28.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.425+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f025801a460 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.425+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f025801b440 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.425+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026819e850 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.425+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02681a1330 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.426+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0248005330 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.427+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0258003840 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.428+0000 7f025f400700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 0x7f024c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.428+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f025804d460 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.428+0000 7f0266000700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 0x7f024c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.429+0000 7f0266000700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 0x7f024c03b490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0254006fd0 tx=0x7f0254006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.432+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f025804c260 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.658+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0248005620 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.664+0000 7f025f400700 1 -- 172.21.15.46:0/3339152041 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f025804c5e0 con 0x7f026819dc60 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 msgr2=0x7f024c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 0x7f024c03b490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0254006fd0 tx=0x7f0254006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:28.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 msgr2=0x7f026819e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:28.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0258009fb0 tx=0x7f0258011b80 comp rx=0 tx=0).stop 2024-03-08T22:56:28.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 shutdown_connections 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.667+0000 7f026d600700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f024c038fd0 0x7f024c03b490 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.668+0000 7f026d600700 1 --2- 172.21.15.46:0/3339152041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f026819dc60 0x7f026819e080 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.668+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 >> 172.21.15.46:0/3339152041 conn(0x7f0268102db0 msgr2=0x7f02681039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.668+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 shutdown_connections 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:27.668+0000 7f026d600700 1 -- 172.21.15.46:0/3339152041 wait complete. 2024-03-08T22:56:28.778 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling the dashboard module... 2024-03-08T22:56:28.872 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:28 smithi046 ceph-mon[31859]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:56:28.872 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:28 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3339152041' entity='client.admin' 2024-03-08T22:56:30.169 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setuser ceph since I am not root 2024-03-08T22:56:30.169 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setgroup ceph since I am not root 2024-03-08T22:56:30.169 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:29 smithi046 ceph-mon[31859]: from='mgr.14120 172.21.15.46:0/3337980244' entity='mgr.x' 2024-03-08T22:56:30.169 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:29 smithi046 ceph-mon[31859]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:56:30.169 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3206885948' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.311+0000 7f5e3a200700 1 Processor -- start 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.311+0000 7f5e3a200700 1 -- start start 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.312+0000 7f5e3a200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.312+0000 7f5e3a200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e341081e0 con 0x7f5e341077f0 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.312+0000 7f5e33400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.312+0000 7f5e33400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36566/0 (socket says 172.21.15.46:36566) 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.312+0000 7f5e33400700 1 -- 172.21.15.46:0/1779702337 learned_addr learned my addr 172.21.15.46:0/1779702337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.313+0000 7f5e33400700 1 -- 172.21.15.46:0/1779702337 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e341089a0 con 0x7f5e341077f0 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.313+0000 7f5e33400700 1 --2- 172.21.15.46:0/1779702337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e240098a0 tx=0x7f5e24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426a76732d6460f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.313+0000 7f5e32000700 1 -- 172.21.15.46:0/1779702337 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e24004160 con 0x7f5e341077f0 2024-03-08T22:56:30.494 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.313+0000 7f5e32000700 1 -- 172.21.15.46:0/1779702337 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e24004300 con 0x7f5e341077f0 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.313+0000 7f5e32000700 1 -- 172.21.15.46:0/1779702337 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e24011b20 con 0x7f5e341077f0 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 -- 172.21.15.46:0/1779702337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 msgr2=0x7f5e34107c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 --2- 172.21.15.46:0/1779702337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5e240098a0 tx=0x7f5e24009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 -- 172.21.15.46:0/1779702337 shutdown_connections 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 --2- 172.21.15.46:0/1779702337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e34107c10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 -- 172.21.15.46:0/1779702337 >> 172.21.15.46:0/1779702337 conn(0x7f5e34102db0 msgr2=0x7f5e341051d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.314+0000 7f5e3a200700 1 -- 172.21.15.46:0/1779702337 shutdown_connections 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.315+0000 7f5e3a200700 1 -- 172.21.15.46:0/1779702337 wait complete. 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e3a200700 1 Processor -- start 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e3a200700 1 -- start start 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e3a200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e3a200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3410bd50 con 0x7f5e341077f0 2024-03-08T22:56:30.495 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e33400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e33400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36574/0 (socket says 172.21.15.46:36574) 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.316+0000 7f5e33400700 1 -- 172.21.15.46:0/3206885948 learned_addr learned my addr 172.21.15.46:0/3206885948 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e33400700 1 -- 172.21.15.46:0/3206885948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3407d0b0 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e33400700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e24000c00 tx=0x7f5e24011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e24003b60 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e2401a910 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3407b2a0 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.317+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5e24024930 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.318+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3407b760 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.319+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34063410 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.319+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f5e24022020 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.320+0000 7f5e2be00700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 0x7f5e1803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.320+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f5e2404d170 con 0x7f5e341077f0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.320+0000 7f5e32a00700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 0x7f5e1803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:30.496 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.321+0000 7f5e32a00700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 0x7f5e1803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e20006fd0 tx=0x7f5e20006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.325+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5e2401ab70 con 0x7f5e341077f0 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.618+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5e3407b9e0 con 0x7f5e341077f0 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.825+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f5e2404cac0 con 0x7f5e341077f0 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.828+0000 7f5e2be00700 1 -- 172.21.15.46:0/3206885948 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5e2402d3f0 con 0x7f5e341077f0 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 msgr2=0x7f5e1803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 0x7f5e1803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5e20006fd0 tx=0x7f5e20006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 msgr2=0x7f5e3407cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5e24000c00 tx=0x7f5e24011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 shutdown_connections 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5e18038f20 0x7f5e1803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 --2- 172.21.15.46:0/3206885948 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e341077f0 0x7f5e3407cb70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.829+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 >> 172.21.15.46:0/3206885948 conn(0x7f5e34102db0 msgr2=0x7f5e341038a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.831+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 shutdown_connections 2024-03-08T22:56:30.497 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:29.831+0000 7f5e3a200700 1 -- 172.21.15.46:0/3206885948 wait complete. 2024-03-08T22:56:30.796 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:30.677+0000 7ff0bc2b6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T22:56:31.100 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:30 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3206885948' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-08T22:56:31.100 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:30 smithi046 ceph-mon[31859]: mgrmap e9: x(active, since 24s) 2024-03-08T22:56:31.100 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:30.805+0000 7ff0bc2b6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T22:56:31.726 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:31.635+0000 7ff0bc2b6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T22:56:32.083 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:31 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/164392899' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-08T22:56:32.084 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:31.763+0000 7ff0bc2b6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T22:56:32.084 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:31.948+0000 7ff0bc2b6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T22:56:32.084 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.078+0000 7ff0bc2b6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T22:56:32.095 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.094+0000 7f5977000700 1 Processor -- start 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.094+0000 7f5977000700 1 -- start start 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.095+0000 7f5977000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:32.106 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.095+0000 7f5977000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970107400 con 0x7f5970108e70 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.095+0000 7f596fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.096+0000 7f596fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36606/0 (socket says 172.21.15.46:36606) 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.096+0000 7f596fe00700 1 -- 172.21.15.46:0/4281806519 learned_addr learned my addr 172.21.15.46:0/4281806519 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.096+0000 7f596fe00700 1 -- 172.21.15.46:0/4281806519 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970107580 con 0x7f5970108e70 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.096+0000 7f596fe00700 1 --2- 172.21.15.46:0/4281806519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5960009e80 tx=0x7f596000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d56a3e148e7ddad3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f596f400700 1 -- 172.21.15.46:0/4281806519 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5960017070 con 0x7f5970108e70 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f596f400700 1 -- 172.21.15.46:0/4281806519 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f596000b8c0 con 0x7f5970108e70 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f596f400700 1 -- 172.21.15.46:0/4281806519 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59600045c0 con 0x7f5970108e70 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f5977000700 1 -- 172.21.15.46:0/4281806519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 msgr2=0x7f5970109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f5977000700 1 --2- 172.21.15.46:0/4281806519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5960009e80 tx=0x7f596000b330 comp rx=0 tx=0).stop 2024-03-08T22:56:32.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.097+0000 7f5977000700 1 -- 172.21.15.46:0/4281806519 shutdown_connections 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.098+0000 7f5977000700 1 --2- 172.21.15.46:0/4281806519 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f5970109290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.098+0000 7f5977000700 1 -- 172.21.15.46:0/4281806519 >> 172.21.15.46:0/4281806519 conn(0x7f5970102d80 msgr2=0x7f59701051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.098+0000 7f5977000700 1 -- 172.21.15.46:0/4281806519 shutdown_connections 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.098+0000 7f5977000700 1 -- 172.21.15.46:0/4281806519 wait complete. 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f5977000700 1 Processor -- start 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f5977000700 1 -- start start 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f5977000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f5977000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597010bd40 con 0x7f5970108e70 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f596fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f596fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36618/0 (socket says 172.21.15.46:36618) 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.099+0000 7f596fe00700 1 -- 172.21.15.46:0/164392899 learned_addr learned my addr 172.21.15.46:0/164392899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:32.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f596fe00700 1 -- 172.21.15.46:0/164392899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f597019dfa0 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f596fe00700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5960009530 tx=0x7f5960003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5960020020 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f596000bba0 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f597019e230 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.100+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5960017040 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.101+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59701a0f20 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.102+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f597010bec0 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.102+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5960023020 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.102+0000 7f596d600700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 0x7f595403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.102+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f596000f050 con 0x7f5970108e70 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.103+0000 7f5967400700 1 -- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 msgr2=0x7f595403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:32.109 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.103+0000 7f5967400700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 0x7f595403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.108+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f596002a020 con 0x7f5970108e70 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.304+0000 7f5967400700 1 -- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 msgr2=0x7f595403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.304+0000 7f5967400700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 0x7f595403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.391+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f59701a14a0 con 0x7f5970108e70 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.392+0000 7f596d600700 1 -- 172.21.15.46:0/164392899 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f596001b530 con 0x7f5970108e70 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 msgr2=0x7f595403b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 0x7f595403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 msgr2=0x7f597019da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5960009530 tx=0x7f5960003d00 comp rx=0 tx=0).stop 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 shutdown_connections 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f5954038f70 0x7f595403b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:32.110 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 --2- 172.21.15.46:0/164392899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5970108e70 0x7f597019da60 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.395+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 >> 172.21.15.46:0/164392899 conn(0x7f5970102d80 msgr2=0x7f59701038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.396+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 shutdown_connections 2024-03-08T22:56:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:31.396+0000 7f5977000700 1 -- 172.21.15.46:0/164392899 wait complete. 2024-03-08T22:56:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for the mgr to restart... 2024-03-08T22:56:32.111 INFO:teuthology.orchestra.run.smithi046.stdout:Waiting for mgr epoch 9... 2024-03-08T22:56:32.393 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.205+0000 7ff0bc2b6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T22:56:32.671 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.561+0000 7ff0bc2b6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T22:56:32.962 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.712+0000 7ff0bc2b6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T22:56:32.962 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.829+0000 7ff0bc2b6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T22:56:32.962 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:32 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:32.956+0000 7ff0bc2b6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T22:56:33.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:33 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:33.182+0000 7ff0bc2b6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T22:56:34.976 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:34 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:34.513+0000 7ff0bc2b6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T22:56:34.976 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:34 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:34.693+0000 7ff0bc2b6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T22:56:35.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:35 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:35.121+0000 7ff0bc2b6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T22:56:35.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:35 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:35.245+0000 7ff0bc2b6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T22:56:35.770 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:35 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:35.509+0000 7ff0bc2b6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:56:36.161 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:35 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:35.764+0000 7ff0bc2b6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T22:56:36.161 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:36 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:36.154+0000 7ff0bc2b6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:56:36.801 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:36 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:36.479+0000 7ff0bc2b6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T22:56:37.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:36 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:36.794+0000 7ff0bc2b6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T22:56:38.226 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:37.871+0000 7ff0bc2b6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T22:56:38.888 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:38 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:38.598+0000 7ff0bc2b6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T22:56:38.888 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:38 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:38.734+0000 7ff0bc2b6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T22:56:39.216 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:56:39.030+0000 7ff0bc2b6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: Active manager daemon x restarted 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: Activating manager daemon x 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: osdmap e3: 0 total, 0 up, 0 in 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: mgrmap e10: x(active, starting, since 0.00465889s) 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T22:56:39.217 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:39 smithi046 ceph-mon[31859]: Manager daemon x is now available 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:39] ENGINE Bus STARTING 2024-03-08T22:56:40.140 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:40 smithi046 ceph-mon[31859]: mgrmap e11: x(active, since 1.01165s) 2024-03-08T22:56:40.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout { 2024-03-08T22:56:40.959 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout } 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.666+0000 7f5615a00700 1 Processor -- start 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.666+0000 7f5615a00700 1 -- start start 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.667+0000 7f5615a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.667+0000 7f5615a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561010a210 con 0x7f5610109820 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.667+0000 7f560fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.668+0000 7f560fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36622/0 (socket says 172.21.15.46:36622) 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.668+0000 7f560fe00700 1 -- 172.21.15.46:0/1971617356 learned_addr learned my addr 172.21.15.46:0/1971617356 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.668+0000 7f560fe00700 1 -- 172.21.15.46:0/1971617356 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561010a9d0 con 0x7f5610109820 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.668+0000 7f560fe00700 1 --2- 172.21.15.46:0/1971617356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56000098a0 tx=0x7f5600009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9071de5185b057c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.669+0000 7f560ea00700 1 -- 172.21.15.46:0/1971617356 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5600004160 con 0x7f5610109820 2024-03-08T22:56:40.960 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.669+0000 7f560ea00700 1 -- 172.21.15.46:0/1971617356 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5600004300 con 0x7f5610109820 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.669+0000 7f560ea00700 1 -- 172.21.15.46:0/1971617356 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5600011b20 con 0x7f5610109820 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.669+0000 7f5615a00700 1 -- 172.21.15.46:0/1971617356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 msgr2=0x7f5610109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.669+0000 7f5615a00700 1 --2- 172.21.15.46:0/1971617356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f56000098a0 tx=0x7f5600009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.670+0000 7f5615a00700 1 -- 172.21.15.46:0/1971617356 shutdown_connections 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.670+0000 7f5615a00700 1 --2- 172.21.15.46:0/1971617356 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f5610109c40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.670+0000 7f5615a00700 1 -- 172.21.15.46:0/1971617356 >> 172.21.15.46:0/1971617356 conn(0x7f561007cd80 msgr2=0x7f56101074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.670+0000 7f5615a00700 1 -- 172.21.15.46:0/1971617356 shutdown_connections 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.670+0000 7f5615a00700 1 -- 172.21.15.46:0/1971617356 wait complete. 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.671+0000 7f5615a00700 1 Processor -- start 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.671+0000 7f5615a00700 1 -- start start 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.671+0000 7f5615a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.671+0000 7f5615a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561010dd80 con 0x7f5610109820 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.671+0000 7f560fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:40.961 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560fe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36626/0 (socket says 172.21.15.46:36626) 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560fe00700 1 -- 172.21.15.46:0/2082299596 learned_addr learned my addr 172.21.15.46:0/2082299596 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560fe00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561019dc70 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560fe00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5600000c00 tx=0x7f5600011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5600003b60 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f560001a910 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.672+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f561019df00 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.673+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5600024930 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.673+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f561019e360 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f5600022020 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560cc00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55f403bb40 con 0x7f55f4038f70 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f560004d180 con 0x7f5610109820 2024-03-08T22:56:40.962 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.674+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.875+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:32.875+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:33.276+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:33.276+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:34.077+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:34.077+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:35.679+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:35.679+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:38.882+0000 7f560f400700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/4209254516 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:38.882+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:39.037+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f560002d3f0 con 0x7f5610109820 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:39.037+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 msgr2=0x7f55f403b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:56:40.963 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:39.037+0000 7f560cc00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.040+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f5600018d00 con 0x7f5610109820 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.041+0000 7f560cc00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 0x7f55f403ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.041+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f55f403bb40 con 0x7f55f403c9f0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.042+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 0x7f55f403ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.044+0000 7f560f400700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 0x7f55f403ede0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55fc003a10 tx=0x7f55fc0092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.045+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f55f403bb40 con 0x7f55f403c9f0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.058+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5610063410 con 0x7f55f403c9f0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f560cc00700 1 -- 172.21.15.46:0/2082299596 <== mgr.14152 v2:172.21.15.46:6800/757291496 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f5610063410 con 0x7f55f403c9f0 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 msgr2=0x7f55f403ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 0x7f55f403ede0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f55fc003a10 tx=0x7f55fc0092b0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 msgr2=0x7f561019d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5600000c00 tx=0x7f5600011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 shutdown_connections 2024-03-08T22:56:40.964 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f55f403c9f0 0x7f55f403ede0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5610109820 0x7f561019d730 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 --2- 172.21.15.46:0/2082299596 >> [v2:172.21.15.46:6800/4209254516,v1:172.21.15.46:6801/4209254516] conn(0x7f55f4038f70 0x7f55f403b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.059+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 >> 172.21.15.46:0/2082299596 conn(0x7f561007cd80 msgr2=0x7f5610106c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.060+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 shutdown_connections 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:40.060+0000 7f5615a00700 1 -- 172.21.15.46:0/2082299596 wait complete. 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:mgr epoch 9 is available 2024-03-08T22:56:40.965 INFO:teuthology.orchestra.run.smithi046.stdout:Generating a dashboard self-signed certificate... 2024-03-08T22:56:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:41 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:39] ENGINE Serving on http://172.21.15.46:8765 2024-03-08T22:56:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:41 smithi046 ceph-mon[31859]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-08T22:56:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:41 smithi046 ceph-mon[31859]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-08T22:56:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:41 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:40] ENGINE Serving on https://172.21.15.46:7150 2024-03-08T22:56:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:41 smithi046 ceph-mon[31859]: [08/Mar/2024:22:56:40] ENGINE Bus STARTED 2024-03-08T22:56:42.360 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:42 smithi046 ceph-mon[31859]: mgrmap e12: x(active, since 2s) 2024-03-08T22:56:42.360 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:43.105 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-08T22:56:43.105 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.969+0000 7f5a10400700 1 Processor -- start 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.969+0000 7f5a10400700 1 -- start start 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.970+0000 7f5a10400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.970+0000 7f5a10400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a08107400 con 0x7f5a08108e70 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.971+0000 7f5a0da00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.971+0000 7f5a0da00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.971+0000 7f5a0da00700 1 -- 172.21.15.46:0/1914679758 learned_addr learned my addr 172.21.15.46:0/1914679758 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:43.107 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.971+0000 7f5a0da00700 1 -- 172.21.15.46:0/1914679758 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a08107580 con 0x7f5a08108e70 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.972+0000 7f5a0da00700 1 --2- 172.21.15.46:0/1914679758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0098a0 tx=0x7f59fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdc7ffe2d3ea783 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.972+0000 7f5a07e00700 1 -- 172.21.15.46:0/1914679758 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc004160 con 0x7f5a08108e70 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.972+0000 7f5a07e00700 1 -- 172.21.15.46:0/1914679758 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59fc004300 con 0x7f5a08108e70 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.972+0000 7f5a07e00700 1 -- 172.21.15.46:0/1914679758 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc011b20 con 0x7f5a08108e70 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 -- 172.21.15.46:0/1914679758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 msgr2=0x7f5a08109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 --2- 172.21.15.46:0/1914679758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f59fc0098a0 tx=0x7f59fc009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 -- 172.21.15.46:0/1914679758 shutdown_connections 2024-03-08T22:56:43.108 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 --2- 172.21.15.46:0/1914679758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a08109290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 -- 172.21.15.46:0/1914679758 >> 172.21.15.46:0/1914679758 conn(0x7f5a08102d80 msgr2=0x7f5a081051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 -- 172.21.15.46:0/1914679758 shutdown_connections 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.973+0000 7f5a10400700 1 -- 172.21.15.46:0/1914679758 wait complete. 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.975+0000 7f5a10400700 1 Processor -- start 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.975+0000 7f5a10400700 1 -- start start 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.975+0000 7f5a10400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.975+0000 7f5a10400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0810bd40 con 0x7f5a08108e70 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.976+0000 7f5a0da00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.976+0000 7f5a0da00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37524/0 (socket says 172.21.15.46:37524) 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.976+0000 7f5a0da00700 1 -- 172.21.15.46:0/4150415032 learned_addr learned my addr 172.21.15.46:0/4150415032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.976+0000 7f5a0da00700 1 -- 172.21.15.46:0/4150415032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0819bd80 con 0x7f5a08108e70 2024-03-08T22:56:43.113 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a0da00700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f59fc000c00 tx=0x7f59fc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc003b60 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59fc01a910 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc024930 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0819c010 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.977+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0819c470 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.978+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0810bec0 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.979+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f59fc022020 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.979+0000 7f5a06000700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.979+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f59fc04d1e0 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.980+0000 7f5a0d000700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.980+0000 7f5a0d000700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 0x7f59f003b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:41.985+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f59fc00b3e0 con 0x7f5a08108e70 2024-03-08T22:56:43.114 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.218+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a08104040 con 0x7f59f0038fd0 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.334+0000 7f5a06000700 1 -- 172.21.15.46:0/4150415032 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5a08104040 con 0x7f59f0038fd0 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.336+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 msgr2=0x7f59f003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.336+0000 7f5a10400700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 0x7f59f003b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59f4006fd0 tx=0x7f59f4006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 msgr2=0x7f5a0819b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f59fc000c00 tx=0x7f59fc011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 shutdown_connections 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 --2- 172.21.15.46:0/4150415032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5a08108e70 0x7f5a0819b840 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 >> 172.21.15.46:0/4150415032 conn(0x7f5a08102d80 msgr2=0x7f5a08103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 shutdown_connections 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:42.337+0000 7f5a10400700 1 -- 172.21.15.46:0/4150415032 wait complete. 2024-03-08T22:56:43.115 INFO:teuthology.orchestra.run.smithi046.stdout:Creating initial admin user... 2024-03-08T22:56:43.420 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:43 smithi046 ceph-mon[31859]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:43.420 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:43 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:43.420 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:43 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Peg7mFbfTLfJiCpF3duNo.P349JI2esI0fxmq3yV4b5APnbx3Llni", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709938604, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.723+0000 7f88eb600700 1 Processor -- start 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.723+0000 7f88eb600700 1 -- start start 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.724+0000 7f88eb600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.724+0000 7f88eb600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e407aef0 con 0x7f88e41082e0 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.724+0000 7f88e8c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.724+0000 7f88e8c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52564/0 (socket says 172.21.15.46:52564) 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.724+0000 7f88e8c00700 1 -- 172.21.15.46:0/4203767780 learned_addr learned my addr 172.21.15.46:0/4203767780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.725+0000 7f88e8c00700 1 -- 172.21.15.46:0/4203767780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e410ac10 con 0x7f88e41082e0 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.725+0000 7f88e8c00700 1 --2- 172.21.15.46:0/4203767780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009e80 tx=0x7f88d800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c097f30b6df151 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.726+0000 7f88e3400700 1 -- 172.21.15.46:0/4203767780 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88d8017070 con 0x7f88e41082e0 2024-03-08T22:56:45.081 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.726+0000 7f88e3400700 1 -- 172.21.15.46:0/4203767780 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d800b8c0 con 0x7f88e41082e0 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.726+0000 7f88e3400700 1 -- 172.21.15.46:0/4203767780 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88d80045c0 con 0x7f88e41082e0 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.726+0000 7f88eb600700 1 -- 172.21.15.46:0/4203767780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 msgr2=0x7f88e410a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.726+0000 7f88eb600700 1 --2- 172.21.15.46:0/4203767780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009e80 tx=0x7f88d800b330 comp rx=0 tx=0).stop 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.727+0000 7f88eb600700 1 -- 172.21.15.46:0/4203767780 shutdown_connections 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.727+0000 7f88eb600700 1 --2- 172.21.15.46:0/4203767780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e410a6d0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.727+0000 7f88eb600700 1 -- 172.21.15.46:0/4203767780 >> 172.21.15.46:0/4203767780 conn(0x7f88e4101c70 msgr2=0x7f88e4104090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.727+0000 7f88eb600700 1 -- 172.21.15.46:0/4203767780 shutdown_connections 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.727+0000 7f88eb600700 1 -- 172.21.15.46:0/4203767780 wait complete. 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.728+0000 7f88eb600700 1 Processor -- start 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.728+0000 7f88eb600700 1 -- start start 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88eb600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:45.082 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88eb600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88e410bc70 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88e8c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88e8c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52574/0 (socket says 172.21.15.46:52574) 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88e8c00700 1 -- 172.21.15.46:0/3758771557 learned_addr learned my addr 172.21.15.46:0/3758771557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88e8c00700 1 -- 172.21.15.46:0/3758771557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e41080a0 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.729+0000 7f88e8c00700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009fd0 tx=0x7f88d8003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.730+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88d8020020 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.730+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88d800bba0 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.730+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e4106290 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.730+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f88d8017040 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.730+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e41067d0 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.731+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88e4063410 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.731+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f88d8023020 con 0x7f88e41082e0 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.732+0000 7f88e1600700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 0x7f88cc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:45.083 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.732+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f88d800f050 con 0x7f88e41082e0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.732+0000 7f88e3e00700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 0x7f88cc03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.733+0000 7f88e3e00700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 0x7f88cc03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88d0006fd0 tx=0x7f88d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.737+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f88d802a020 con 0x7f88e41082e0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:43.992+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f88e4106e90 con 0x7f88cc038fd0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.260+0000 7f88e1600700 1 -- 172.21.15.46:0/3758771557 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f88e4106e90 con 0x7f88cc038fd0 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 msgr2=0x7f88cc03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 0x7f88cc03b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f88d0006fd0 tx=0x7f88d0006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 msgr2=0x7f88e4107b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f88d8009fd0 tx=0x7f88d8003d00 comp rx=0 tx=0).stop 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 shutdown_connections 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f88cc038fd0 0x7f88cc03b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 --2- 172.21.15.46:0/3758771557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f88e41082e0 0x7f88e4107b60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.264+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 >> 172.21.15.46:0/3758771557 conn(0x7f88e4101c70 msgr2=0x7f88e41027c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:45.084 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.265+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 shutdown_connections 2024-03-08T22:56:45.085 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:44.265+0000 7f88eb600700 1 -- 172.21.15.46:0/3758771557 wait complete. 2024-03-08T22:56:45.085 INFO:teuthology.orchestra.run.smithi046.stdout:Fetching dashboard port number... 2024-03-08T22:56:45.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:44 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:45.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:44 smithi046 ceph-mon[31859]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:45.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:44 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:46.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:45 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3932191300' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-08T22:56:47.403 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:46 smithi046 ceph-mon[31859]: mgrmap e13: x(active, since 6s) 2024-03-08T22:56:47.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stdout 8443 2024-03-08T22:56:47.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.672+0000 7faf10c00700 1 Processor -- start 2024-03-08T22:56:47.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.672+0000 7faf10c00700 1 -- start start 2024-03-08T22:56:47.412 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.672+0000 7faf10c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:47.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.672+0000 7faf10c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c107400 con 0x7faf0c108e70 2024-03-08T22:56:47.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.673+0000 7faf09e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:47.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.673+0000 7faf09e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52578/0 (socket says 172.21.15.46:52578) 2024-03-08T22:56:47.417 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.673+0000 7faf09e00700 1 -- 172.21.15.46:0/2189638056 learned_addr learned my addr 172.21.15.46:0/2189638056 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.673+0000 7faf09e00700 1 -- 172.21.15.46:0/2189638056 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c107580 con 0x7faf0c108e70 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.673+0000 7faf09e00700 1 --2- 172.21.15.46:0/2189638056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faefc0098a0 tx=0x7faefc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89527650e463fed1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.674+0000 7faf08a00700 1 -- 172.21.15.46:0/2189638056 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc004160 con 0x7faf0c108e70 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.674+0000 7faf08a00700 1 -- 172.21.15.46:0/2189638056 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faefc004300 con 0x7faf0c108e70 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.674+0000 7faf08a00700 1 -- 172.21.15.46:0/2189638056 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc011b20 con 0x7faf0c108e70 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.674+0000 7faf10c00700 1 -- 172.21.15.46:0/2189638056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 msgr2=0x7faf0c109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.674+0000 7faf10c00700 1 --2- 172.21.15.46:0/2189638056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7faefc0098a0 tx=0x7faefc009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.675+0000 7faf10c00700 1 -- 172.21.15.46:0/2189638056 shutdown_connections 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.675+0000 7faf10c00700 1 --2- 172.21.15.46:0/2189638056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c109290 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.675+0000 7faf10c00700 1 -- 172.21.15.46:0/2189638056 >> 172.21.15.46:0/2189638056 conn(0x7faf0c102d80 msgr2=0x7faf0c1051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.675+0000 7faf10c00700 1 -- 172.21.15.46:0/2189638056 shutdown_connections 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.675+0000 7faf10c00700 1 -- 172.21.15.46:0/2189638056 wait complete. 2024-03-08T22:56:47.418 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.676+0000 7faf10c00700 1 Processor -- start 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.676+0000 7faf10c00700 1 -- start start 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.676+0000 7faf10c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.676+0000 7faf10c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c10bd40 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.677+0000 7faf09e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.677+0000 7faf09e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52590/0 (socket says 172.21.15.46:52590) 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.677+0000 7faf09e00700 1 -- 172.21.15.46:0/3932191300 learned_addr learned my addr 172.21.15.46:0/3932191300 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.677+0000 7faf09e00700 1 -- 172.21.15.46:0/3932191300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c07d0b0 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.677+0000 7faf09e00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.678+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc003b60 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.678+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faefc01a910 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.678+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0c07b2a0 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.678+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c07b760 con 0x7faf0c108e70 2024-03-08T22:56:47.419 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.678+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7faefc024930 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.679+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7faefc022020 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.680+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c10bec0 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.680+0000 7faf02a00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 0x7faef003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.680+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faefc04d270 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.684+0000 7faf09400700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 0x7faef003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.685+0000 7faf09400700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 0x7faef003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faef8006fd0 tx=0x7faef8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.691+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7faefc01ab70 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.921+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7faf0c07baa0 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.922+0000 7faf02a00700 1 -- 172.21.15.46:0/3932191300 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7faefc02d3f0 con 0x7faf0c108e70 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 msgr2=0x7faef003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 0x7faef003b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7faef8006fd0 tx=0x7faef8006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 msgr2=0x7faf0c07cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faefc000c00 tx=0x7faefc011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:47.420 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 shutdown_connections 2024-03-08T22:56:47.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faef0038f70 0x7faef003b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:47.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 --2- 172.21.15.46:0/3932191300 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf0c108e70 0x7faf0c07cb70 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:47.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 >> 172.21.15.46:0/3932191300 conn(0x7faf0c102d80 msgr2=0x7faf0c103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:47.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.925+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 shutdown_connections 2024-03-08T22:56:47.421 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:45.926+0000 7faf10c00700 1 -- 172.21.15.46:0/3932191300 wait complete. 2024-03-08T22:56:47.444 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T22:56:47.444 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout disabled 2024-03-08T22:56:47.469 INFO:teuthology.orchestra.run.smithi046.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T22:56:47.469 INFO:teuthology.orchestra.run.smithi046.stdout:systemctl: stdout inactive 2024-03-08T22:56:47.469 INFO:teuthology.orchestra.run.smithi046.stdout:firewalld.service is not enabled 2024-03-08T22:56:47.469 INFO:teuthology.orchestra.run.smithi046.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout:Ceph Dashboard is now available at: 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: URL: https://smithi046.front.sepia.ceph.com:8443/ 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: User: admin 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: Password: 8268d5jqbd 2024-03-08T22:56:47.476 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:47.478 INFO:teuthology.orchestra.run.smithi046.stdout:Saving cluster configuration to /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config directory 2024-03-08T22:56:47.480 INFO:teuthology.orchestra.run.smithi046.stdout:Enabling autotune for osd_memory_target 2024-03-08T22:56:49.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.099+0000 7fde67600700 1 Processor -- start 2024-03-08T22:56:49.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.099+0000 7fde67600700 1 -- start start 2024-03-08T22:56:49.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.100+0000 7fde67600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:49.070 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.100+0000 7fde67600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde601070b0 con 0x7fde601066c0 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.100+0000 7fde64c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.101+0000 7fde64c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52592/0 (socket says 172.21.15.46:52592) 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.101+0000 7fde64c00700 1 -- 172.21.15.46:0/564092317 learned_addr learned my addr 172.21.15.46:0/564092317 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.101+0000 7fde64c00700 1 -- 172.21.15.46:0/564092317 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde601078c0 con 0x7fde601066c0 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.101+0000 7fde64c00700 1 --2- 172.21.15.46:0/564092317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde5400bb40 tx=0x7fde5400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9c2f8fa57e8bce server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:49.071 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.102+0000 7fde5f400700 1 -- 172.21.15.46:0/564092317 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde5400dbe0 con 0x7fde601066c0 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.102+0000 7fde5f400700 1 -- 172.21.15.46:0/564092317 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde54004510 con 0x7fde601066c0 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.102+0000 7fde5f400700 1 -- 172.21.15.46:0/564092317 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde54010470 con 0x7fde601066c0 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 -- 172.21.15.46:0/564092317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 msgr2=0x7fde60106ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 --2- 172.21.15.46:0/564092317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde5400bb40 tx=0x7fde5400be50 comp rx=0 tx=0).stop 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 -- 172.21.15.46:0/564092317 shutdown_connections 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 --2- 172.21.15.46:0/564092317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde60106ae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:49.072 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 -- 172.21.15.46:0/564092317 >> 172.21.15.46:0/564092317 conn(0x7fde60101cc0 msgr2=0x7fde601040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 -- 172.21.15.46:0/564092317 shutdown_connections 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.103+0000 7fde67600700 1 -- 172.21.15.46:0/564092317 wait complete. 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.104+0000 7fde67600700 1 Processor -- start 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde67600700 1 -- start start 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde67600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde67600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde60079fd0 con 0x7fde601066c0 2024-03-08T22:56:49.073 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde64c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde64c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52604/0 (socket says 172.21.15.46:52604) 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.105+0000 7fde64c00700 1 -- 172.21.15.46:0/860917365 learned_addr learned my addr 172.21.15.46:0/860917365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde64c00700 1 -- 172.21.15.46:0/860917365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6019de50 con 0x7fde601066c0 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde64c00700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde540046e0 tx=0x7fde540040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde54010720 con 0x7fde601066c0 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fde5401d070 con 0x7fde601066c0 2024-03-08T22:56:49.074 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fde54021b10 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.106+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6019e0e0 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.107+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde601a0dd0 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.108+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fde5401f030 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.108+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde60063410 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.108+0000 7fde5d600700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 0x7fde4803b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.108+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fde5404d210 con 0x7fde601066c0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.108+0000 7fde5fe00700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 0x7fde4803b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:49.075 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.109+0000 7fde5fe00700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 0x7fde4803b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde4c006fd0 tx=0x7fde4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.114+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fde54015070 con 0x7fde601066c0 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.354+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fde60196e70 con 0x7fde601066c0 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.355+0000 7fde5d600700 1 -- 172.21.15.46:0/860917365 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fde60196e70 con 0x7fde601066c0 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 msgr2=0x7fde4803b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 0x7fde4803b520 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fde4c006fd0 tx=0x7fde4c006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 msgr2=0x7fde6019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fde540046e0 tx=0x7fde540040b0 comp rx=0 tx=0).stop 2024-03-08T22:56:49.076 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 shutdown_connections 2024-03-08T22:56:49.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fde48039060 0x7fde4803b520 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:49.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 --2- 172.21.15.46:0/860917365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fde601066c0 0x7fde6019d910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:49.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 >> 172.21.15.46:0/860917365 conn(0x7fde60101cc0 msgr2=0x7fde60193f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:49.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.358+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 shutdown_connections 2024-03-08T22:56:49.077 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:48.359+0000 7fde67600700 1 -- 172.21.15.46:0/860917365 wait complete. 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.664+0000 7f2ce0200700 1 Processor -- start 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.664+0000 7f2ce0200700 1 -- start start 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.664+0000 7f2ce0200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.665+0000 7f2ce0200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd810a3f0 con 0x7f2cd8109a00 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.665+0000 7f2cdd800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.665+0000 7f2cdd800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52620/0 (socket says 172.21.15.46:52620) 2024-03-08T22:56:50.769 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.665+0000 7f2cdd800700 1 -- 172.21.15.46:0/1841598794 learned_addr learned my addr 172.21.15.46:0/1841598794 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:50.770 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.666+0000 7f2cdd800700 1 -- 172.21.15.46:0/1841598794 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd810abb0 con 0x7f2cd8109a00 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.666+0000 7f2cdd800700 1 --2- 172.21.15.46:0/1841598794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098a0 tx=0x7f2ccc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe021ae78064141a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.666+0000 7f2cd7e00700 1 -- 172.21.15.46:0/1841598794 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc004160 con 0x7f2cd8109a00 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.666+0000 7f2cd7e00700 1 -- 172.21.15.46:0/1841598794 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc004300 con 0x7f2cd8109a00 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.666+0000 7f2cd7e00700 1 -- 172.21.15.46:0/1841598794 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc011b20 con 0x7f2cd8109a00 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 -- 172.21.15.46:0/1841598794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 msgr2=0x7f2cd8109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 --2- 172.21.15.46:0/1841598794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc0098a0 tx=0x7f2ccc009bb0 comp rx=0 tx=0).stop 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 -- 172.21.15.46:0/1841598794 shutdown_connections 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 --2- 172.21.15.46:0/1841598794 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd8109e20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 -- 172.21.15.46:0/1841598794 >> 172.21.15.46:0/1841598794 conn(0x7f2cd807cec0 msgr2=0x7f2cd81074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.667+0000 7f2ce0200700 1 -- 172.21.15.46:0/1841598794 shutdown_connections 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.668+0000 7f2ce0200700 1 -- 172.21.15.46:0/1841598794 wait complete. 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.669+0000 7f2ce0200700 1 Processor -- start 2024-03-08T22:56:50.775 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.669+0000 7f2ce0200700 1 -- start start 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.669+0000 7f2ce0200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.669+0000 7f2ce0200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd810df60 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.670+0000 7f2cdd800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.670+0000 7f2cdd800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52634/0 (socket says 172.21.15.46:52634) 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.670+0000 7f2cdd800700 1 -- 172.21.15.46:0/3351626459 learned_addr learned my addr 172.21.15.46:0/3351626459 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.670+0000 7f2cdd800700 1 -- 172.21.15.46:0/3351626459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd819dde0 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.670+0000 7f2cdd800700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc000c00 tx=0x7f2ccc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.671+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc003b60 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.671+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ccc01a910 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.671+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2ccc024930 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.671+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd819e070 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.671+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd819e4d0 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.672+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd8063410 con 0x7f2cd8109a00 2024-03-08T22:56:50.776 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.672+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f2ccc022020 con 0x7f2cd8109a00 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.673+0000 7f2cd6000700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 0x7f2cc003b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.673+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2ccc04d2b0 con 0x7f2cd8109a00 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.673+0000 7f2cdce00700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 0x7f2cc003b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.674+0000 7f2cdce00700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 0x7f2cc003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006fd0 tx=0x7f2cc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:49.678+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2ccc01ab70 con 0x7f2cd8109a00 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.032+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f2cd81a0ea0 con 0x7f2cd8109a00 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.037+0000 7f2cd6000700 1 -- 172.21.15.46:0/3351626459 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (secure 0 0 0) 0x7f2ccc02d3f0 con 0x7f2cd8109a00 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 msgr2=0x7f2cc003b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 0x7f2cc003b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006fd0 tx=0x7f2cc4006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 msgr2=0x7f2cd819d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc000c00 tx=0x7f2ccc011c10 comp rx=0 tx=0).stop 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 shutdown_connections 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2cc0039070 0x7f2cc003b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:50.777 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 --2- 172.21.15.46:0/3351626459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2cd8109a00 0x7f2cd819d8a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:50.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 >> 172.21.15.46:0/3351626459 conn(0x7f2cd807cec0 msgr2=0x7f2cd8105b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:50.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 shutdown_connections 2024-03-08T22:56:50.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr 2024-03-08T22:56:50.040+0000 7f2ce0200700 1 -- 172.21.15.46:0/3351626459 wait complete. 2024-03-08T22:56:50.778 INFO:teuthology.orchestra.run.smithi046.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout:Or, if you are only running a single cluster on this host: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: ceph telemetry on 2024-03-08T22:56:50.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.780 INFO:teuthology.orchestra.run.smithi046.stdout:For more information see: 2024-03-08T22:56:50.780 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.780 INFO:teuthology.orchestra.run.smithi046.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-08T22:56:50.780 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:56:50.780 INFO:teuthology.orchestra.run.smithi046.stdout:Bootstrap complete. 2024-03-08T22:56:50.914 INFO:tasks.cephadm:Fetching config... 2024-03-08T22:56:50.914 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:56:50.914 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-08T22:56:50.966 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-08T22:56:50.966 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:56:50.966 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-08T22:56:51.008 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-08T22:56:51.009 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:56:51.009 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-08T22:56:51.145 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-08T22:56:51.145 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:56:51.145 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-08T22:56:51.198 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-08T22:56:51.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T22:56:51.294 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:51 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3351626459' entity='client.admin' 2024-03-08T22:56:51.335 INFO:teuthology.orchestra.run.smithi046.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:56:51.400 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T22:56:51.454 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:56:51.472 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T22:56:51.518 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDO8lrbjRalx40K+6AXA9IVNBJnlVRxUqrjXZDhbV3ni2OdzGgcBHYF5A/KVRZG83ZgRAx/Zw5bzDRkxnk4SAzJWrsmYTqDIDHeZVNx5m3lZem5mkLyxsTGstzqGKyiTEk17oZRBZ9iiK0fE4/TzH7WM/VXxcnougJqm+Rud0exI0leYIu7NkCRU+6bJ3866Yb8Bm8RYQ2w/0VrkOFNwM4zKpE4wkFaboup+AAMpB7Amqq8HHkDacSQuiv8CP3A3TpJmp5k/99218IgMmBvU3xarItfC5GUYvZm2sSACILsEbIOuSOtORBiMuuxe5+zH/4ClQDJG3QCHtr+PLQxuUX5Q67uObyTj1aBPKsJICaAkDvvLkYT28WyngWpXvC8AokSfAzoAbw5oscYhT3iMEufoKmjGZ9V28m4squptFJJMCeuF3ekbKaNRba0+9sH2O4MgkJWGpcTXeD9QtpMP92H/G6u22NW4cQW9bpt/mc5jp2j5Tm1n4csc663RnlFM20= ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:56:51.533 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-08T22:56:51.972 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:56:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 -- 172.21.15.46:0/61419289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 msgr2=0x7f4d8c0741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 --2- 172.21.15.46:0/61419289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c0741f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009910 tx=0x7f4d7c009c20 comp rx=0 tx=0).stop 2024-03-08T22:56:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 -- 172.21.15.46:0/61419289 shutdown_connections 2024-03-08T22:56:54.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 --2- 172.21.15.46:0/61419289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c0741f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:54.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 -- 172.21.15.46:0/61419289 >> 172.21.15.46:0/61419289 conn(0x7f4d8c0fcdd0 msgr2=0x7f4d8c0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:54.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 -- 172.21.15.46:0/61419289 shutdown_connections 2024-03-08T22:56:54.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.456+0000 7f4d91000700 1 -- 172.21.15.46:0/61419289 wait complete. 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.457+0000 7f4d91000700 1 Processor -- start 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.457+0000 7f4d91000700 1 -- start start 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d91000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d91000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8c109ab0 con 0x7f4d8c075d90 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d89e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55992/0 (socket says 172.21.15.46:55992) 2024-03-08T22:56:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d89e00700 1 -- 172.21.15.46:0/4028267547 learned_addr learned my addr 172.21.15.46:0/4028267547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.458+0000 7f4d89e00700 1 -- 172.21.15.46:0/4028267547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d8c199640 con 0x7f4d8c075d90 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d89e00700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c005950 tx=0x7f4d7c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d7c026070 con 0x7f4d8c075d90 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d7c00bc30 con 0x7f4d8c075d90 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4d7c003b30 con 0x7f4d8c075d90 2024-03-08T22:56:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8c1998d0 con 0x7f4d8c075d90 2024-03-08T22:56:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.459+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8c19c5c0 con 0x7f4d8c075d90 2024-03-08T22:56:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.460+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4d7c029070 con 0x7f4d8c075d90 2024-03-08T22:56:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.461+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8c10a230 con 0x7f4d8c075d90 2024-03-08T22:56:54.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.461+0000 7f4d82a00700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 0x7f4d7003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:54.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.461+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4d7c01e070 con 0x7f4d8c075d90 2024-03-08T22:56:54.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.461+0000 7f4d89400700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 0x7f4d7003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:54.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.462+0000 7f4d89400700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 0x7f4d7003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006fd0 tx=0x7f4d78006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.467+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4d7c013070 con 0x7f4d8c075d90 2024-03-08T22:56:54.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.693+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f4d8c02d940 con 0x7f4d8c075d90 2024-03-08T22:56:54.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.696+0000 7f4d82a00700 1 -- 172.21.15.46:0/4028267547 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4d7c030020 con 0x7f4d8c075d90 2024-03-08T22:56:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.700+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 msgr2=0x7f4d7003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.700+0000 7f4d91000700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 0x7f4d7003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006fd0 tx=0x7f4d78006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.700+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 msgr2=0x7f4d8c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:54.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.700+0000 7f4d91000700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c005950 tx=0x7f4d7c005030 comp rx=0 tx=0).stop 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 shutdown_connections 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4d70039020 0x7f4d7003b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 --2- 172.21.15.46:0/4028267547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4d8c075d90 0x7f4d8c199100 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 >> 172.21.15.46:0/4028267547 conn(0x7f4d8c0fcdd0 msgr2=0x7f4d8c0fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 shutdown_connections 2024-03-08T22:56:54.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:54.702+0000 7f4d91000700 1 -- 172.21.15.46:0/4028267547 wait complete. 2024-03-08T22:56:54.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi046", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4028267547' entity='client.admin' 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:54.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:55.501 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-08T22:56:55.501 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-08T22:56:55.956 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:56:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.933+0000 7efdef000700 1 -- 172.21.15.46:0/1048913632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 msgr2=0x7efde81083c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.933+0000 7efdef000700 1 --2- 172.21.15.46:0/1048913632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde81083c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efdd800b0a0 tx=0x7efdd800b3b0 comp rx=0 tx=0).stop 2024-03-08T22:56:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.934+0000 7efdef000700 1 -- 172.21.15.46:0/1048913632 shutdown_connections 2024-03-08T22:56:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.934+0000 7efdef000700 1 --2- 172.21.15.46:0/1048913632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde81083c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:57.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.934+0000 7efdef000700 1 -- 172.21.15.46:0/1048913632 >> 172.21.15.46:0/1048913632 conn(0x7efde8076ed0 msgr2=0x7efde8079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.934+0000 7efdef000700 1 -- 172.21.15.46:0/1048913632 shutdown_connections 2024-03-08T22:56:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.934+0000 7efdef000700 1 -- 172.21.15.46:0/1048913632 wait complete. 2024-03-08T22:56:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.935+0000 7efdef000700 1 Processor -- start 2024-03-08T22:56:57.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.935+0000 7efdef000700 1 -- start start 2024-03-08T22:56:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.935+0000 7efdef000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.935+0000 7efdef000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde810de60 con 0x7efde8107fa0 2024-03-08T22:56:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.936+0000 7efde7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.936+0000 7efde7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56022/0 (socket says 172.21.15.46:56022) 2024-03-08T22:56:57.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.936+0000 7efde7e00700 1 -- 172.21.15.46:0/2901061157 learned_addr learned my addr 172.21.15.46:0/2901061157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.936+0000 7efde7e00700 1 -- 172.21.15.46:0/2901061157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde819d9a0 con 0x7efde8107fa0 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.936+0000 7efde7e00700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdd8000f80 tx=0x7efdd8007670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.937+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdd8017050 con 0x7efde8107fa0 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.937+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdd8023070 con 0x7efde8107fa0 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.937+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde819dc30 con 0x7efde8107fa0 2024-03-08T22:56:57.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.937+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdd8004190 con 0x7efde8107fa0 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.937+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde81a0920 con 0x7efde8107fa0 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.938+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7efdd8008d00 con 0x7efde8107fa0 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.938+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde810e5e0 con 0x7efde8107fa0 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.938+0000 7efde5600700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 0x7efdcc03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.938+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efdd800f050 con 0x7efde8107fa0 2024-03-08T22:56:57.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.939+0000 7efddf400700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 0x7efdcc03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:56:57.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.939+0000 7efddf400700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 0x7efdcc03b4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006fd0 tx=0x7efdd0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:56:57.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:57.944+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efdd80269b0 con 0x7efde8107fa0 2024-03-08T22:56:58.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.170+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7efde802d940 con 0x7efdcc039010 2024-03-08T22:56:58.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.178+0000 7efde5600700 1 -- 172.21.15.46:0/2901061157 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efde802d940 con 0x7efdcc039010 2024-03-08T22:56:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 msgr2=0x7efdcc03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 0x7efdcc03b4d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efdd0006fd0 tx=0x7efdd0006e40 comp rx=0 tx=0).stop 2024-03-08T22:56:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 msgr2=0x7efde819d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:56:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdd8000f80 tx=0x7efdd8007670 comp rx=0 tx=0).stop 2024-03-08T22:56:58.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 shutdown_connections 2024-03-08T22:56:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efdcc039010 0x7efdcc03b4d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 --2- 172.21.15.46:0/2901061157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efde8107fa0 0x7efde819d460 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:56:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 >> 172.21.15.46:0/2901061157 conn(0x7efde8076ed0 msgr2=0x7efde8077880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:56:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 shutdown_connections 2024-03-08T22:56:58.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:56:58.181+0000 7efdef000700 1 -- 172.21.15.46:0/2901061157 wait complete. 2024-03-08T22:56:58.945 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi062 2024-03-08T22:56:58.945 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:56:58.945 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/etc/ceph/ceph.conf 2024-03-08T22:56:58.964 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:56:58.964 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:56:59.023 INFO:tasks.cephadm:Adding host smithi062 to orchestrator... 2024-03-08T22:56:59.023 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch host add smithi062 2024-03-08T22:56:59.461 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:56:59.503 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T22:56:59.504 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:56:59 smithi046 ceph-mon[31859]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:00.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:00 smithi046 ceph-mon[31859]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:00.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:00 smithi046 ceph-mon[31859]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 -- 172.21.15.46:0/2274398739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 msgr2=0x7facc010e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 --2- 172.21.15.46:0/2274398739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc010e0e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7facb0009910 tx=0x7facb0009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 -- 172.21.15.46:0/2274398739 shutdown_connections 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 --2- 172.21.15.46:0/2274398739 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc010e0e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 -- 172.21.15.46:0/2274398739 >> 172.21.15.46:0/2274398739 conn(0x7facc00773c0 msgr2=0x7facc00797e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.563+0000 7facbf200700 1 -- 172.21.15.46:0/2274398739 shutdown_connections 2024-03-08T22:57:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.564+0000 7facbf200700 1 -- 172.21.15.46:0/2274398739 wait complete. 2024-03-08T22:57:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.565+0000 7facbf200700 1 Processor -- start 2024-03-08T22:57:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.565+0000 7facbf200700 1 -- start start 2024-03-08T22:57:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.565+0000 7facbf200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.565+0000 7facbf200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facc0112250 con 0x7facc010dd00 2024-03-08T22:57:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.566+0000 7facbde00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.566+0000 7facbde00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56040/0 (socket says 172.21.15.46:56040) 2024-03-08T22:57:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.566+0000 7facbde00700 1 -- 172.21.15.46:0/3635335555 learned_addr learned my addr 172.21.15.46:0/3635335555 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.566+0000 7facbde00700 1 -- 172.21.15.46:0/3635335555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc019db60 con 0x7facc010dd00 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.566+0000 7facbde00700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facb0006010 tx=0x7facb0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.567+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facb0026070 con 0x7facc010dd00 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.567+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7facb000bbd0 con 0x7facc010dd00 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.567+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facc019de50 con 0x7facc010dd00 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.567+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facb00039d0 con 0x7facc010dd00 2024-03-08T22:57:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.567+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facc01a1ef0 con 0x7facc010dd00 2024-03-08T22:57:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.568+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7facb0029070 con 0x7facc010dd00 2024-03-08T22:57:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.568+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc0109340 con 0x7facc010dd00 2024-03-08T22:57:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.569+0000 7facb6a00700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 0x7faca403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.569+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7facb001e070 con 0x7facc010dd00 2024-03-08T22:57:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.569+0000 7facbd400700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 0x7faca403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.570+0000 7facbd400700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 0x7faca403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facac006fd0 tx=0x7facac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:01.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.574+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7facb0013070 con 0x7facc010dd00 2024-03-08T22:57:01.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:01.806+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi062", "target": ["mon-mgr", ""]}) v1 -- 0x7facc019e980 con 0x7faca4038fc0 2024-03-08T22:57:02.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:01 smithi046 ceph-mon[31859]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T22:57:02.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:02.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:02.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:02 smithi046 ceph-mon[31859]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:02 smithi046 ceph-mon[31859]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:04.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:03 smithi046 ceph-mon[31859]: Deploying cephadm binary to smithi062 2024-03-08T22:57:05.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:04 smithi046 ceph-mon[31859]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:06.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.841+0000 7facb6a00700 1 -- 172.21.15.46:0/3635335555 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7facc019e980 con 0x7faca4038fc0 2024-03-08T22:57:06.846 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi062' with addr '172.21.15.62' 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 msgr2=0x7faca403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 0x7faca403b480 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7facac006fd0 tx=0x7facac006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 msgr2=0x7facc019d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7facb0006010 tx=0x7facb0005030 comp rx=0 tx=0).stop 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 shutdown_connections 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faca4038fc0 0x7faca403b480 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 --2- 172.21.15.46:0/3635335555 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7facc010dd00 0x7facc019d620 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.845+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 >> 172.21.15.46:0/3635335555 conn(0x7facc00773c0 msgr2=0x7facc00777a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.846+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 shutdown_connections 2024-03-08T22:57:06.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:06.846+0000 7facbf200700 1 -- 172.21.15.46:0/3635335555 wait complete. 2024-03-08T22:57:07.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:06 smithi046 ceph-mon[31859]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:07.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:07.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:07.597 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-08T22:57:08.047 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:08.192 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:07 smithi046 ceph-mon[31859]: Added host smithi062 2024-03-08T22:57:08.192 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:08 smithi046 ceph-mon[31859]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:10.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 -- 172.21.15.46:0/3969229727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 msgr2=0x7f15ec109dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 --2- 172.21.15.46:0/3969229727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec109dd0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009910 tx=0x7f15dc009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 -- 172.21.15.46:0/3969229727 shutdown_connections 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 --2- 172.21.15.46:0/3969229727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec109dd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 -- 172.21.15.46:0/3969229727 >> 172.21.15.46:0/3969229727 conn(0x7f15ec0ff4c0 msgr2=0x7f15ec1018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 -- 172.21.15.46:0/3969229727 shutdown_connections 2024-03-08T22:57:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.174+0000 7f15f2800700 1 -- 172.21.15.46:0/3969229727 wait complete. 2024-03-08T22:57:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.175+0000 7f15f2800700 1 Processor -- start 2024-03-08T22:57:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.175+0000 7f15f2800700 1 -- start start 2024-03-08T22:57:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15f2800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15f2800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec10e030 con 0x7f15ec1099f0 2024-03-08T22:57:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15eb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15eb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59606/0 (socket says 172.21.15.46:59606) 2024-03-08T22:57:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15eb400700 1 -- 172.21.15.46:0/3830622982 learned_addr learned my addr 172.21.15.46:0/3830622982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.176+0000 7f15eb400700 1 -- 172.21.15.46:0/3830622982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec199a10 con 0x7f15ec1099f0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15eb400700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f15dc006010 tx=0x7f15dc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15dc026070 con 0x7f15ec1099f0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec199d00 con 0x7f15ec1099f0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15dc00bbd0 con 0x7f15ec1099f0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15dc003910 con 0x7f15ec1099f0 2024-03-08T22:57:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.177+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec19dda0 con 0x7f15ec1099f0 2024-03-08T22:57:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.178+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f15dc029070 con 0x7f15ec1099f0 2024-03-08T22:57:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.178+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec105130 con 0x7f15ec1099f0 2024-03-08T22:57:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.179+0000 7f15e3e00700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 0x7f15d003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.179+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f15dc01e070 con 0x7f15ec1099f0 2024-03-08T22:57:10.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.179+0000 7f15eaa00700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 0x7f15d003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:10.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.180+0000 7f15eaa00700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 0x7f15d003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f15d8006fd0 tx=0x7f15d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:10.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.185+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f15dc013070 con 0x7f15ec1099f0 2024-03-08T22:57:10.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.415+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15ec19a830 con 0x7f15d0039020 2024-03-08T22:57:10.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.418+0000 7f15e3e00700 1 -- 172.21.15.46:0/3830622982 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f15ec19a830 con 0x7f15d0039020 2024-03-08T22:57:10.422 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:57:10.422 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}] 2024-03-08T22:57:10.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.420+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 msgr2=0x7f15d003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:10.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.420+0000 7f15f2800700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 0x7f15d003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f15d8006fd0 tx=0x7f15d8006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.420+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 msgr2=0x7f15ec1994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.420+0000 7f15f2800700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f15dc006010 tx=0x7f15dc005030 comp rx=0 tx=0).stop 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 shutdown_connections 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f15d0039020 0x7f15d003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 --2- 172.21.15.46:0/3830622982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f15ec1099f0 0x7f15ec1994d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 >> 172.21.15.46:0/3830622982 conn(0x7f15ec0ff4c0 msgr2=0x7f15ec10b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 shutdown_connections 2024-03-08T22:57:10.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:10.421+0000 7f15f2800700 1 -- 172.21.15.46:0/3830622982 wait complete. 2024-03-08T22:57:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:10 smithi046 ceph-mon[31859]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:10 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:11.149 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-03-08T22:57:11.149 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:57:11.149 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-03-08T22:57:11.168 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:57:11.168 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:11.225 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-03-08T22:57:11.226 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch host add smithi192 2024-03-08T22:57:11.669 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:11.964 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:11 smithi046 ceph-mon[31859]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T22:57:12.845 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:12 smithi046 ceph-mon[31859]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 -- 172.21.15.46:0/2222178711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 msgr2=0x7f4294100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 --2- 172.21.15.46:0/2222178711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294100960 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4284009910 tx=0x7f4284009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 -- 172.21.15.46:0/2222178711 shutdown_connections 2024-03-08T22:57:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 --2- 172.21.15.46:0/2222178711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294100960 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 -- 172.21.15.46:0/2222178711 >> 172.21.15.46:0/2222178711 conn(0x7f4294077320 msgr2=0x7f42940fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 -- 172.21.15.46:0/2222178711 shutdown_connections 2024-03-08T22:57:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.691+0000 7f4299600700 1 -- 172.21.15.46:0/2222178711 wait complete. 2024-03-08T22:57:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.693+0000 7f4299600700 1 Processor -- start 2024-03-08T22:57:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.693+0000 7f4299600700 1 -- start start 2024-03-08T22:57:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.693+0000 7f4299600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.693+0000 7f4299600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429410e040 con 0x7f4294100580 2024-03-08T22:57:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.694+0000 7f4292a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.694+0000 7f4292a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49212/0 (socket says 172.21.15.46:49212) 2024-03-08T22:57:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.694+0000 7f4292a00700 1 -- 172.21.15.46:0/2934464887 learned_addr learned my addr 172.21.15.46:0/2934464887 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.694+0000 7f4292a00700 1 -- 172.21.15.46:0/2934464887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4294199a40 con 0x7f4294100580 2024-03-08T22:57:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.694+0000 7f4292a00700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f4284005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:13.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.695+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4284026070 con 0x7f4294100580 2024-03-08T22:57:13.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.695+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f428400bc30 con 0x7f4294100580 2024-03-08T22:57:13.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.695+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42840039d0 con 0x7f4294100580 2024-03-08T22:57:13.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.695+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4294199cd0 con 0x7f4294100580 2024-03-08T22:57:13.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.695+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429419a130 con 0x7f4294100580 2024-03-08T22:57:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.696+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f4284029070 con 0x7f4294100580 2024-03-08T22:57:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.697+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4294069a30 con 0x7f4294100580 2024-03-08T22:57:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.697+0000 7f428b400700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 0x7f427803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.697+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f428401e070 con 0x7f4294100580 2024-03-08T22:57:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.697+0000 7f4292000700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 0x7f427803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:13.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.698+0000 7f4292000700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 0x7f427803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4280006fd0 tx=0x7f4280006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:13.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.704+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4284013070 con 0x7f4294100580 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:13.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:13 smithi046 ceph-mon[31859]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T22:57:13.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:13.990+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) v1 -- 0x7f429419a8d0 con 0x7f4278039020 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:15 smithi046 ceph-mon[31859]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:15 smithi046 ceph-mon[31859]: Deploying cephadm binary to smithi192 2024-03-08T22:57:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:16 smithi046 ceph-mon[31859]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:18.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.834+0000 7f428b400700 1 -- 172.21.15.46:0/2934464887 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f429419a8d0 con 0x7f4278039020 2024-03-08T22:57:18.839 INFO:teuthology.orchestra.run.smithi046.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-03-08T22:57:18.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 msgr2=0x7f427803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 0x7f427803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4280006fd0 tx=0x7f4280006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 msgr2=0x7f4294199500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4284006010 tx=0x7f4284005030 comp rx=0 tx=0).stop 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 shutdown_connections 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4278039020 0x7f427803b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 --2- 172.21.15.46:0/2934464887 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4294100580 0x7f4294199500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 >> 172.21.15.46:0/2934464887 conn(0x7f4294077320 msgr2=0x7f42940fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.838+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 shutdown_connections 2024-03-08T22:57:18.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:18.839+0000 7f4299600700 1 -- 172.21.15.46:0/2934464887 wait complete. 2024-03-08T22:57:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:18 smithi046 ceph-mon[31859]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:19.645 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-08T22:57:20.109 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:20.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:19 smithi046 ceph-mon[31859]: Added host smithi192 2024-03-08T22:57:20.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:19 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:20.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:19 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:21.168 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:20 smithi046 ceph-mon[31859]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 -- 172.21.15.46:0/2076024634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 msgr2=0x7fa0d8107a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 --2- 172.21.15.46:0/2076024634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d8107a70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8009910 tx=0x7fa0c8009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 -- 172.21.15.46:0/2076024634 shutdown_connections 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 --2- 172.21.15.46:0/2076024634 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d8107a70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 -- 172.21.15.46:0/2076024634 >> 172.21.15.46:0/2076024634 conn(0x7fa0d80fd110 msgr2=0x7fa0d80ff530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 -- 172.21.15.46:0/2076024634 shutdown_connections 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.946+0000 7fa0dd400700 1 -- 172.21.15.46:0/2076024634 wait complete. 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.947+0000 7fa0dd400700 1 Processor -- start 2024-03-08T22:57:21.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0dd400700 1 -- start start 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0dd400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0dd400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0d810bc40 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0d7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0d7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49232/0 (socket says 172.21.15.46:49232) 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.948+0000 7fa0d7e00700 1 -- 172.21.15.46:0/1889616874 learned_addr learned my addr 172.21.15.46:0/1889616874 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0d7e00700 1 -- 172.21.15.46:0/1889616874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0d81997f0 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0d7e00700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004f80 tx=0x7fa0c8005060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0c8026070 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0c800b6f0 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0d8199ae0 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.949+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0c8003910 con 0x7fa0d8107690 2024-03-08T22:57:21.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.950+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0d819db80 con 0x7fa0d8107690 2024-03-08T22:57:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.951+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0d8102d40 con 0x7fa0d8107690 2024-03-08T22:57:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.951+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fa0c8029070 con 0x7fa0d8107690 2024-03-08T22:57:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.951+0000 7fa0d4c00700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 0x7fa0bc03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.951+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa0c801e070 con 0x7fa0d8107690 2024-03-08T22:57:21.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.951+0000 7fa0d7400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 0x7fa0bc03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:21.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.952+0000 7fa0d7400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 0x7fa0bc03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4006fd0 tx=0x7fa0c4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:21.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:21.957+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa0c8013070 con 0x7fa0d8107690 2024-03-08T22:57:22.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.214+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa0d819a610 con 0x7fa0bc038fc0 2024-03-08T22:57:22.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.215+0000 7fa0d4c00700 1 -- 172.21.15.46:0/1889616874 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7fa0d819a610 con 0x7fa0bc038fc0 2024-03-08T22:57:22.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T22:57:22.220 INFO:teuthology.orchestra.run.smithi046.stdout:[{"addr": "172.21.15.46", "hostname": "smithi046", "labels": [], "status": ""}, {"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 msgr2=0x7fa0bc03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 0x7fa0bc03b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4006fd0 tx=0x7fa0c4006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 msgr2=0x7fa0d81992b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa0c8004f80 tx=0x7fa0c8005060 comp rx=0 tx=0).stop 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 shutdown_connections 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0bc038fc0 0x7fa0bc03b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 --2- 172.21.15.46:0/1889616874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0d8107690 0x7fa0d81992b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:22.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 >> 172.21.15.46:0/1889616874 conn(0x7fa0d80fd110 msgr2=0x7fa0d8075a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 shutdown_connections 2024-03-08T22:57:22.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:22.218+0000 7fa0dd400700 1 -- 172.21.15.46:0/1889616874 wait complete. 2024-03-08T22:57:22.874 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:22 smithi046 ceph-mon[31859]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:22.874 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:22.949 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-08T22:57:22.950 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-08T22:57:23.405 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:23.844 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:23 smithi046 ceph-mon[31859]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T22:57:24.948 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:24.949 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:25.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.486+0000 7fb6bf200700 1 -- 172.21.15.46:0/3227015126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 msgr2=0x7fb6b80745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:25.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.486+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3227015126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b80745f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8009910 tx=0x7fb6a8009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:25.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.486+0000 7fb6bf200700 1 -- 172.21.15.46:0/3227015126 shutdown_connections 2024-03-08T22:57:25.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.486+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3227015126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b80745f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.486+0000 7fb6bf200700 1 -- 172.21.15.46:0/3227015126 >> 172.21.15.46:0/3227015126 conn(0x7fb6b80fd2c0 msgr2=0x7fb6b80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.487+0000 7fb6bf200700 1 -- 172.21.15.46:0/3227015126 shutdown_connections 2024-03-08T22:57:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.487+0000 7fb6bf200700 1 -- 172.21.15.46:0/3227015126 wait complete. 2024-03-08T22:57:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.488+0000 7fb6bf200700 1 Processor -- start 2024-03-08T22:57:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.488+0000 7fb6bf200700 1 -- start start 2024-03-08T22:57:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6bf200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6bf200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6b810e060 con 0x7fb6b8074210 2024-03-08T22:57:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6b7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6b7e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37920/0 (socket says 172.21.15.46:37920) 2024-03-08T22:57:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6b7e00700 1 -- 172.21.15.46:0/3217499817 learned_addr learned my addr 172.21.15.46:0/3217499817 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:25.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.489+0000 7fb6b7e00700 1 -- 172.21.15.46:0/3217499817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6b8199ab0 con 0x7fb6b8074210 2024-03-08T22:57:25.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6b7e00700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006010 tx=0x7fb6a8005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:25.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6a8026070 con 0x7fb6b8074210 2024-03-08T22:57:25.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6a800bc30 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6b8199d40 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6a80039a0 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.490+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6b819a1a0 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.491+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb6a8029070 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.492+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6b80758c0 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.492+0000 7fb6b5600700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 0x7fb69c03b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.492+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb6a801e070 con 0x7fb6b8074210 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.492+0000 7fb6afe00700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 0x7fb69c03b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.493+0000 7fb6afe00700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 0x7fb69c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:25.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.498+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb6a8013070 con 0x7fb6b8074210 2024-03-08T22:57:25.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:25.732+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb6b819a8d0 con 0x7fb6b8074210 2024-03-08T22:57:25.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:25 smithi046 ceph-mon[31859]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T22:57:25.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:25 smithi046 ceph-mon[31859]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:26.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.663+0000 7fb6b5600700 1 -- 172.21.15.46:0/3217499817 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb6a8030090 con 0x7fb6b8074210 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 msgr2=0x7fb69c03b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 0x7fb69c03b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0006fd0 tx=0x7fb6a0006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 msgr2=0x7fb6b8199570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb6a8006010 tx=0x7fb6a8005030 comp rx=0 tx=0).stop 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 shutdown_connections 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb69c039020 0x7fb69c03b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 --2- 172.21.15.46:0/3217499817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6b8074210 0x7fb6b8199570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 >> 172.21.15.46:0/3217499817 conn(0x7fb6b80fd2c0 msgr2=0x7fb6b8105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.667+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 shutdown_connections 2024-03-08T22:57:26.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:26.668+0000 7fb6bf200700 1 -- 172.21.15.46:0/3217499817 wait complete. 2024-03-08T22:57:26.673 INFO:teuthology.orchestra.run.smithi046.stderr:adjusted tunables profile to default 2024-03-08T22:57:26.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:26.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:26.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T22:57:26.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3217499817' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-08T22:57:26.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:26.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:26.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:27.411 INFO:tasks.cephadm:Adding mon.a on smithi046 2024-03-08T22:57:27.411 INFO:tasks.cephadm:Adding mon.b on smithi062 2024-03-08T22:57:27.411 INFO:tasks.cephadm:Adding mon.c on smithi192 2024-03-08T22:57:27.412 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c' 2024-03-08T22:57:27.650 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:27.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:27 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3217499817' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-08T22:57:27.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:27 smithi046 ceph-mon[31859]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:28.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:28 smithi046 ceph-mon[31859]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- 172.21.15.192:0/3690119962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 msgr2=0x7f42f8106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/3690119962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8106ab0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f42f4009910 tx=0x7f42f4009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- 172.21.15.192:0/3690119962 shutdown_connections 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/3690119962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8106ab0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- 172.21.15.192:0/3690119962 >> 172.21.15.192:0/3690119962 conn(0x7f42f80fc1e0 msgr2=0x7f42f80fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:29.150 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- 172.21.15.192:0/3690119962 shutdown_connections 2024-03-08T22:57:29.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- 172.21.15.192:0/3690119962 wait complete. 2024-03-08T22:57:29.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 Processor -- start 2024-03-08T22:57:29.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.145+0000 7f42ffe8a700 1 -- start start 2024-03-08T22:57:29.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.146+0000 7f42ffe8a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:29.151 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.146+0000 7f42ffe8a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42f810ad10 con 0x7f42f81066d0 2024-03-08T22:57:29.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.146+0000 7f42fdc26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:29.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.146+0000 7f42fdc26700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.192:44010/0 (socket says 172.21.15.192:44010) 2024-03-08T22:57:29.152 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.146+0000 7f42fdc26700 1 -- 172.21.15.192:0/2677609869 learned_addr learned my addr 172.21.15.192:0/2677609869 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.147+0000 7f42fdc26700 1 -- 172.21.15.192:0/2677609869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42f8199a70 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.147+0000 7f42fdc26700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42f4005850 tx=0x7f42f4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.147+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42f4026070 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.147+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42f8199d60 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.147+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42f819de00 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.148+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f42f400b6f0 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.148+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f42f4030060 con 0x7f42f81066d0 2024-03-08T22:57:29.153 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.148+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42dc005330 con 0x7f42f81066d0 2024-03-08T22:57:29.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.149+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f42f4029070 con 0x7f42f81066d0 2024-03-08T22:57:29.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.149+0000 7f42eeffd700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 0x7f42e403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:29.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.149+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f42f401e070 con 0x7f42f81066d0 2024-03-08T22:57:29.154 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.150+0000 7f42fd425700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 0x7f42e403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:29.155 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.151+0000 7f42fd425700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 0x7f42e403b430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006fd0 tx=0x7f42e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:29.169 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.164+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f42f4013070 con 0x7f42f81066d0 2024-03-08T22:57:29.380 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.375+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f42dc000c80 con 0x7f42e4038f70 2024-03-08T22:57:29.395 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.391+0000 7f42eeffd700 1 -- 172.21.15.192:0/2677609869 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f42dc000c80 con 0x7f42e4038f70 2024-03-08T22:57:29.395 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-03-08T22:57:29.397 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 msgr2=0x7f42e403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 0x7f42e403b430 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f42e8006fd0 tx=0x7f42e8006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 msgr2=0x7f42f8199530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f42f4005850 tx=0x7f42f4005030 comp rx=0 tx=0).stop 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 shutdown_connections 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f42e4038f70 0x7f42e403b430 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 --2- 172.21.15.192:0/2677609869 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f42f81066d0 0x7f42f8199530 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:29.398 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 >> 172.21.15.192:0/2677609869 conn(0x7f42f80fc1e0 msgr2=0x7f42f8069bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:29.399 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 shutdown_connections 2024-03-08T22:57:29.399 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:29.392+0000 7f42ffe8a700 1 -- 172.21.15.192:0/2677609869 wait complete. 2024-03-08T22:57:29.889 DEBUG:teuthology.orchestra.run.smithi062:mon.b> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.b.service 2024-03-08T22:57:29.891 DEBUG:teuthology.orchestra.run.smithi192:mon.c> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.c.service 2024-03-08T22:57:29.894 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T22:57:29.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T22:57:30.635 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c;count:3 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:30.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:30 smithi046 ceph-mon[31859]: Deploying daemon mon.c on smithi192 2024-03-08T22:57:32.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:32 smithi046 ceph-mon[31859]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 -- 172.21.15.192:0/921241521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 msgr2=0x7fb6c0106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/921241521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c0106a80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4009910 tx=0x7fb6b4009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 -- 172.21.15.192:0/921241521 shutdown_connections 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/921241521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c0106a80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 -- 172.21.15.192:0/921241521 >> 172.21.15.192:0/921241521 conn(0x7fb6c00fc190 msgr2=0x7fb6c00fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 -- 172.21.15.192:0/921241521 shutdown_connections 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.525+0000 7fb6c71a9700 1 -- 172.21.15.192:0/921241521 wait complete. 2024-03-08T22:57:33.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.526+0000 7fb6c71a9700 1 Processor -- start 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.526+0000 7fb6c71a9700 1 -- start start 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.526+0000 7fb6c71a9700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.526+0000 7fb6c71a9700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6c010ace0 con 0x7fb6c01066a0 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.527+0000 7fb6c4f45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.527+0000 7fb6c4f45700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.192:32820/0 (socket says 172.21.15.192:32820) 2024-03-08T22:57:33.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.527+0000 7fb6c4f45700 1 -- 172.21.15.192:0/3829901423 learned_addr learned my addr 172.21.15.192:0/3829901423 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.527+0000 7fb6c4f45700 1 -- 172.21.15.192:0/3829901423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c00fe530 con 0x7fb6c01066a0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6c4f45700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005850 tx=0x7fb6b4005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b4026070 con 0x7fb6c01066a0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c00fcbe0 con 0x7fb6c01066a0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c00fd0a0 con 0x7fb6c01066a0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6b400b6f0 con 0x7fb6c01066a0 2024-03-08T22:57:33.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.528+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb6b4030060 con 0x7fb6c01066a0 2024-03-08T22:57:33.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.529+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4005330 con 0x7fb6c01066a0 2024-03-08T22:57:33.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.530+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb6b4029070 con 0x7fb6c01066a0 2024-03-08T22:57:33.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.530+0000 7fb6bdffb700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 0x7fb6b003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:33.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.530+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb6b401e070 con 0x7fb6c01066a0 2024-03-08T22:57:33.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.531+0000 7fb6bffff700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 0x7fb6b003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:33.536 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.531+0000 7fb6bffff700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 0x7fb6b003b430 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006fd0 tx=0x7fb6ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:33.549 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.544+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb6b4013070 con 0x7fb6c01066a0 2024-03-08T22:57:33.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.841+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6a40058f0 con 0x7fb6c01066a0 2024-03-08T22:57:33.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.842+0000 7fb6bdffb700 1 -- 172.21.15.192:0/3829901423 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fb6b402b440 con 0x7fb6c01066a0 2024-03-08T22:57:33.847 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:57:33.847 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","modified":"2024-03-08T22:55:28.027578Z","created":"2024-03-08T22:55:28.027578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 msgr2=0x7fb6b003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 0x7fb6b003b430 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac006fd0 tx=0x7fb6ac006e40 comp rx=0 tx=0).stop 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 msgr2=0x7fb6c00fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb6b4005850 tx=0x7fb6b4005030 comp rx=0 tx=0).stop 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 shutdown_connections 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb6b0038f70 0x7fb6b003b430 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 --2- 172.21.15.192:0/3829901423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb6c01066a0 0x7fb6c00fc640 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:33.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 >> 172.21.15.192:0/3829901423 conn(0x7fb6c00fc190 msgr2=0x7fb6c0069a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:33.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.844+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 shutdown_connections 2024-03-08T22:57:33.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:33.845+0000 7fb6c71a9700 1 -- 172.21.15.192:0/3829901423 wait complete. 2024-03-08T22:57:33.850 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-03-08T22:57:34.530 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 systemd[1]: Starting Ceph mon.c for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:57:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:34 smithi046 ceph-mon[31859]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:34 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/3829901423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:34.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 podman[36682]: 2024-03-08 22:57:34.727647245 +0000 UTC m=+0.094606794 container create 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T22:57:34.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 podman[36682]: 2024-03-08 22:57:34.651258339 +0000 UTC m=+0.018217888 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 podman[36682]: 2024-03-08 22:57:34.946982251 +0000 UTC m=+0.313941808 container init 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c, maintainer=Guillaume Abrioux , ceph=True, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 podman[36682]: 2024-03-08 22:57:34.949870307 +0000 UTC m=+0.316829857 container start 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: set uid:gid to 167:167 (ceph:ceph) 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev), process ceph-mon, pid 2 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: pidfile_write: ignore empty --pid-file 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: load: jerasure load: lrc 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: RocksDB version: 7.9.2 2024-03-08T22:57:35.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Git sha 0 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Compile date 2024-03-08 15:06:51 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: DB SUMMARY 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: DB Session ID: JVGGIIFU6X7R9K2S9WTR 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: CURRENT file: CURRENT 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: IDENTITY file: IDENTITY 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.error_if_exists: 0 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.create_if_missing: 0 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.paranoid_checks: 1 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-08T22:57:35.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.env: 0x5618e1061b20 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.fs: PosixFileSystem 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.info_log: 0x5618e18e78e0 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_file_opening_threads: 16 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.statistics: (nil) 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.use_fsync: 0 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_log_file_size: 0 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.keep_log_file_num: 1000 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.recycle_log_file_num: 0 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_fallocate: 1 2024-03-08T22:57:35.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_mmap_reads: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_mmap_writes: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.use_direct_reads: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.create_missing_column_families: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.db_log_dir: 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.wal_dir: 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.advise_random_on_open: 1 2024-03-08T22:57:35.318 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.db_write_buffer_size: 0 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.write_buffer_manager: 0x5618e18bb400 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.rate_limiter: (nil) 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.wal_recovery_mode: 2 2024-03-08T22:57:35.319 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enable_thread_tracking: 0 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enable_pipelined_write: 0 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.unordered_write: 0 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.row_cache: None 2024-03-08T22:57:35.320 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.wal_filter: None 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_ingest_behind: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.two_write_queues: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.manual_wal_flush: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.wal_compression: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.atomic_flush: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.log_readahead_size: 0 2024-03-08T22:57:35.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.best_efforts_recovery: 0 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.allow_data_in_errors: 0 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.db_host_id: __hostname__ 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_background_jobs: 2 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_background_compactions: -1 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_subcompactions: 1 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_total_wal_size: 0 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-08T22:57:35.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_open_files: -1 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bytes_per_sync: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_readahead_size: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_background_flushes: -1 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Compression algorithms supported: 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kZSTD supported: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kXpressCompression supported: 0 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kLZ4HCCompression supported: 1 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kZlibCompression supported: 1 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kSnappyCompression supported: 1 2024-03-08T22:57:35.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kLZ4Compression supported: 1 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: kBZip2Compression supported: 0 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.merge_operator: 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_filter: None 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_filter_factory: None 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.sst_partitioner_factory: None 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5618e18e79e0) 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-08T22:57:35.325 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_type: 0 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_index_type: 0 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_shortening: 1 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: checksum: 4 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: no_block_cache: 0 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache: 0x5618e18ed090 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-03-08T22:57:35.326 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_options: 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: capacity : 536870912 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_shard_bits : 4 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: strict_capacity_limit : 0 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_cache_compressed: (nil) 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: persistent_cache: (nil) 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size: 4096 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_size_deviation: 10 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_restart_interval: 16 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: index_block_restart_interval: 1 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: metadata_block_size: 4096 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: partition_filters: 0 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: use_delta_encoding: 1 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: filter_policy: bloomfilter 2024-03-08T22:57:35.327 INFO:journalctl@ceph.mon.c.smithi192.stdout: whole_key_filtering: 1 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: verify_compression: 0 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: format_version: 5 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_index_compression: 1 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: block_align: 0 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: max_auto_readahead_size: 262144 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: prepopulate_block_cache: 0 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: initial_auto_readahead_size: 8192 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout: num_file_reads_for_auto_readahead: 2 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.write_buffer_size: 33554432 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_write_buffer_number: 2 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression: NoCompression 2024-03-08T22:57:35.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression: Disabled 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.prefix_extractor: nullptr 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.num_levels: 7 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-08T22:57:35.329 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.level: 32767 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.strategy: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.enabled: false 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-08T22:57:35.330 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.target_file_size_base: 67108864 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-08T22:57:35.331 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.arena_block_size: 1048576 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.disable_auto_compactions: 0 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-08T22:57:35.332 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.table_properties_collectors: 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.inplace_update_support: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.bloom_locality: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.max_successive_merges: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.paranoid_file_checks: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.force_consistency_checks: 1 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.report_bg_io_stats: 0 2024-03-08T22:57:35.333 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.ttl: 2592000 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enable_blob_files: false 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.min_blob_size: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_file_size: 268435456 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.blob_file_starting_level: 0 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-08T22:57:35.334 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 339f8cae-bb5b-4e15-9dff-0e6e6888b050 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938654979538, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938654979946, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709938654, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "339f8cae-bb5b-4e15-9dff-0e6e6888b050", "db_session_id": "JVGGIIFU6X7R9K2S9WTR", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709938654980045, "job": 1, "event": "recovery_finished"} 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5618e19e0000 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: DB pointer 0x5618e19ca000 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-08T22:57:35.335 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** DB Stats ** 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T22:57:35.336 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** Compaction Stats [default] ** 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T22:57:35.337 INFO:journalctl@ceph.mon.c.smithi192.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache BinnedLRUCache@0x5618e18ed090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: using public_addr v2:172.21.15.192:0/0 -> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: starting mon.c rank -1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:34 smithi192 ceph-mon[36700]: mon.c@-1(???) e0 preinit fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).mds e1 new map 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: e1 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T22:57:35.338 INFO:journalctl@ceph.mon.c.smithi192.stdout: legacy client fscid: -1 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout: No filesystems configured 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T22:57:35.339 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Added host smithi062 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:35.340 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Deploying cephadm binary to smithi192 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.341 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Added host smithi192 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.342 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3217499817' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3217499817' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:35.343 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Saving service mon spec with placement smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c;count:3 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: Deploying daemon mon.c on smithi192 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/3829901423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 ceph-mon[36700]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-08T22:57:35.344 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 bash[36682]: 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 2024-03-08T22:57:35.345 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:35 smithi192 systemd[1]: Started Ceph mon.c for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:57:35.559 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T22:57:35.559 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T22:57:35.796 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T22:57:40.279 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 -- 172.21.15.192:0/1800089324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03b80035c0 msgr2=0x7f03b8005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 --2- 172.21.15.192:0/1800089324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03b80035c0 0x7f03b8005a50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f03d01017f0 tx=0x7f03c000feb0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 -- 172.21.15.192:0/1800089324 shutdown_connections 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 --2- 172.21.15.192:0/1800089324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03b80035c0 0x7f03b8005a50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 -- 172.21.15.192:0/1800089324 >> 172.21.15.192:0/1800089324 conn(0x7f03d00fd410 msgr2=0x7f03d00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 -- 172.21.15.192:0/1800089324 shutdown_connections 2024-03-08T22:57:40.280 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 -- 172.21.15.192:0/1800089324 wait complete. 2024-03-08T22:57:40.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.275+0000 7f03d5823700 1 Processor -- start 2024-03-08T22:57:40.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03d5823700 1 -- start start 2024-03-08T22:57:40.281 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03d5823700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03d5823700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 0x7f03d019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03d5823700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d0109960 con 0x7f03d01979a0 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03d5823700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01097e0 con 0x7f03b80035c0 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03ceffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03ceffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:44400/0 (socket says 172.21.15.192:44400) 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.276+0000 7f03ceffd700 1 -- 172.21.15.192:0/380286569 learned_addr learned my addr 172.21.15.192:0/380286569 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ceffd700 1 -- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 msgr2=0x7f03d0197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ceffd700 1 -- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 msgr2=0x7f03d0197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ceffd700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ceffd700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:57:40.282 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ce7fc700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 0x7f03d019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:40.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ce7fc700 1 -- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 msgr2=0x7f03d0197460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:57:40.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ce7fc700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.277+0000 7f03ce7fc700 1 -- 172.21.15.192:0/380286569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d0197fe0 con 0x7f03d01979a0 2024-03-08T22:57:40.283 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03ce7fc700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 0x7f03d019bd30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03c400b4d0 tx=0x7f03c400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f03c4011630 con 0x7f03d01979a0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c4011cb0 con 0x7f03d01979a0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f03c4010e60 con 0x7f03d01979a0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d019c270 con 0x7f03d01979a0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.278+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d019c7d0 con 0x7f03d01979a0 2024-03-08T22:57:40.284 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.280+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f03c4018020 con 0x7f03d01979a0 2024-03-08T22:57:40.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.280+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d0103000 con 0x7f03d01979a0 2024-03-08T22:57:40.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.280+0000 7f03d4821700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 0x7f03bc03b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:40.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.280+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f03c404b960 con 0x7f03d01979a0 2024-03-08T22:57:40.285 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.281+0000 7f03ceffd700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 0x7f03bc03b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:40.286 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.282+0000 7f03ceffd700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 0x7f03bc03b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f03c000fa60 tx=0x7f03c0005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:40.300 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.295+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f03c401f020 con 0x7f03d01979a0 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: mon.a calling monitor election 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: mon.c calling monitor election 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: monmap e2: 2 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: fsmap 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: mgrmap e13: x(active, since 61s) 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: overall HEALTH_OK 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:40.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:40.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: mon.a calling monitor election 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: mon.c calling monitor election 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T22:57:40.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: monmap e2: 2 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: fsmap 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: mgrmap e13: x(active, since 61s) 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: overall HEALTH_OK 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:40.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:40.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.576+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f03d019cae0 con 0x7f03d01979a0 2024-03-08T22:57:40.581 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.577+0000 7f03d4821700 1 -- 172.21.15.192:0/380286569 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f03c400fe40 con 0x7f03d01979a0 2024-03-08T22:57:40.582 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:57:40.582 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","modified":"2024-03-08T22:57:35.014391Z","created":"2024-03-08T22:55:28.027578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-08T22:57:40.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 msgr2=0x7f03bc03b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:40.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 0x7f03bc03b1d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f03c000fa60 tx=0x7f03c0005a90 comp rx=0 tx=0).stop 2024-03-08T22:57:40.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 msgr2=0x7f03d019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:40.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 0x7f03d019bd30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f03c400b4d0 tx=0x7f03c400b890 comp rx=0 tx=0).stop 2024-03-08T22:57:40.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 shutdown_connections 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03bc038d10 0x7f03bc03b1d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03b80035c0 0x7f03d0197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 --2- 172.21.15.192:0/380286569 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d01979a0 0x7f03d019bd30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.579+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 >> 172.21.15.192:0/380286569 conn(0x7f03d00fd410 msgr2=0x7f03d00ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.580+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 shutdown_connections 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:40.580+0000 7f03d5823700 1 -- 172.21.15.192:0/380286569 wait complete. 2024-03-08T22:57:40.585 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-03-08T22:57:41.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:41.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:41.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:41.205 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: Deploying daemon mon.b on smithi062 2024-03-08T22:57:41.206 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/380286569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:41.206 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: Deploying daemon mon.b on smithi062 2024-03-08T22:57:41.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/380286569' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:41.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:42.134 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:57:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:57:42.015+0000 7ff071600700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-08T22:57:42.234 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T22:57:42.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T22:57:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:42 smithi046 ceph-mon[31859]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:42.479 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T22:57:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:42 smithi192 ceph-mon[36700]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 -- 172.21.15.192:0/859127561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3381067b0 msgr2=0x7fe338106b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 --2- 172.21.15.192:0/859127561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3381067b0 0x7fe338106b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe328009910 tx=0x7fe328009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 -- 172.21.15.192:0/859127561 shutdown_connections 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 --2- 172.21.15.192:0/859127561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3381067b0 0x7fe338106b90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 -- 172.21.15.192:0/859127561 >> 172.21.15.192:0/859127561 conn(0x7fe3380fc280 msgr2=0x7fe3380fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 -- 172.21.15.192:0/859127561 shutdown_connections 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.060+0000 7fe340089700 1 -- 172.21.15.192:0/859127561 wait complete. 2024-03-08T22:57:44.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 Processor -- start 2024-03-08T22:57:44.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 -- start start 2024-03-08T22:57:44.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:44.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 0x7fe33819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:44.067 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33810adf0 con 0x7fe338196990 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe340089700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe33810ac70 con 0x7fe3381067b0 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:50354/0 (socket says 172.21.15.192:50354) 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 -- 172.21.15.192:0/3143314246 learned_addr learned my addr 172.21.15.192:0/3143314246 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 -- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 msgr2=0x7fe338196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 -- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 msgr2=0x7fe338196450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33de25700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.061+0000 7fe33d624700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 0x7fe33819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.062+0000 7fe33d624700 1 -- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 msgr2=0x7fe338196450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.062+0000 7fe33d624700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.062+0000 7fe33d624700 1 -- 172.21.15.192:0/3143314246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe338196fd0 con 0x7fe338196990 2024-03-08T22:57:44.068 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.062+0000 7fe33d624700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 0x7fe33819ad20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe33400b4d0 tx=0x7fe33400b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:44.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe334011630 con 0x7fe338196990 2024-03-08T22:57:44.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe33819b260 con 0x7fe338196990 2024-03-08T22:57:44.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe33819b7c0 con 0x7fe338196990 2024-03-08T22:57:44.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe334011cb0 con 0x7fe338196990 2024-03-08T22:57:44.069 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fe3340193b0 con 0x7fe338196990 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.063+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe31c005330 con 0x7fe338196990 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.064+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fe334018020 con 0x7fe338196990 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.064+0000 7fe32effd700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 0x7fe32403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.064+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe33404bbc0 con 0x7fe338196990 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.065+0000 7fe33de25700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 0x7fe32403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:44.070 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.066+0000 7fe33de25700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 0x7fe32403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe32800b5c0 tx=0x7fe328009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:44.085 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.080+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe33401f020 con 0x7fe338196990 2024-03-08T22:57:44.344 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.339+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe334022710 con 0x7fe338196990 2024-03-08T22:57:44.359 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:44.355+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe31c0058f0 con 0x7fe338196990 2024-03-08T22:57:44.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:44 smithi046 ceph-mon[31859]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:44.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:44 smithi192 ceph-mon[36700]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:49.362 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.357+0000 7fe32effd700 1 -- 172.21.15.192:0/3143314246 <== mon.0 v2:172.21.15.46:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1279 (secure 0 0 0) 0x7fe334023d00 con 0x7fe338196990 2024-03-08T22:57:49.363 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-03-08T22:57:49.363 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","modified":"2024-03-08T22:57:44.338255Z","created":"2024-03-08T22:55:28.027578Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:3300","nonce":0},{"type":"v1","addr":"172.21.15.46:6789","nonce":0}]},"addr":"172.21.15.46:6789/0","public_addr":"172.21.15.46:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-08T22:57:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.360+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 msgr2=0x7fe32403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.360+0000 7fe340089700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 0x7fe32403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe32800b5c0 tx=0x7fe328009650 comp rx=0 tx=0).stop 2024-03-08T22:57:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.360+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 msgr2=0x7fe33819ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.360+0000 7fe340089700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 0x7fe33819ad20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe33400b4d0 tx=0x7fe33400b890 comp rx=0 tx=0).stop 2024-03-08T22:57:49.365 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 shutdown_connections 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe324038d60 0x7fe32403b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3381067b0 0x7fe338196450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 --2- 172.21.15.192:0/3143314246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe338196990 0x7fe33819ad20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 >> 172.21.15.192:0/3143314246 conn(0x7fe3380fc280 msgr2=0x7fe3381022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 shutdown_connections 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:49.361+0000 7fe340089700 1 -- 172.21.15.192:0/3143314246 wait complete. 2024-03-08T22:57:49.366 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: mon.c calling monitor election 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: mon.a calling monitor election 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/3143314246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: fsmap 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: mgrmap e13: x(active, since 70s) 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: mon.b (rank 2) addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] is down (out of quorum) 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:49.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:49.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: mon.c calling monitor election 2024-03-08T22:57:49.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: mon.a calling monitor election 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/3143314246' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T22:57:49.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: fsmap 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: mgrmap e13: x(active, since 70s) 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: mon.b (rank 2) addr [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] is down (out of quorum) 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:49.786 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:50.063 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-08T22:57:50.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-08T22:57:50.509 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:50.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:50 smithi046 ceph-mon[31859]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:50.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:50.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:50.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:50.702 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:50.710 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:50 smithi192 ceph-mon[36700]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:50.711 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:50.711 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:50.711 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:50.711 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:51.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:51 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:52.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mon.b calling monitor election 2024-03-08T22:57:52.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T22:57:52.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T22:57:52.093 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mon.b calling monitor election 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mon.c calling monitor election 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mon.a calling monitor election 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: fsmap 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: mgrmap e13: x(active, since 72s) 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T22:57:52.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: overall HEALTH_OK 2024-03-08T22:57:52.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.095 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:52 smithi046 ceph-mon[31859]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mon.b calling monitor election 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mon.b calling monitor election 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mon.c calling monitor election 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mon.a calling monitor election 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: fsmap 2024-03-08T22:57:52.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: mgrmap e13: x(active, since 72s) 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: overall HEALTH_OK 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:52 smithi192 ceph-mon[36700]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mon.b calling monitor election 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mon.b calling monitor election 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mon.c calling monitor election 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mon.a calling monitor election 2024-03-08T22:57:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: monmap e3: 3 mons at {a=[v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0],b=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: fsmap 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: mgrmap e13: x(active, since 72s) 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-08T22:57:52.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: overall HEALTH_OK 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:52.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:52 smithi062 ceph-mon[35761]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 -- 172.21.15.46:0/2683641369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 msgr2=0x7faaa80745f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 --2- 172.21.15.46:0/2683641369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa80745f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009910 tx=0x7faa9c009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 -- 172.21.15.46:0/2683641369 shutdown_connections 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 --2- 172.21.15.46:0/2683641369 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa80745f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 -- 172.21.15.46:0/2683641369 >> 172.21.15.46:0/2683641369 conn(0x7faaa80fd2c0 msgr2=0x7faaa80ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:52.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 -- 172.21.15.46:0/2683641369 shutdown_connections 2024-03-08T22:57:52.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.560+0000 7faab0200700 1 -- 172.21.15.46:0/2683641369 wait complete. 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 Processor -- start 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 -- start start 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa819f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 0x7faaa8199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa8199b40 0x7faaa8199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa810e060 con 0x7faaa8074210 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa810dee0 con 0x7faaa8199b40 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faab0200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa810e1e0 con 0x7faaa819f980 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.562+0000 7faaad800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa819f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaad800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa819f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58148/0 (socket says 172.21.15.46:58148) 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaad800700 1 -- 172.21.15.46:0/1552870839 learned_addr learned my addr 172.21.15.46:0/1552870839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 0x7faaa8199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:52.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaae200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa8199b40 0x7faaa8199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:52.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 -- 172.21.15.46:0/1552870839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa8199b40 msgr2=0x7faaa8199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:52.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa8199b40 0x7faaa8199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 -- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 msgr2=0x7faaa819f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa819f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 -- 172.21.15.46:0/1552870839 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa81a6310 con 0x7faaa819f980 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.563+0000 7faaace00700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 0x7faaa8199600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faa9400b550 tx=0x7faa9400b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.564+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94004230 con 0x7faaa819f980 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.564+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa940043d0 con 0x7faaa819f980 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.564+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa94011b20 con 0x7faaa819f980 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.564+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa81a6540 con 0x7faaa819f980 2024-03-08T22:57:52.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.564+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa81a6ad0 con 0x7faaa819f980 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.565+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa8063410 con 0x7faaa819f980 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.565+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7faa94011ce0 con 0x7faaa819f980 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.565+0000 7faaa6000700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 0x7faa9003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.565+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7faa9404bb20 con 0x7faaa819f980 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.566+0000 7faaad800700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 0x7faa9003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:52.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.566+0000 7faaad800700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 0x7faa9003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009750 tx=0x7faa9c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:52.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.571+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7faa940204e0 con 0x7faaa819f980 2024-03-08T22:57:52.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.804+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7faaa81a6df0 con 0x7faaa819f980 2024-03-08T22:57:52.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.805+0000 7faaa6000700 1 -- 172.21.15.46:0/1552870839 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7faa9401a780 con 0x7faaa819f980 2024-03-08T22:57:52.810 INFO:teuthology.orchestra.run.smithi046.stdout:# minimal ceph.conf for ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:57:52.810 INFO:teuthology.orchestra.run.smithi046.stdout:[global] 2024-03-08T22:57:52.811 INFO:teuthology.orchestra.run.smithi046.stdout: fsid = ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T22:57:52.811 INFO:teuthology.orchestra.run.smithi046.stdout: mon_host = [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-03-08T22:57:52.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 msgr2=0x7faa9003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:52.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 0x7faa9003b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa9c009750 tx=0x7faa9c00b540 comp rx=0 tx=0).stop 2024-03-08T22:57:52.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 msgr2=0x7faaa8199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 0x7faaa8199600 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faa9400b550 tx=0x7faa9400b860 comp rx=0 tx=0).stop 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 shutdown_connections 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faa90039450 0x7faa9003b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faaa8199b40 0x7faaa8199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faaa8074210 0x7faaa819f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 --2- 172.21.15.46:0/1552870839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faaa819f980 0x7faaa8199600 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 >> 172.21.15.46:0/1552870839 conn(0x7faaa80fd2c0 msgr2=0x7faaa8105b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 shutdown_connections 2024-03-08T22:57:52.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:57:52.808+0000 7faab0200700 1 -- 172.21.15.46:0/1552870839 wait complete. 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.181 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T22:57:53.182 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:53.182 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:53.182 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.182 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: Reconfiguring daemon mon.a on smithi046 2024-03-08T22:57:53.182 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1552870839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: Reconfiguring daemon mon.a on smithi046 2024-03-08T22:57:53.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:53 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1552870839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:57:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: Reconfiguring daemon mon.a on smithi046 2024-03-08T22:57:53.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:53 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1552870839' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:53.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 22:57:53 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T22:57:53.338+0000 7ff071600700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-08T22:57:53.548 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-08T22:57:53.548 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:57:53.548 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T22:57:53.620 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:57:53.621 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:53.761 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:57:53.761 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T22:57:53.801 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:57:53.801 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:53.877 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:57:53.877 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T22:57:53.912 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:57:53.912 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T22:57:53.983 INFO:tasks.cephadm:Adding mgr.x on smithi046 2024-03-08T22:57:53.983 INFO:tasks.cephadm:Adding mgr.y on smithi062 2024-03-08T22:57:53.983 INFO:tasks.cephadm:Adding mgr.z on smithi192 2024-03-08T22:57:53.983 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi046=x;smithi062=y;smithi192=z' 2024-03-08T22:57:54.261 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T22:57:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:54 smithi192 ceph-mon[36700]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:54 smithi062 ceph-mon[35761]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:54.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:54 smithi046 ceph-mon[31859]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:55.421 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.421 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.421 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.422 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:55.422 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:55.422 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:55.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:55.754 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 -- 172.21.15.192:0/607209655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600ffbd0 msgr2=0x7f32600ffff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:55.829 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 --2- 172.21.15.192:0/607209655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600ffbd0 0x7f32600ffff0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3250009910 tx=0x7f3250009c20 comp rx=0 tx=0).stop 2024-03-08T22:57:55.829 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 -- 172.21.15.192:0/607209655 shutdown_connections 2024-03-08T22:57:55.829 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 --2- 172.21.15.192:0/607209655 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32600ffbd0 0x7f32600ffff0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.829 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 -- 172.21.15.192:0/607209655 >> 172.21.15.192:0/607209655 conn(0x7f32600fb530 msgr2=0x7f32600fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 -- 172.21.15.192:0/607209655 shutdown_connections 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.748+0000 7f32661ca700 1 -- 172.21.15.192:0/607209655 wait complete. 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 Processor -- start 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 -- start start 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3260197750 0x7f326019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3260197bd0 0x7f326019ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32601079a0 con 0x7f3260197750 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260107820 con 0x7f3260197bd0 2024-03-08T22:57:55.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f32661ca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3260107b20 con 0x7f32600ffbd0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f325f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f325f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:36162/0 (socket says 172.21.15.192:36162) 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f325f7fe700 1 -- 172.21.15.192:0/3114187903 learned_addr learned my addr 172.21.15.192:0/3114187903 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.749+0000 7f325ffff700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3260197bd0 0x7f326019ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 -- 172.21.15.192:0/3114187903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3260197bd0 msgr2=0x7f326019ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325effd700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3260197750 0x7f326019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3260197bd0 0x7f326019ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 -- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3260197750 msgr2=0x7f326019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3260197750 0x7f326019c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 -- 172.21.15.192:0/3114187903 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326019f620 con 0x7f32600ffbd0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325f7fe700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3250005e00 tx=0x7f3250004f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:55.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3250027070 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326019f8b0 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f325000be40 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000f460 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.750+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326019fde0 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.751+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f325002a070 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.751+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3260063410 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.751+0000 7f325cff9700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 0x7f324803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.751+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f325004d1c0 con 0x7f32600ffbd0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.752+0000 7f325effd700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 0x7f324803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.753+0000 7f325effd700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 0x7f324803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f32540078e0 tx=0x7f3254008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:57:55.834 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.756+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3250013070 con 0x7f32600ffbd0 2024-03-08T22:57:55.965 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.960+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi062=y;smithi192=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f32600622b0 con 0x7f3248039450 2024-03-08T22:57:55.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.974+0000 7f325cff9700 1 -- 172.21.15.192:0/3114187903 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f32600622b0 con 0x7f3248039450 2024-03-08T22:57:55.979 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 msgr2=0x7f324803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 0x7f324803b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f32540078e0 tx=0x7f3254008040 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 msgr2=0x7f3260197210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3250005e00 tx=0x7f3250004f80 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 shutdown_connections 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f3248039450 0x7f324803b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3260197bd0 0x7f326019ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32600ffbd0 0x7f3260197210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 --2- 172.21.15.192:0/3114187903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3260197750 0x7f326019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 >> 172.21.15.192:0/3114187903 conn(0x7f32600fb530 msgr2=0x7f32600fc210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:57:55.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 shutdown_connections 2024-03-08T22:57:55.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:57:55.976+0000 7f32661ca700 1 -- 172.21.15.192:0/3114187903 wait complete. 2024-03-08T22:57:56.572 DEBUG:teuthology.orchestra.run.smithi062:mgr.y> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.y.service 2024-03-08T22:57:56.576 DEBUG:teuthology.orchestra.run.smithi192:mgr.z> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.z.service 2024-03-08T22:57:56.578 INFO:tasks.cephadm:Deploying OSDs... 2024-03-08T22:57:56.578 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T22:57:56.579 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T22:57:56.640 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T22:57:56.640 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_1 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 765 Links: 1 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-08 22:56:52.439539450 +0000 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-08 22:50:27.108203589 +0000 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-08 22:50:27.108203589 +0000 2024-03-08T22:57:56.694 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-08 22:50:27.108203589 +0000 2024-03-08T22:57:56.695 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T22:57:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-08T22:57:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-08T22:57:56.792 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000280881 s, 1.8 MB/s 2024-03-08T22:57:56.797 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T22:57:56.855 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_2 2024-03-08T22:57:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T22:57:56.908 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 784 Links: 1 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-08 22:56:52.507538747 +0000 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-08 22:50:27.665197647 +0000 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-08 22:50:27.665197647 +0000 2024-03-08T22:57:56.909 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-08 22:50:27.665197647 +0000 2024-03-08T22:57:56.909 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T22:57:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-08T22:57:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-08T22:57:57.006 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000309457 s, 1.7 MB/s 2024-03-08T22:57:57.010 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T22:57:57.069 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_3 2024-03-08T22:57:57.118 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 803 Links: 1 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-08 22:56:52.570538096 +0000 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-08 22:50:28.243191479 +0000 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-08 22:50:28.243191479 +0000 2024-03-08T22:57:57.119 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-08 22:50:28.243191479 +0000 2024-03-08T22:57:57.120 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T22:57:57.206 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: Reconfiguring mon.b (monmap changed)... 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: Reconfiguring daemon mon.b on smithi062 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi062=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: Saving service mgr spec with placement smithi046=x;smithi062=y;smithi192=z;count:3 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:57.207 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:57.218 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-08T22:57:57.218 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-08T22:57:57.218 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000245843 s, 2.1 MB/s 2024-03-08T22:57:57.223 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T22:57:57.280 DEBUG:teuthology.orchestra.run.smithi046:> stat /dev/vg_nvme/lv_4 2024-03-08T22:57:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: Reconfiguring mon.b (monmap changed)... 2024-03-08T22:57:57.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: Reconfiguring daemon mon.b on smithi062 2024-03-08T22:57:57.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:57.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi062=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:57.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: Saving service mgr spec with placement smithi046=x;smithi062=y;smithi192=z;count:3 2024-03-08T22:57:57.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.298 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:57.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:57.299 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 5h/5d Inode: 822 Links: 1 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-08 22:56:52.615537631 +0000 2024-03-08T22:57:57.333 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-08 22:50:28.813185398 +0000 2024-03-08T22:57:57.334 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-08 22:50:28.813185398 +0000 2024-03-08T22:57:57.334 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-08 22:50:28.813185398 +0000 2024-03-08T22:57:57.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T22:57:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: Reconfiguring mon.b (monmap changed)... 2024-03-08T22:57:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: Reconfiguring daemon mon.b on smithi062 2024-03-08T22:57:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi046=x;smithi062=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:57:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: Saving service mgr spec with placement smithi046=x;smithi062=y;smithi192=z;count:3 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T22:57:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records in 2024-03-08T22:57:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:1+0 records out 2024-03-08T22:57:57.432 INFO:teuthology.orchestra.run.smithi046.stderr:512 bytes copied, 0.000256566 s, 2.0 MB/s 2024-03-08T22:57:57.436 DEBUG:teuthology.orchestra.run.smithi046:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T22:57:57.492 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-03-08T22:57:57.492 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T22:57:57.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T22:57:57.512 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-03-08T22:57:57.570 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-08 22:57:43.768977469 +0000 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-08 22:49:35.124341696 +0000 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-08 22:49:35.124341696 +0000 2024-03-08T22:57:57.571 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-08 22:49:35.124341696 +0000 2024-03-08T22:57:57.571 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T22:57:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-08T22:57:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-08T22:57:57.645 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000275895 s, 1.9 MB/s 2024-03-08T22:57:57.646 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T22:57:57.705 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-08 22:57:43.769977454 +0000 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-08 22:49:35.490335670 +0000 2024-03-08T22:57:57.763 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-08 22:49:35.490335670 +0000 2024-03-08T22:57:57.764 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-08 22:49:35.490335670 +0000 2024-03-08T22:57:57.764 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T22:57:57.837 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-08T22:57:57.837 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-08T22:57:57.837 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000217304 s, 2.4 MB/s 2024-03-08T22:57:57.839 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T22:57:57.897 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-08 22:57:43.769977454 +0000 2024-03-08T22:57:57.955 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-08 22:49:35.852329711 +0000 2024-03-08T22:57:57.956 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-08 22:49:35.852329711 +0000 2024-03-08T22:57:57.956 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-08 22:49:35.852329711 +0000 2024-03-08T22:57:57.956 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T22:57:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-08T22:57:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-08T22:57:58.027 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000236796 s, 2.2 MB/s 2024-03-08T22:57:58.028 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T22:57:58.086 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-03-08T22:57:58.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:57 smithi192 ceph-mon[36700]: Reconfiguring mon.c (monmap changed)... 2024-03-08T22:57:58.134 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:57 smithi192 ceph-mon[36700]: Reconfiguring daemon mon.c on smithi192 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-03-08 22:57:43.770977438 +0000 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-03-08 22:49:36.215323734 +0000 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-03-08 22:49:36.215323734 +0000 2024-03-08T22:57:58.142 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-03-08 22:49:36.215323734 +0000 2024-03-08T22:57:58.143 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T22:57:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-03-08T22:57:58.212 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-03-08T22:57:58.213 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000215648 s, 2.4 MB/s 2024-03-08T22:57:58.214 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T22:57:58.271 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-03-08T22:57:58.271 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T22:57:58.285 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T22:57:58.286 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-08 22:57:34.288629411 +0000 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-08 22:48:51.926839228 +0000 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-08 22:48:51.926839228 +0000 2024-03-08T22:57:58.343 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-08 22:48:51.926839228 +0000 2024-03-08T22:57:58.344 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T22:57:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:57 smithi062 ceph-mon[35761]: Reconfiguring mon.c (monmap changed)... 2024-03-08T22:57:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:57 smithi062 ceph-mon[35761]: Reconfiguring daemon mon.c on smithi192 2024-03-08T22:57:58.410 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-08T22:57:58.410 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-08T22:57:58.410 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000198693 s, 2.6 MB/s 2024-03-08T22:57:58.411 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T22:57:58.467 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-03-08T22:57:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:57 smithi046 ceph-mon[31859]: Reconfiguring mon.c (monmap changed)... 2024-03-08T22:57:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:57 smithi046 ceph-mon[31859]: Reconfiguring daemon mon.c on smithi192 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-08 22:57:34.288629411 +0000 2024-03-08T22:57:58.522 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-08 22:48:52.277832093 +0000 2024-03-08T22:57:58.523 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-08 22:48:52.277832093 +0000 2024-03-08T22:57:58.523 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-08 22:48:52.277832093 +0000 2024-03-08T22:57:58.523 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T22:57:58.588 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-08T22:57:58.588 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-08T22:57:58.588 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000224705 s, 2.3 MB/s 2024-03-08T22:57:58.589 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T22:57:58.644 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-08 22:57:34.288629411 +0000 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-08 22:48:52.607825384 +0000 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-08 22:48:52.607825384 +0000 2024-03-08T22:57:58.699 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-08 22:48:52.607825384 +0000 2024-03-08T22:57:58.700 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T22:57:58.767 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-08T22:57:58.767 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-08T22:57:58.767 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000201465 s, 2.5 MB/s 2024-03-08T22:57:58.768 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T22:57:58.824 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-03-08 22:57:34.288629411 +0000 2024-03-08T22:57:58.879 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-03-08 22:48:52.949818432 +0000 2024-03-08T22:57:58.880 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-03-08 22:48:52.949818432 +0000 2024-03-08T22:57:58.880 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-03-08 22:48:52.949818432 +0000 2024-03-08T22:57:58.880 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T22:57:58.944 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-03-08T22:57:58.944 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-03-08T22:57:58.944 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000246777 s, 2.1 MB/s 2024-03-08T22:57:58.945 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T22:57:59.001 INFO:tasks.cephadm:Deploying osd.0 on smithi046 with /dev/vg_nvme/lv_4... 2024-03-08T22:57:59.002 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T22:57:59.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:59.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:58 smithi192 ceph-mon[36700]: Deploying daemon mgr.z on smithi192 2024-03-08T22:57:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:58 smithi062 ceph-mon[35761]: Deploying daemon mgr.z on smithi192 2024-03-08T22:57:59.457 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:57:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:57:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:57:59.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:58 smithi046 ceph-mon[31859]: Deploying daemon mgr.z on smithi192 2024-03-08T22:58:00.096 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:57:59 smithi192 ceph-mon[36700]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:00.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:57:59 smithi046 ceph-mon[31859]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:00.293 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:57:59 smithi062 ceph-mon[35761]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:00.478 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 systemd[1]: Starting Ceph mgr.z for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:58:00.732 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 podman[38546]: 2024-03-08 22:58:00.473618937 +0000 UTC m=+0.112467364 container create 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T22:58:00.732 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 podman[38546]: 2024-03-08 22:58:00.378520668 +0000 UTC m=+0.017369104 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T22:58:00.732 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 podman[38546]: 2024-03-08 22:58:00.701084274 +0000 UTC m=+0.339932706 container init 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-08T22:58:00.733 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 podman[38546]: 2024-03-08 22:58:00.703711485 +0000 UTC m=+0.342559913 container start 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-08T22:58:01.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 ceph-mgr[38565]: -- 172.21.15.192:0/1850979282 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55826feca5a0 con 0x55826fe9f000 2024-03-08T22:58:01.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 bash[38546]: 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d 2024-03-08T22:58:01.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:00 smithi192 systemd[1]: Started Ceph mgr.z for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:58:01.564 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:01.241+0000 7fe311fce200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T22:58:01.564 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:01.334+0000 7fe311fce200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T22:58:02.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:01.973+0000 7fe311fce200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T22:58:02.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:02.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:01 smithi192 ceph-mon[36700]: Deploying daemon mgr.y on smithi062 2024-03-08T22:58:02.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:02.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:01 smithi062 ceph-mon[35761]: Deploying daemon mgr.y on smithi062 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:02.158 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:02.159 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:01 smithi046 ceph-mon[31859]: Deploying daemon mgr.y on smithi062 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0380267 s, 276 MB/s 2024-03-08T22:58:02.289 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-08T22:58:02.342 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.064+0000 7fe311fce200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T22:58:02.342 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.205+0000 7fe311fce200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T22:58:02.342 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.298+0000 7fe311fce200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T22:58:02.617 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.386+0000 7fe311fce200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T22:58:02.924 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.654+0000 7fe311fce200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T22:58:02.924 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.768+0000 7fe311fce200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T22:58:02.924 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.857+0000 7fe311fce200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T22:58:02.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:02 smithi046 ceph-mon[31859]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:03.109 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_4 2024-03-08T22:58:03.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:02 smithi062 ceph-mon[35761]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:03.143 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:02 smithi062 systemd[1]: Starting Ceph mgr.y for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T22:58:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-mon[36700]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:03.314 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:02.954+0000 7fe311fce200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T22:58:03.314 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:03 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:03.136+0000 7fe311fce200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T22:58:03.439 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 podman[37140]: 2024-03-08 22:58:03.166181048 +0000 UTC m=+0.113162047 container create 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD) 2024-03-08T22:58:03.439 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 podman[37140]: 2024-03-08 22:58:03.071118101 +0000 UTC m=+0.018099123 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T22:58:03.440 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 podman[37140]: 2024-03-08 22:58:03.376762543 +0000 UTC m=+0.323743550 container init 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T22:58:03.440 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 podman[37140]: 2024-03-08 22:58:03.379548899 +0000 UTC m=+0.326529906 container start 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-08T22:58:03.440 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 ceph-mgr[37159]: -- 172.21.15.62:0/1279093840 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561abd0ce5a0 con 0x561abd0a2c00 2024-03-08T22:58:03.440 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 bash[37140]: 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 2024-03-08T22:58:03.578 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:58:03.892 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 systemd[1]: Started Ceph mgr.y for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T22:58:04.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:03.965+0000 7fbddf812200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T22:58:04.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:04.065+0000 7fbddf812200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T22:58:04.523 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:04.131+0000 7fe311fce200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T22:58:04.523 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:04.266+0000 7fe311fce200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T22:58:04.773 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:04.773 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.773 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.773 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.773 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.774 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:04.774 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.814 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:04.623+0000 7fe311fce200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T22:58:04.814 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:04.713+0000 7fe311fce200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.864 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:04.865 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:05.117 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:04.768+0000 7fbddf812200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T22:58:05.117 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:04.864+0000 7fbddf812200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T22:58:05.117 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.010+0000 7fbddf812200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T22:58:05.314 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:04.915+0000 7fe311fce200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:58:05.314 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:05.096+0000 7fe311fce200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T22:58:05.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.113+0000 7fbddf812200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T22:58:05.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.199+0000 7fbddf812200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T22:58:05.651 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:05.384+0000 7fe311fce200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:58:05.651 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:05.648+0000 7fe311fce200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T22:58:05.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 -- 172.21.15.46:0/777484970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b00100ff0 msgr2=0x7f1b00101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:05.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 --2- 172.21.15.46:0/777484970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b00100ff0 0x7f1b00101470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009990 tx=0x7f1ae8009ca0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 -- 172.21.15.46:0/777484970 shutdown_connections 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 --2- 172.21.15.46:0/777484970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b001019b0 0x7f1b00101e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 --2- 172.21.15.46:0/777484970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 0x7f1b00100ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 --2- 172.21.15.46:0/777484970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b00100ff0 0x7f1b00101470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 -- 172.21.15.46:0/777484970 >> 172.21.15.46:0/777484970 conn(0x7f1b000fc2b0 msgr2=0x7f1b000fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 -- 172.21.15.46:0/777484970 shutdown_connections 2024-03-08T22:58:05.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.692+0000 7f1b05200700 1 -- 172.21.15.46:0/777484970 wait complete. 2024-03-08T22:58:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.693+0000 7f1b05200700 1 Processor -- start 2024-03-08T22:58:05.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.693+0000 7f1b05200700 1 -- start start 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.693+0000 7f1b05200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 0x7f1b0019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1b05200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1b05200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b001019b0 0x7f1b001a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1b05200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0010fc00 con 0x7f1b00100690 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1b05200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0010fa80 con 0x7f1b00100ff0 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1b05200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b0010fd80 con 0x7f1b001019b0 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1afde00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 0x7f1b0019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1afe800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b001019b0 0x7f1b001a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37414/0 (socket says 172.21.15.46:37414) 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 -- 172.21.15.46:0/1080271086 learned_addr learned my addr 172.21.15.46:0/1080271086 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:58:05.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 -- 172.21.15.46:0/1080271086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b001019b0 msgr2=0x7f1b001a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:05.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b001019b0 0x7f1b001a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 -- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 msgr2=0x7f1b0019a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:05.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 0x7f1b0019a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:05.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.694+0000 7f1af7e00700 1 -- 172.21.15.46:0/1080271086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b001a1f70 con 0x7f1b00100ff0 2024-03-08T22:58:05.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1af7e00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005910 tx=0x7f1ae800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8026070 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ae8003d70 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae8021410 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b001a2200 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.695+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b001a2600 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.696+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b00068aa0 con 0x7f1b00100ff0 2024-03-08T22:58:05.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.697+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f1ae8029070 con 0x7f1b00100ff0 2024-03-08T22:58:05.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.697+0000 7f1af6a00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 0x7f1ae403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:05.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.697+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1ae804cf10 con 0x7f1b00100ff0 2024-03-08T22:58:05.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.698+0000 7f1afde00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 0x7f1ae403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:05.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.698+0000 7f1afde00700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 0x7f1ae403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009c60 tx=0x7f1af0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:58:05.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.703+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1ae8013070 con 0x7f1b00100ff0 2024-03-08T22:58:05.774 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.470+0000 7fbddf812200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T22:58:05.774 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.584+0000 7fbddf812200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T22:58:05.774 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.672+0000 7fbddf812200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T22:58:05.774 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.770+0000 7fbddf812200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T22:58:05.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:05.948+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b00062220 con 0x7f1ae4039200 2024-03-08T22:58:06.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:05.881+0000 7fe311fce200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T22:58:06.143 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:05.950+0000 7fbddf812200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T22:58:06.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:06.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: Reconfiguring daemon mgr.x on smithi046 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:06.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:06 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: Reconfiguring daemon mgr.x on smithi046 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:06.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:06.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:06.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: Reconfiguring daemon mgr.x on smithi046 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:06.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:06.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:07.064 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:06.671+0000 7fe311fce200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T22:58:07.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:06.970+0000 7fbddf812200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T22:58:07.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:07.104+0000 7fbddf812200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T22:58:07.543 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:07.236+0000 7fe311fce200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T22:58:07.544 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:07.330+0000 7fe311fce200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T22:58:07.544 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 22:58:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T22:58:07.539+0000 7fe311fce200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T22:58:07.742 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:07.446+0000 7fbddf812200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T22:58:07.742 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:07.538+0000 7fbddf812200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T22:58:07.742 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:07.738+0000 7fbddf812200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:58:08.142 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:07.927+0000 7fbddf812200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T22:58:08.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:08.192+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f1ae8009d10 con 0x7f1b00100ff0 2024-03-08T22:58:08.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: Standby manager daemon z started 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:08.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.491 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:08.220+0000 7fbddf812200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T22:58:08.491 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:08.487+0000 7fbddf812200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T22:58:08.491 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: Standby manager daemon z started 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:08.492 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: Standby manager daemon z started 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/945499956' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:08.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:08.892 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:08.728+0000 7fbddf812200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T22:58:09.196 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:09 smithi046 ceph-mon[31859]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-08T22:58:09.197 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T22:58:09.560 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:09 smithi062 ceph-mon[35761]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-08T22:58:09.560 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T22:58:09.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:09 smithi192 ceph-mon[36700]: mgrmap e14: x(active, since 89s), standbys: z 2024-03-08T22:58:09.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T22:58:09.892 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:09.556+0000 7fbddf812200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T22:58:10.470 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:10 smithi062 ceph-mon[35761]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:10.470 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:10.146+0000 7fbddf812200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T22:58:10.470 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:10.246+0000 7fbddf812200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T22:58:10.470 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 22:58:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T22:58:10.465+0000 7fbddf812200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T22:58:10.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:10 smithi046 ceph-mon[31859]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:10 smithi192 ceph-mon[36700]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:11.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:11.227+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mon.2 v2:172.21.15.62:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f1ae800e3e0 con 0x7f1b00100ff0 2024-03-08T22:58:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T22:58:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: Standby manager daemon y started 2024-03-08T22:58:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T22:58:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:11.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]: dispatch 2024-03-08T22:58:11.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]': finished 2024-03-08T22:58:11.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T22:58:11.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:11 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:11.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T22:58:11.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: Standby manager daemon y started 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]: dispatch 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]': finished 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T22:58:11.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:11 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:11.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T22:58:11.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T22:58:11.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: Standby manager daemon y started 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1244835372' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]: dispatch 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3923449213' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6108879-28e8-496c-9422-46c94c3d7ca1"}]': finished 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T22:58:11.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:11 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:12 smithi046 ceph-mon[31859]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:12 smithi046 ceph-mon[31859]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-08T22:58:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T22:58:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:12 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3887299350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:12.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:12 smithi192 ceph-mon[36700]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:12.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:12 smithi192 ceph-mon[36700]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-08T22:58:12.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T22:58:12.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:12 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3887299350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:12 smithi062 ceph-mon[35761]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:12 smithi062 ceph-mon[35761]: mgrmap e15: x(active, since 92s), standbys: z, y 2024-03-08T22:58:12.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T22:58:12.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:12 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3887299350' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:14 smithi192 ceph-mon[36700]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:14.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:14 smithi062 ceph-mon[35761]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:14.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:14 smithi046 ceph-mon[31859]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:16 smithi192 ceph-mon[36700]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:16 smithi062 ceph-mon[35761]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:16.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:16 smithi046 ceph-mon[31859]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:18.441 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:18 smithi046 ceph-mon[31859]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:18.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:18 smithi192 ceph-mon[36700]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:18.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:18 smithi062 ceph-mon[35761]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:20 smithi192 ceph-mon[36700]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T22:58:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:20 smithi192 ceph-mon[36700]: Deploying daemon osd.0 on smithi046 2024-03-08T22:58:20.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:20 smithi062 ceph-mon[35761]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:20.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T22:58:20.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:20 smithi062 ceph-mon[35761]: Deploying daemon osd.0 on smithi046 2024-03-08T22:58:20.708 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:20 smithi046 ceph-mon[31859]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:20.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T22:58:20.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:20.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:20 smithi046 ceph-mon[31859]: Deploying daemon osd.0 on smithi046 2024-03-08T22:58:22.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:22 smithi046 ceph-mon[31859]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:22.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:22 smithi192 ceph-mon[36700]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:22.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:22 smithi062 ceph-mon[35761]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:24 smithi192 ceph-mon[36700]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:24.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:24 smithi062 ceph-mon[35761]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:24.653 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:24 smithi046 ceph-mon[31859]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:26.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:26 smithi046 ceph-mon[31859]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:26.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:26 smithi192 ceph-mon[36700]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:26.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:26 smithi062 ceph-mon[35761]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:28 smithi046 ceph-mon[31859]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:28 smithi192 ceph-mon[36700]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:28 smithi062 ceph-mon[35761]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:30 smithi192 ceph-mon[36700]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:30.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:30 smithi062 ceph-mon[35761]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:30.688 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:30 smithi046 ceph-mon[31859]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:31.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:31.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:31.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:31.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:31.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:31.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:32.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:32.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.647+0000 7f1af6a00700 1 -- 172.21.15.46:0/1080271086 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1b00062220 con 0x7f1ae4039200 2024-03-08T22:58:32.652 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 0 on host 'smithi046' 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.650+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 msgr2=0x7f1ae403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.650+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 0x7f1ae403b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1af0009c60 tx=0x7f1af0009260 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.650+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 msgr2=0x7f1b0019a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.650+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8005910 tx=0x7f1ae800bbb0 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 shutdown_connections 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ae4039200 0x7f1ae403b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1b001019b0 0x7f1b001a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1b00100690 0x7f1b0019a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 --2- 172.21.15.46:0/1080271086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1b00100ff0 0x7f1b0019a850 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:32.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 >> 172.21.15.46:0/1080271086 conn(0x7f1b000fc2b0 msgr2=0x7f1b000fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:58:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 shutdown_connections 2024-03-08T22:58:32.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:58:32.651+0000 7f1b05200700 1 -- 172.21.15.46:0/1080271086 wait complete. 2024-03-08T22:58:32.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:32.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:32.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:32.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:32.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:33.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:33.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:33.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:33.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:33.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:33.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:33.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:33.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:33.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T22:58:33.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:33.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:33.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:33.434 DEBUG:teuthology.orchestra.run.smithi046:osd.0> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.0.service 2024-03-08T22:58:33.438 INFO:tasks.cephadm:Deploying osd.1 on smithi062 with /dev/vg_nvme/lv_4... 2024-03-08T22:58:33.438 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T22:58:33.696 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T22:58:33.808 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:33 smithi062 ceph-mon[35761]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T22:58:33.808 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:33 smithi062 ceph-mon[35761]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:33.808 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:33 smithi062 ceph-mon[35761]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T22:58:33.808 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:33.808 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:33 smithi062 ceph-mon[35761]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:33.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:33 smithi046 ceph-mon[31859]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T22:58:33.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:33 smithi046 ceph-mon[31859]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:33.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:33 smithi046 ceph-mon[31859]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T22:58:33.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:33.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:33 smithi046 ceph-mon[31859]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:34.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:33 smithi192 ceph-mon[36700]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T22:58:34.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:33 smithi192 ceph-mon[36700]: from='osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:34.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:33 smithi192 ceph-mon[36700]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T22:58:34.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:34.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:33 smithi192 ceph-mon[36700]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:58:34.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0[43885]: 2024-03-08T22:58:34.320+0000 7febe9e00700 -1 osd.0 0 waiting for initial osdmap 2024-03-08T22:58:34.676 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0[43885]: 2024-03-08T22:58:34.337+0000 7febdf200700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T22:58:34.972 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T22:58:34.972 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T22:58:34.972 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:34.972 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:58:34.972 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T22:58:34.973 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:34.973 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:34.973 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:34 smithi062 ceph-mon[35761]: from='osd.0 ' entity='osd.0' 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:34.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:34 smithi046 ceph-mon[31859]: from='osd.0 ' entity='osd.0' 2024-03-08T22:58:35.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T22:58:35.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T22:58:35.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:35.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:58:35.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T22:58:35.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:35.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:35.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:34 smithi192 ceph-mon[36700]: from='osd.0 ' entity='osd.0' 2024-03-08T22:58:35.769 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T22:58:35.769 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T22:58:35.770 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-03-08T22:58:35.770 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-03-08T22:58:35.770 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112426 s, 933 MB/s 2024-03-08T22:58:35.770 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-03-08T22:58:35.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:35.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:35 smithi046 ceph-mon[31859]: osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] boot 2024-03-08T22:58:35.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:35 smithi046 ceph-mon[31859]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T22:58:35.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:36.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:36.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:35 smithi192 ceph-mon[36700]: osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] boot 2024-03-08T22:58:36.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:35 smithi192 ceph-mon[36700]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T22:58:36.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:36.077 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:36.078 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:35 smithi062 ceph-mon[35761]: osd.0 [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] boot 2024-03-08T22:58:36.078 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:35 smithi062 ceph-mon[35761]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T22:58:36.078 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T22:58:36.323 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi062:vg_nvme/lv_4 2024-03-08T22:58:36.570 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T22:58:36.935 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:36 smithi062 ceph-mon[35761]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:36.935 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:36 smithi062 ceph-mon[35761]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T22:58:36.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:36 smithi046 ceph-mon[31859]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:36.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:36 smithi046 ceph-mon[31859]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T22:58:37.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:36 smithi192 ceph-mon[36700]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T22:58:37.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:36 smithi192 ceph-mon[36700]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T22:58:38.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2557950462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db41028b0 msgr2=0x7f1db4109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:38.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2557950462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db41028b0 0x7f1db4109160 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1dac00b0d0 tx=0x7f1dac00b3e0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2557950462 shutdown_connections 2024-03-08T22:58:38.003 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2557950462 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db41028b0 0x7f1db4109160 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2557950462 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db4100d10 0x7f1db4101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2557950462 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4101f10 0x7f1db4102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2557950462 >> 172.21.15.62:0/2557950462 conn(0x7f1db40fc2b0 msgr2=0x7f1db40fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:58:38.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2557950462 shutdown_connections 2024-03-08T22:58:38.004 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:37.999+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2557950462 wait complete. 2024-03-08T22:58:38.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 Processor -- start 2024-03-08T22:58:38.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 -- start start 2024-03-08T22:58:38.006 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4100d10 0x7f1db4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db41028b0 0x7f1db419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db410c950 con 0x7f1db4100d10 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db410c7d0 con 0x7f1db4101f10 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1dba9b2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db410cad0 con 0x7f1db41028b0 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:57188/0 (socket says 172.21.15.62:57188) 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 -- 172.21.15.62:0/2330907488 learned_addr learned my addr 172.21.15.62:0/2330907488 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db8f4f700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db41028b0 0x7f1db419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 -- 172.21.15.62:0/2330907488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db41028b0 msgr2=0x7f1db419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db41028b0 0x7f1db419f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.007 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db3fff700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4100d10 0x7f1db4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 -- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4100d10 msgr2=0x7f1db4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4100d10 0x7f1db4198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.000+0000 7f1db37fe700 1 -- 172.21.15.62:0/2330907488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db419fcc0 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1db37fe700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1da800b530 tx=0x7f1da800b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da8015020 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1da8011d10 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db419ffb0 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da800f2e0 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.001+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db41a04e0 con 0x7f1db4101f10 2024-03-08T22:58:38.008 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.002+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db4068aa0 con 0x7f1db4101f10 2024-03-08T22:58:38.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.003+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f1da8018020 con 0x7f1db4101f10 2024-03-08T22:58:38.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.003+0000 7f1db17fa700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 0x7f1d9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:58:38.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.004+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f1da80cc7a0 con 0x7f1db4101f10 2024-03-08T22:58:38.009 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.004+0000 7f1db3fff700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 0x7f1d9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:58:38.011 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.007+0000 7f1db3fff700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 0x7f1d9c0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005950 tx=0x7f1da40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:58:38.012 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.008+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1da801f020 con 0x7f1db4101f10 2024-03-08T22:58:38.226 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:38.222+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1db40fe440 con 0x7f1d9c0a2800 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:38.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:38.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:38.980 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:39.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:39.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:58:39.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:58:39.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:39.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:39.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:39.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:58:39.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:39.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:39.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:58:39.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:39.944 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:39 smithi062 ceph-mon[35761]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:39.944 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:39 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi046 2024-03-08T22:58:39.944 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:39 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 17095M 2024-03-08T22:58:39.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:39 smithi046 ceph-mon[31859]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:39.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:39 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi046 2024-03-08T22:58:39.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:39 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 17095M 2024-03-08T22:58:40.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:39 smithi192 ceph-mon[36700]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:58:40.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:39 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi046 2024-03-08T22:58:40.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:39 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 17095M 2024-03-08T22:58:40.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:40 smithi046 ceph-mon[31859]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:40 smithi192 ceph-mon[36700]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:41.107 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:40 smithi062 ceph-mon[35761]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:41.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/642131358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:41.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:41.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]': finished 2024-03-08T22:58:41.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T22:58:41.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:41.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/2427168806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:42.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/642131358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:42.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:42.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]': finished 2024-03-08T22:58:42.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T22:58:42.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:42.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/2427168806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:42.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/642131358' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:42.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]: dispatch 2024-03-08T22:58:42.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b19e310-9857-4d83-97f8-eab41f626164"}]': finished 2024-03-08T22:58:42.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T22:58:42.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:42.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/2427168806' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:58:42.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:42 smithi046 ceph-mon[31859]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:43.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:42 smithi192 ceph-mon[36700]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:43.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:42 smithi062 ceph-mon[35761]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:45.002 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:44 smithi062 ceph-mon[35761]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:45.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:44 smithi192 ceph-mon[36700]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:45.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:44 smithi046 ceph-mon[31859]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:46.970 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:46 smithi062 ceph-mon[35761]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:47.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:46 smithi192 ceph-mon[36700]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:47.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:46 smithi046 ceph-mon[31859]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:49.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:48 smithi192 ceph-mon[36700]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:49.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T22:58:49.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:49.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:48 smithi192 ceph-mon[36700]: Deploying daemon osd.1 on smithi062 2024-03-08T22:58:49.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:48 smithi062 ceph-mon[35761]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T22:58:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:48 smithi062 ceph-mon[35761]: Deploying daemon osd.1 on smithi062 2024-03-08T22:58:49.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:48 smithi046 ceph-mon[31859]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:49.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T22:58:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:58:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:48 smithi046 ceph-mon[31859]: Deploying daemon osd.1 on smithi062 2024-03-08T22:58:50.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:50 smithi062 ceph-mon[35761]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:50 smithi192 ceph-mon[36700]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:50 smithi046 ceph-mon[31859]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:52.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:52 smithi062 ceph-mon[35761]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:52.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:52 smithi046 ceph-mon[31859]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:52 smithi192 ceph-mon[36700]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:54.136 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.136 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.136 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:54.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:54.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:54.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:58:55.119 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:54 smithi062 ceph-mon[35761]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:54 smithi046 ceph-mon[31859]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:54 smithi192 ceph-mon[36700]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:56.013 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.007+0000 7f1db17fa700 1 -- 172.21.15.62:0/2330907488 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1db40fe440 con 0x7f1d9c0a2800 2024-03-08T22:58:56.013 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 1 on host 'smithi062' 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 msgr2=0x7f1d9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 0x7f1d9c0a4cc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1da4005950 tx=0x7f1da40058e0 comp rx=0 tx=0).stop 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 msgr2=0x7f1db41985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1da800b530 tx=0x7f1da800b840 comp rx=0 tx=0).stop 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 shutdown_connections 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d9c0a2800 0x7f1d9c0a4cc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1db41028b0 0x7f1db419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:56.015 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1db4100d10 0x7f1db4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:56.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 --2- 172.21.15.62:0/2330907488 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1db4101f10 0x7f1db41985a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:58:56.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 >> 172.21.15.62:0/2330907488 conn(0x7f1db40fc2b0 msgr2=0x7f1db40fdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:58:56.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 shutdown_connections 2024-03-08T22:58:56.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T22:58:56.010+0000 7f1dba9b2700 1 -- 172.21.15.62:0/2330907488 wait complete. 2024-03-08T22:58:56.687 DEBUG:teuthology.orchestra.run.smithi062:osd.1> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.1.service 2024-03-08T22:58:56.689 INFO:tasks.cephadm:Deploying osd.2 on smithi192 with /dev/vg_nvme/lv_4... 2024-03-08T22:58:56.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T22:58:56.933 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T22:58:56.975 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T22:58:56.732+0000 7f3b4950a7c0 -1 osd.1 0 log_to_monitors true 2024-03-08T22:58:57.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:57.268 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:56 smithi192 ceph-mon[36700]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:57.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:56 smithi062 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:58:57.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:57.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:56 smithi046 ceph-mon[31859]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T22:58:58.257 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:58.257 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T22:58:58.257 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.258 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T22:58:58.258 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:58.258 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:58 smithi192 ceph-mon[36700]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.392 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T22:58:58.130+0000 7f3b3eb6f700 -1 osd.1 0 waiting for initial osdmap 2024-03-08T22:58:58.393 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T22:58:58.133+0000 7f3b3a962700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:58 smithi062 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:58:58.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T22:58:58.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: from='osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T22:58:58.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:58.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:58 smithi046 ceph-mon[31859]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112437 s, 933 MB/s 2024-03-08T22:58:58.824 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-08T22:58:59.309 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:59 smithi062 ceph-mon[35761]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T22:58:59.309 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:59 smithi062 ceph-mon[35761]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T22:58:59.309 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:58:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:59.452 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:59 smithi192 ceph-mon[36700]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T22:58:59.453 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:59 smithi192 ceph-mon[36700]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T22:58:59.453 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:58:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:59 smithi046 ceph-mon[31859]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T22:58:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:59 smithi046 ceph-mon[31859]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T22:58:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:58:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:58:59.479 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-03-08T22:58:59.715 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] boot 2024-03-08T22:59:00.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi062 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 17080M 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:00.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] boot 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T22:59:00.534 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi062 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 17080M 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: pgmap v72: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: osd.1 [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] boot 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi062 2024-03-08T22:59:00.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 17080M 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:00.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:01.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 -- 172.21.15.192:0/2906650946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 msgr2=0x7f87a410a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:01.348 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 --2- 172.21.15.192:0/2906650946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a410a420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f879c00b0a0 tx=0x7f879c00b3b0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.349 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 -- 172.21.15.192:0/2906650946 shutdown_connections 2024-03-08T22:59:01.349 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 --2- 172.21.15.192:0/2906650946 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a410a420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.349 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 --2- 172.21.15.192:0/2906650946 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a4103430 0x7f87a4103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.349 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 --2- 172.21.15.192:0/2906650946 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 0x7f87a4100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.349 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 -- 172.21.15.192:0/2906650946 >> 172.21.15.192:0/2906650946 conn(0x7f87a40fba70 msgr2=0x7f87a40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:59:01.350 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 -- 172.21.15.192:0/2906650946 shutdown_connections 2024-03-08T22:59:01.350 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.343+0000 7f87a8955700 1 -- 172.21.15.192:0/2906650946 wait complete. 2024-03-08T22:59:01.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 Processor -- start 2024-03-08T22:59:01.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 -- start start 2024-03-08T22:59:01.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 0x7f87a4197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:01.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:01.353 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a4103430 0x7f87a419f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a410da10 con 0x7f87a4103430 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a410d890 con 0x7f87a40ffeb0 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a8955700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a410db90 con 0x7f87a4100850 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a1d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a1d9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:40528/0 (socket says 172.21.15.192:40528) 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a1d9b700 1 -- 172.21.15.192:0/2682571265 learned_addr learned my addr 172.21.15.192:0/2682571265 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T22:59:01.354 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.344+0000 7f87a259c700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 0x7f87a4197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 -- 172.21.15.192:0/2682571265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 msgr2=0x7f87a4197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 0x7f87a4197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 -- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a4103430 msgr2=0x7f87a419f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a4103430 0x7f87a419f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 -- 172.21.15.192:0/2682571265 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a419fc20 con 0x7f87a4100850 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a1d9b700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f879800d9b0 tx=0x7f879800dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798017070 con 0x7f87a4100850 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8798005020 con 0x7f87a4100850 2024-03-08T22:59:01.355 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8798009bd0 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87a419ff10 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.345+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87a41a0370 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.346+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87a4068aa0 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.347+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f87980051d0 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.348+0000 7f87937fe700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 0x7f878c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.348+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f87980cd4f0 con 0x7f87a4100850 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.348+0000 7f87a259c700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 0x7f878c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:01.356 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.349+0000 7f87a259c700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 0x7f878c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87a406a240 tx=0x7f8794009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:59:01.357 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.351+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8798020020 con 0x7f87a4100850 2024-03-08T22:59:01.559 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:01.554+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f87a4062220 con 0x7f878c0a2800 2024-03-08T22:59:01.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:01 smithi192 ceph-mon[36700]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T22:59:01.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:01.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:01.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:01.930 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:01 smithi062 ceph-mon[35761]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T22:59:01.931 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:01.931 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:01.931 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:01.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:01 smithi046 ceph-mon[31859]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T22:59:01.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:01.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:01.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:02.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:02 smithi192 ceph-mon[36700]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:02.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:02 smithi192 ceph-mon[36700]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:02.918 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:02 smithi062 ceph-mon[35761]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:02.918 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:02 smithi062 ceph-mon[35761]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:02.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:02 smithi046 ceph-mon[31859]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:02.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:02 smithi046 ceph-mon[31859]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/3716671116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]': finished 2024-03-08T22:59:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T22:59:04.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:04 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/3716671116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]': finished 2024-03-08T22:59:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T22:59:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:04 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:05.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:05.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/3716671116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:05.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]: dispatch 2024-03-08T22:59:05.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df"}]': finished 2024-03-08T22:59:05.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T22:59:05.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:04 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:05 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/1363024139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:06.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:05 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/1363024139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:06.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:05 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/1363024139' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:06 smithi046 ceph-mon[31859]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:07.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:06 smithi192 ceph-mon[36700]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:07.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:06 smithi062 ceph-mon[35761]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:08.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:08 smithi046 ceph-mon[31859]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:09.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:08 smithi192 ceph-mon[36700]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:08 smithi062 ceph-mon[35761]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:10.883 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:10 smithi192 ceph-mon[36700]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:10.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:10 smithi046 ceph-mon[31859]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:11.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:10 smithi062 ceph-mon[35761]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:11.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:11 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T22:59:11.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:11 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:12.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:11 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T22:59:12.004 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:11 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:12.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:11 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T22:59:12.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:11 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:12.717 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:12 smithi192 ceph-mon[36700]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:12.717 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:12 smithi192 ceph-mon[36700]: Deploying daemon osd.2 on smithi192 2024-03-08T22:59:12.964 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:12 smithi062 ceph-mon[35761]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:12.964 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:12 smithi062 ceph-mon[35761]: Deploying daemon osd.2 on smithi192 2024-03-08T22:59:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:12 smithi046 ceph-mon[31859]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:12 smithi046 ceph-mon[31859]: Deploying daemon osd.2 on smithi192 2024-03-08T22:59:14.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:14 smithi046 ceph-mon[31859]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:14.978 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:14 smithi192 ceph-mon[36700]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:15.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:14 smithi062 ceph-mon[35761]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:16.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:16 smithi046 ceph-mon[31859]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:16.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:16.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:16.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:17.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:16 smithi192 ceph-mon[36700]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:16 smithi062 ceph-mon[35761]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:18.963 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:18 smithi192 ceph-mon[36700]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:18.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:18 smithi046 ceph-mon[31859]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:19.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:18 smithi062 ceph-mon[35761]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:20.042 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.037+0000 7f87937fe700 1 -- 172.21.15.192:0/2682571265 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f87a4062220 con 0x7f878c0a2800 2024-03-08T22:59:20.042 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 2 on host 'smithi192' 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 msgr2=0x7f878c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 0x7f878c0a4cc0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f87a406a240 tx=0x7f8794009190 comp rx=0 tx=0).stop 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 msgr2=0x7f87a4198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f879800d9b0 tx=0x7f879800dcc0 comp rx=0 tx=0).stop 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 shutdown_connections 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f878c0a2800 0x7f878c0a4cc0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:20.044 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87a4103430 0x7f87a419f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:20.045 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87a40ffeb0 0x7f87a4197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:20.045 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 --2- 172.21.15.192:0/2682571265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87a4100850 0x7f87a4198500 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:20.045 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 >> 172.21.15.192:0/2682571265 conn(0x7f87a40fba70 msgr2=0x7f87a4106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:59:20.045 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 shutdown_connections 2024-03-08T22:59:20.045 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T22:59:20.039+0000 7f87a8955700 1 -- 172.21.15.192:0/2682571265 wait complete. 2024-03-08T22:59:20.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.593 DEBUG:teuthology.orchestra.run.smithi192:osd.2> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.2.service 2024-03-08T22:59:20.595 INFO:tasks.cephadm:Deploying osd.3 on smithi046 with /dev/vg_nvme/lv_3... 2024-03-08T22:59:20.595 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-08T22:59:20.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:20.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:21.069 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:59:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:21 smithi192 ceph-mon[36700]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:21 smithi192 ceph-mon[36700]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:21 smithi046 ceph-mon[31859]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:21 smithi046 ceph-mon[31859]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:21.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:21 smithi062 ceph-mon[35761]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:21.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:21 smithi062 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T22:59:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T22:59:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.296 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T22:59:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:22.297 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-mon[36700]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:22.470 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:22 smithi046 ceph-mon[31859]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: pgmap v86: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:22.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T22:59:22.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: from='osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T22:59:22.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:22.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:22 smithi062 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T22:59:22.928 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2[41845]: 2024-03-08T22:59:22.662+0000 7f19997a1700 -1 osd.2 0 waiting for initial osdmap 2024-03-08T22:59:22.928 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2[41845]: 2024-03-08T22:59:22.666+0000 7f1990d8b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T22:59:23.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:23 smithi046 ceph-mon[31859]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T22:59:23.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:23 smithi046 ceph-mon[31859]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T22:59:23.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:23 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:23 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:23 smithi046 ceph-mon[31859]: from='osd.2 ' entity='osd.2' 2024-03-08T22:59:23.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:23 smithi192 ceph-mon[36700]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T22:59:23.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:23 smithi192 ceph-mon[36700]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T22:59:23.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:23 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:23 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.535 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:23 smithi192 ceph-mon[36700]: from='osd.2 ' entity='osd.2' 2024-03-08T22:59:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:23 smithi062 ceph-mon[35761]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T22:59:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:23 smithi062 ceph-mon[35761]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T22:59:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:23 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:23 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:23.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:23 smithi062 ceph-mon[35761]: from='osd.2 ' entity='osd.2' 2024-03-08T22:59:23.954 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-08T22:59:23.955 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-08T22:59:23.955 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-08T22:59:23.955 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-08T22:59:23.955 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0370697 s, 283 MB/s 2024-03-08T22:59:23.955 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi192 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:24.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:24.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] boot 2024-03-08T22:59:24.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T22:59:24.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T22:59:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi192 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] boot 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T22:59:24.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: pgmap v89: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi192 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 17080M 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:24.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:24.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:24.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: osd.2 [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] boot 2024-03-08T22:59:24.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T22:59:24.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T22:59:24.749 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_3 2024-03-08T22:59:25.208 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T22:59:25.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:25 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:25.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:25 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:25.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:25 smithi046 ceph-mon[31859]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T22:59:25.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:25 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:26.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:25 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:26.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:25 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:26.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:25 smithi192 ceph-mon[36700]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T22:59:26.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:25 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:26.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:25 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:26.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:25 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:26.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:25 smithi062 ceph-mon[35761]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T22:59:26.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:25 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T22:59:27.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:26 smithi192 ceph-mon[36700]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:27.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:27.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:26 smithi192 ceph-mon[36700]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T22:59:27.129 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:26 smithi046 ceph-mon[31859]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:27.129 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:27.129 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:26 smithi046 ceph-mon[31859]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T22:59:27.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:26 smithi062 ceph-mon[35761]: pgmap v91: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:27.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T22:59:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:26 smithi062 ceph-mon[35761]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T22:59:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.300+0000 7f05bbe00700 1 -- 172.21.15.46:0/3006351394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc103f10 msgr2=0x7f05bc104350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.300+0000 7f05bbe00700 1 --2- 172.21.15.46:0/3006351394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc103f10 0x7f05bc104350 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f05ac009990 tx=0x7f05ac009ca0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 -- 172.21.15.46:0/3006351394 shutdown_connections 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 --2- 172.21.15.46:0/3006351394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc105ab0 0x7f05bc10c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 --2- 172.21.15.46:0/3006351394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc103f10 0x7f05bc104350 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 --2- 172.21.15.46:0/3006351394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 0x7f05bc105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 -- 172.21.15.46:0/3006351394 >> 172.21.15.46:0/3006351394 conn(0x7f05bc0ff500 msgr2=0x7f05bc101940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 -- 172.21.15.46:0/3006351394 shutdown_connections 2024-03-08T22:59:27.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.301+0000 7f05bbe00700 1 -- 172.21.15.46:0/3006351394 wait complete. 2024-03-08T22:59:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.302+0000 7f05bbe00700 1 Processor -- start 2024-03-08T22:59:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.302+0000 7f05bbe00700 1 -- start start 2024-03-08T22:59:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.302+0000 7f05bbe00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.302+0000 7f05bbe00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 0x7f05bc19a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:27.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05bbe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc105ab0 0x7f05bc1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05bbe00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc10fb40 con 0x7f05bc105ab0 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05bbe00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc10f9c0 con 0x7f05bc103f10 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05bbe00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05bc10fcc0 con 0x7f05bc1050f0 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48906/0 (socket says 172.21.15.46:48906) 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 -- 172.21.15.46:0/2445563107 learned_addr learned my addr 172.21.15.46:0/2445563107 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05bb400700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc105ab0 0x7f05bc1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:27.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05ba000700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 0x7f05bc19a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:27.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 -- 172.21.15.46:0/2445563107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 msgr2=0x7f05bc19a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:27.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 0x7f05bc19a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 -- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc105ab0 msgr2=0x7f05bc1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc105ab0 0x7f05bc1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.303+0000 7f05baa00700 1 -- 172.21.15.46:0/2445563107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05bc1a1d80 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05baa00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00b2e0 tx=0x7f05ac00bb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac027070 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05bc1a2010 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05bc1a24c0 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05ac003d70 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.304+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ac00f460 con 0x7f05bc103f10 2024-03-08T22:59:27.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.305+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f059c005360 con 0x7f05bc103f10 2024-03-08T22:59:27.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.307+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f05ac02a070 con 0x7f05bc103f10 2024-03-08T22:59:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.309+0000 7f05b3400700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 0x7f05a00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T22:59:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.309+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f05ac0ce820 con 0x7f05bc103f10 2024-03-08T22:59:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.309+0000 7f05ba000700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 0x7f05a00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T22:59:27.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.310+0000 7f05ba000700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 0x7f05a00a4dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009c60 tx=0x7f05a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T22:59:27.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.312+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f05ac013070 con 0x7f05bc103f10 2024-03-08T22:59:27.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:27.581+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f059c000bf0 con 0x7f05a00a2900 2024-03-08T22:59:27.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:27.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:27.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:28.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:28.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:28.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:28.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T22:59:28.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T22:59:28.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:28.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:28.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:28.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48352]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:28.490 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48352]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:29.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:28 smithi192 ceph-mon[36700]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:29.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:28 smithi192 ceph-mon[36700]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:29.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:28 smithi192 ceph-mon[36700]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T22:59:29.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:28 smithi062 ceph-mon[35761]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:29.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:28 smithi062 ceph-mon[35761]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:29.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:28 smithi062 ceph-mon[35761]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T22:59:29.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48406]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:29.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48406]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:29.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48406]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:29.226 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 22:59:28 smithi046 sudo[48406]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:29.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:28 smithi046 ceph-mon[31859]: pgmap v94: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:29.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:28 smithi046 ceph-mon[31859]: from='client.24188 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T22:59:29.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:28 smithi046 ceph-mon[31859]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T22:59:29.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44384]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:29.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44384]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:29.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44384]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:29.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44384]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:29.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:29.857+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f05ac009d10 con 0x7f05bc103f10 2024-03-08T22:59:29.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:29.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:30.143 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44387]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:30.143 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44387]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:30.143 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44387]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:30.143 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 22:59:29 smithi062 sudo[44387]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:30.281 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:30.281 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:30.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45772]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:30.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45772]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:30.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45772]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:30.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45772]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:30.564 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45775]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:30.564 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45775]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:30.565 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45775]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:30.565 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 22:59:30 smithi192 sudo[45775]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:31.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:30 smithi046 sudo[48465]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:31.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:30 smithi046 sudo[48465]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:31.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:30 smithi046 sudo[48465]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:31.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:30 smithi046 sudo[48465]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:31.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 sudo[48498]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 sudo[48498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 sudo[48498]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 sudo[48498]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 ceph-mon[31859]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 ceph-mon[31859]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:31.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:31 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 ceph-mon[36700]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 ceph-mon[36700]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45778]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45778]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45778]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:31.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45778]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:31 smithi062 ceph-mon[35761]: pgmap v96: 1 pgs: 1 unknown; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:31 smithi062 ceph-mon[35761]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T22:59:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:31 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:31.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:31 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:32.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45781]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:32.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45781]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:32.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45781]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:32.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:31 smithi192 sudo[45781]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:32.874 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44390]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T22:59:32.874 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44390]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:32.874 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44390]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:32.875 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44390]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:33.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:33.208 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4213010814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]': finished 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:32 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44393]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44393]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44393]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 sudo[44393]: pam_unix(sudo:session): session closed for user root 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.210 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4213010814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]': finished 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.211 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:32 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4213010814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.230 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]: dispatch 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f714570-8e8a-416b-a6f3-b0a42f737b81"}]': finished 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T22:59:33.231 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:32 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T22:59:34.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:33 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:34.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:33 smithi046 ceph-mon[31859]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:34.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:33 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4027572176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:34.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:33 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:34.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:33 smithi192 ceph-mon[36700]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:34.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:33 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4027572176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:34.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:33 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T22:59:34.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:33 smithi062 ceph-mon[35761]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T22:59:34.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:33 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4027572176' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T22:59:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:34 smithi046 ceph-mon[31859]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:34 smithi192 ceph-mon[36700]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:35.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:34 smithi062 ceph-mon[35761]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:37.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:36 smithi046 ceph-mon[31859]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:37.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:36 smithi192 ceph-mon[36700]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:37.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:36 smithi062 ceph-mon[35761]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:38.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:38 smithi046 ceph-mon[31859]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-08T22:59:39.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:38 smithi192 ceph-mon[36700]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-08T22:59:39.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:38 smithi062 ceph-mon[35761]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-08T22:59:41.147 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:40 smithi046 ceph-mon[31859]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-03-08T22:59:41.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:40 smithi192 ceph-mon[36700]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-03-08T22:59:41.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:40 smithi062 ceph-mon[35761]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 98 KiB/s wr, 7 op/s 2024-03-08T22:59:42.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T22:59:42.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:42.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T22:59:42.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:42.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T22:59:42.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:43.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:42 smithi046 ceph-mon[31859]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-08T22:59:43.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:42 smithi046 ceph-mon[31859]: Deploying daemon osd.3 on smithi046 2024-03-08T22:59:43.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:42 smithi192 ceph-mon[36700]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-08T22:59:43.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:42 smithi192 ceph-mon[36700]: Deploying daemon osd.3 on smithi046 2024-03-08T22:59:43.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:42 smithi062 ceph-mon[35761]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-08T22:59:43.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:42 smithi062 ceph-mon[35761]: Deploying daemon osd.3 on smithi046 2024-03-08T22:59:44.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:44 smithi046 ceph-mon[31859]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-08T22:59:45.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:44 smithi192 ceph-mon[36700]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-08T22:59:45.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:44 smithi062 ceph-mon[35761]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-03-08T22:59:47.094 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:46 smithi046 ceph-mon[31859]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:46 smithi192 ceph-mon[36700]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:46 smithi062 ceph-mon[35761]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:49.171 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:48 smithi046 ceph-mon[31859]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:49.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:48 smithi192 ceph-mon[36700]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:49.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:48 smithi062 ceph-mon[35761]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-08T22:59:50.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:50.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:50.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:50.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:50.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:50.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:51.056 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:50 smithi046 ceph-mon[31859]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:51.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:50 smithi192 ceph-mon[36700]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:51.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:50 smithi062 ceph-mon[35761]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:53.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:52 smithi046 ceph-mon[31859]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:53.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:52 smithi192 ceph-mon[36700]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:52 smithi062 ceph-mon[35761]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:54 smithi046 ceph-mon[31859]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T22:59:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.590+0000 7f05b3400700 1 -- 172.21.15.46:0/2445563107 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f059c000bf0 con 0x7f05a00a2900 2024-03-08T22:59:54.596 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 3 on host 'smithi046' 2024-03-08T22:59:54.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 msgr2=0x7f05a00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:54.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 0x7f05a00a4dc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f05a8009c60 tx=0x7f05a8009260 comp rx=0 tx=0).stop 2024-03-08T22:59:54.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 msgr2=0x7f05bc19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T22:59:54.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f05ac00b2e0 tx=0x7f05ac00bb90 comp rx=0 tx=0).stop 2024-03-08T22:59:54.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 shutdown_connections 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f05a00a2900 0x7f05a00a4dc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f05bc105ab0 0x7f05bc1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f05bc103f10 0x7f05bc19a120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 --2- 172.21.15.46:0/2445563107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f05bc1050f0 0x7f05bc19a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.594+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 >> 172.21.15.46:0/2445563107 conn(0x7f05bc0ff500 msgr2=0x7f05bc108520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.595+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 shutdown_connections 2024-03-08T22:59:54.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T22:59:54.595+0000 7f05bbe00700 1 -- 172.21.15.46:0/2445563107 wait complete. 2024-03-08T22:59:54.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:54.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:54.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:54.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:54 smithi062 ceph-mon[35761]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T22:59:54.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:54.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T22:59:54.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T22:59:54.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:54.763 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:54 smithi192 ceph-mon[36700]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T22:59:55.368 DEBUG:teuthology.orchestra.run.smithi046:osd.3> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.3.service 2024-03-08T22:59:55.374 INFO:tasks.cephadm:Deploying osd.4 on smithi062 with /dev/vg_nvme/lv_3... 2024-03-08T22:59:55.374 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-08T22:59:55.633 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:55.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:55.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T22:59:55.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T22:59:55.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:59:55.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:55.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:55.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:55.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:55.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:55.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T22:59:56.647 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:56.647 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:59:56.647 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T22:59:56.648 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.648 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.648 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] boot 2024-03-08T22:59:56.648 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T22:59:56.648 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] boot 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T22:59:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: from='osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: osd.3 [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] boot 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T22:59:56.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112339 s, 933 MB/s 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 2024-03-08T22:59:57.503 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-03-08T22:59:57.675 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:57 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T22:59:57.675 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:57 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T22:59:57.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:57 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T22:59:57.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:57 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T22:59:57.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:57 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T22:59:57.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:57 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T22:59:58.212 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi062:vg_nvme/lv_3 2024-03-08T22:59:58.462 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T22:59:58.584 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:58 smithi062 ceph-mon[35761]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T22:59:58.584 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:58 smithi062 ceph-mon[35761]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T22:59:58.584 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:58 smithi062 ceph-mon[35761]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T22:59:58.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:58 smithi046 ceph-mon[31859]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T22:59:58.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:58 smithi046 ceph-mon[31859]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T22:59:58.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:58 smithi046 ceph-mon[31859]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T22:59:58.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:58 smithi192 ceph-mon[36700]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T22:59:58.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:58 smithi192 ceph-mon[36700]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T22:59:58.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:58 smithi192 ceph-mon[36700]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- 172.21.15.62:0/2484709811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 msgr2=0x7f5af4109fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 --2- 172.21.15.62:0/2484709811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af4109fe0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5aec00b0a0 tx=0x7f5aec00b3b0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- 172.21.15.62:0/2484709811 shutdown_connections 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 --2- 172.21.15.62:0/2484709811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af4109fe0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 --2- 172.21.15.62:0/2484709811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4102910 0x7f5af4104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 --2- 172.21.15.62:0/2484709811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4105240 0x7f5af4107630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- 172.21.15.62:0/2484709811 >> 172.21.15.62:0/2484709811 conn(0x7f5af40fc340 msgr2=0x7f5af40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- 172.21.15.62:0/2484709811 shutdown_connections 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- 172.21.15.62:0/2484709811 wait complete. 2024-03-08T23:00:00.095 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 Processor -- start 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.087+0000 7f5afc326700 1 -- start start 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4102910 0x7f5af4197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4105240 0x7f5af4198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410d9f0 con 0x7f5af4105240 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410d870 con 0x7f5af4107b70 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afc326700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410db70 con 0x7f5af4102910 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49090/0 (socket says 172.21.15.62:49090) 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 -- 172.21.15.62:0/1247214894 learned_addr learned my addr 172.21.15.62:0/1247214894 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa0c2700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4102910 0x7f5af4197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5af98c1700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4105240 0x7f5af4198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 -- 172.21.15.62:0/1247214894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4102910 msgr2=0x7f5af4197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:00.096 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4102910 0x7f5af4197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 -- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4105240 msgr2=0x7f5af4198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4105240 0x7f5af4198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 -- 172.21.15.62:0/1247214894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af419fc30 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5afa8c3700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007f00 tx=0x7f5aec004d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.088+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec021070 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.089+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aec007cc0 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.089+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af419fec0 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.089+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec0053e0 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.089+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af41a02c0 con 0x7f5af4107b70 2024-03-08T23:00:00.097 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.089+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af4068aa0 con 0x7f5af4107b70 2024-03-08T23:00:00.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.091+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5aec024070 con 0x7f5af4107b70 2024-03-08T23:00:00.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.091+0000 7f5aeb7fe700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 0x7f5ae00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:00.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.092+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7f5aec0cea90 con 0x7f5af4107b70 2024-03-08T23:00:00.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.092+0000 7f5afa0c2700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 0x7f5ae00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:00.098 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.093+0000 7f5afa0c2700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 0x7f5ae00a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005950 tx=0x7f5af000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:00.101 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.096+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5aec09cde0 con 0x7f5af4107b70 2024-03-08T23:00:00.101 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 22:59:59 smithi062 ceph-mon[35761]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:00:00.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 22:59:59 smithi046 ceph-mon[31859]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:00:00.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 22:59:59 smithi192 ceph-mon[36700]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:00:00.315 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:00.309+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5af4062220 con 0x7f5ae00a2890 2024-03-08T23:00:01.188 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:01.188 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: osdmap e29: 4 total, 4 up, 4 in 2024-03-08T23:00:01.188 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: overall HEALTH_OK 2024-03-08T23:00:01.189 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:01.189 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:01.189 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:00 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: osdmap e29: 4 total, 4 up, 4 in 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: overall HEALTH_OK 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:01.249 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:00 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: osdmap e29: 4 total, 4 up, 4 in 2024-03-08T23:00:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: overall HEALTH_OK 2024-03-08T23:00:01.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:01.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:01.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:00 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:02.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:02.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:01 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.310 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:02.311 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:01 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:02.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:02.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:02.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:02.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:01 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:02 smithi046 ceph-mon[31859]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-08T23:00:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:02 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi046 2024-03-08T23:00:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:02 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 8547M 2024-03-08T23:00:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:02 smithi192 ceph-mon[36700]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-08T23:00:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:02 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi046 2024-03-08T23:00:03.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:02 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 8547M 2024-03-08T23:00:03.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:02 smithi062 ceph-mon[35761]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-08T23:00:03.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:02 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi046 2024-03-08T23:00:03.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:02 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 8547M 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/4247266873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]': finished 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: osdmap e30: 5 total, 4 up, 5 in 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:04.270 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:03 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/1151447140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:04.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/4247266873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]': finished 2024-03-08T23:00:04.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: osdmap e30: 5 total, 4 up, 5 in 2024-03-08T23:00:04.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:04.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:03 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/1151447140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:04.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/4247266873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]: dispatch 2024-03-08T23:00:04.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c5a9bc30-1ba9-4a47-8568-bb143164fc44"}]': finished 2024-03-08T23:00:04.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: osdmap e30: 5 total, 4 up, 5 in 2024-03-08T23:00:04.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:04.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:03 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/1151447140' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:05.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:04 smithi192 ceph-mon[36700]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-08T23:00:05.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:04 smithi062 ceph-mon[35761]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-08T23:00:05.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:04 smithi046 ceph-mon[31859]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-08T23:00:07.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:06 smithi062 ceph-mon[35761]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-08T23:00:07.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:06 smithi192 ceph-mon[36700]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-08T23:00:07.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:06 smithi046 ceph-mon[31859]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-08T23:00:09.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:08 smithi192 ceph-mon[36700]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:00:09.369 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:08 smithi062 ceph-mon[35761]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:00:09.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:08 smithi046 ceph-mon[31859]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:00:11.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:10 smithi062 ceph-mon[35761]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:00:11.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:10 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-08T23:00:11.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:10 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:11.279 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:10 smithi046 ceph-mon[31859]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:00:11.279 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:10 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-08T23:00:11.279 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:10 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:11.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:10 smithi192 ceph-mon[36700]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:00:11.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:10 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-08T23:00:11.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:10 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:12.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:12 smithi062 ceph-mon[35761]: Deploying daemon osd.4 on smithi062 2024-03-08T23:00:12.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:12 smithi046 ceph-mon[31859]: Deploying daemon osd.4 on smithi062 2024-03-08T23:00:12.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:12 smithi192 ceph-mon[36700]: Deploying daemon osd.4 on smithi062 2024-03-08T23:00:13.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:13 smithi062 ceph-mon[35761]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:13.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:13 smithi046 ceph-mon[31859]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:13.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:13 smithi192 ceph-mon[36700]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:15.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:15 smithi192 ceph-mon[36700]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:15 smithi062 ceph-mon[35761]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:15.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:15 smithi046 ceph-mon[31859]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:16.124 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:16 smithi062 ceph-mon[35761]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:16.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:16 smithi192 ceph-mon[36700]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:16.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:16 smithi046 ceph-mon[31859]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:17.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:17.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:17.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:17.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:17.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:17.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:18.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:18.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:18.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:18.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:18 smithi062 ceph-mon[35761]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:18.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:18 smithi046 ceph-mon[31859]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:19.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:18 smithi192 ceph-mon[36700]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:20.354 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:20.354 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.354 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.355 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:20.355 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:20.355 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.452 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.447+0000 7f5aeb7fe700 1 -- 172.21.15.62:0/1247214894 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5af4062220 con 0x7f5ae00a2890 2024-03-08T23:00:20.452 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 4 on host 'smithi062' 2024-03-08T23:00:20.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 msgr2=0x7f5ae00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:20.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 0x7f5ae00a4d50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5af0005950 tx=0x7f5af000a300 comp rx=0 tx=0).stop 2024-03-08T23:00:20.454 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 msgr2=0x7f5af419f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5aec007f00 tx=0x7f5aec004d70 comp rx=0 tx=0).stop 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 shutdown_connections 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5ae00a2890 0x7f5ae00a4d50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5af4107b70 0x7f5af419f590 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5af4102910 0x7f5af4197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 --2- 172.21.15.62:0/1247214894 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5af4105240 0x7f5af4198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 >> 172.21.15.62:0/1247214894 conn(0x7f5af40fc340 msgr2=0x7f5af41031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 shutdown_connections 2024-03-08T23:00:20.455 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:00:20.450+0000 7f5afc326700 1 -- 172.21.15.62:0/1247214894 wait complete. 2024-03-08T23:00:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:20.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:20.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:20.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:20.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:20.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:20.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:20.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.084 DEBUG:teuthology.orchestra.run.smithi062:osd.4> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.4.service 2024-03-08T23:00:21.086 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-03-08T23:00:21.086 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-08T23:00:21.332 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:00:21.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:21.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:21 smithi062 ceph-mon[35761]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:21.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:21 smithi062 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:21.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:21.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:21 smithi192 ceph-mon[36700]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:21.683 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:21 smithi192 ceph-mon[36700]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:21.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:21.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:21.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:21 smithi046 ceph-mon[31859]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:21.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:21 smithi046 ceph-mon[31859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: osdmap e31: 5 total, 4 up, 5 in 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:22.474 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:22.703 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:22.703 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-08T23:00:22.704 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:22.704 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: osdmap e31: 5 total, 4 up, 5 in 2024-03-08T23:00:22.704 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:22.704 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:22 smithi192 ceph-mon[36700]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: from='osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: osdmap e31: 5 total, 4 up, 5 in 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:22.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:22 smithi046 ceph-mon[31859]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:00:23.330 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:00:22.890+0000 7f95b0a63700 -1 osd.4 0 waiting for initial osdmap 2024-03-08T23:00:23.330 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:00:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:00:22.900+0000 7f95a884e700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114276 s, 918 MB/s 2024-03-08T23:00:23.401 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-08T23:00:23.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:23 smithi192 ceph-mon[36700]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:00:23.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:23 smithi192 ceph-mon[36700]: osdmap e32: 5 total, 4 up, 5 in 2024-03-08T23:00:23.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:23 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:23.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:23 smithi192 ceph-mon[36700]: from='osd.4 ' entity='osd.4' 2024-03-08T23:00:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:23 smithi062 ceph-mon[35761]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:00:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:23 smithi062 ceph-mon[35761]: osdmap e32: 5 total, 4 up, 5 in 2024-03-08T23:00:23.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:23 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:23.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:23 smithi062 ceph-mon[35761]: from='osd.4 ' entity='osd.4' 2024-03-08T23:00:23.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:23 smithi046 ceph-mon[31859]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:00:23.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:23 smithi046 ceph-mon[31859]: osdmap e32: 5 total, 4 up, 5 in 2024-03-08T23:00:23.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:23 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:23.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:23 smithi046 ceph-mon[31859]: from='osd.4 ' entity='osd.4' 2024-03-08T23:00:24.071 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-03-08T23:00:24.318 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:00:24.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:00:24.689 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] boot 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: osdmap e33: 5 total, 5 up, 5 in 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.690 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:24.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:24.691 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] boot 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: osdmap e33: 5 total, 5 up, 5 in 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:24.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:24.728 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:00:24.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: osd.4 [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] boot 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: osdmap e33: 5 total, 5 up, 5 in 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:24.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:25.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:25 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi062 2024-03-08T23:00:25.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:25 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 8540M 2024-03-08T23:00:25.756 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:25 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi062 2024-03-08T23:00:25.756 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:25 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 8540M 2024-03-08T23:00:25.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:25 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi062 2024-03-08T23:00:25.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:25 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 8540M 2024-03-08T23:00:25.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 -- 172.21.15.192:0/802141134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 msgr2=0x7f8d8c104c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:25.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 --2- 172.21.15.192:0/802141134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c104c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009990 tx=0x7f8d88009ca0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 -- 172.21.15.192:0/802141134 shutdown_connections 2024-03-08T23:00:25.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 --2- 172.21.15.192:0/802141134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c107af0 0x7f8d8c109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 --2- 172.21.15.192:0/802141134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c104c80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 --2- 172.21.15.192:0/802141134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d8c1051c0 0x7f8d8c1075b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 -- 172.21.15.192:0/802141134 >> 172.21.15.192:0/802141134 conn(0x7f8d8c0fc2a0 msgr2=0x7f8d8c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 -- 172.21.15.192:0/802141134 shutdown_connections 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.973+0000 7f8d9424c700 1 -- 172.21.15.192:0/802141134 wait complete. 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 Processor -- start 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 -- start start 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c1051c0 0x7f8d8c100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d8c107af0 0x7f8d8c100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c10da80 con 0x7f8d8c107af0 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c10d900 con 0x7f8d8c1051c0 2024-03-08T23:00:25.981 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d9424c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8c10dc00 con 0x7f8d8c102890 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:36112/0 (socket says 172.21.15.192:36112) 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 -- 172.21.15.192:0/957576072 learned_addr learned my addr 172.21.15.192:0/957576072 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d917e7700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c1051c0 0x7f8d8c100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 -- 172.21.15.192:0/957576072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c1051c0 msgr2=0x7f8d8c100840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c1051c0 0x7f8d8c100840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 -- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d8c107af0 msgr2=0x7f8d8c100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d8c107af0 0x7f8d8c100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.974+0000 7f8d91fe8700 1 -- 172.21.15.192:0/957576072 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c1015c0 con 0x7f8d8c102890 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d91fe8700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d88005910 tx=0x7f8d880047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d88027070 con 0x7f8d8c102890 2024-03-08T23:00:25.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8c1a2140 con 0x7f8d8c102890 2024-03-08T23:00:25.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8c1a2670 con 0x7f8d8c102890 2024-03-08T23:00:25.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d88022470 con 0x7f8d8c102890 2024-03-08T23:00:25.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.975+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800f6c0 con 0x7f8d8c102890 2024-03-08T23:00:25.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.976+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d8c10e240 con 0x7f8d8c102890 2024-03-08T23:00:25.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.977+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8d8802a070 con 0x7f8d8c102890 2024-03-08T23:00:25.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.978+0000 7f8d82ffd700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 0x7f8d780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:25.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.978+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f8d880cebc0 con 0x7f8d8c102890 2024-03-08T23:00:25.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.978+0000 7f8d917e7700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 0x7f8d780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:25.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.979+0000 7f8d917e7700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 0x7f8d780a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009a10 tx=0x7f8d7c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:25.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:25.982+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8d88013070 con 0x7f8d8c102890 2024-03-08T23:00:26.190 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:26.186+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d8c062220 con 0x7f8d780a2890 2024-03-08T23:00:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:26 smithi046 ceph-mon[31859]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:26 smithi046 ceph-mon[31859]: osdmap e34: 5 total, 5 up, 5 in 2024-03-08T23:00:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:26.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:26 smithi192 ceph-mon[36700]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:26.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:26 smithi192 ceph-mon[36700]: osdmap e34: 5 total, 5 up, 5 in 2024-03-08T23:00:26.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:26.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:26.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:26.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:26 smithi062 ceph-mon[35761]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:26.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:26 smithi062 ceph-mon[35761]: osdmap e34: 5 total, 5 up, 5 in 2024-03-08T23:00:26.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:26.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:26.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:27.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:27 smithi046 ceph-mon[31859]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:27.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:27 smithi192 ceph-mon[36700]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:27.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:27 smithi062 ceph-mon[35761]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:28 smithi192 ceph-mon[36700]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:28.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:28 smithi046 ceph-mon[31859]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:28.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:28 smithi062 ceph-mon[35761]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:29.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/2312487359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:29 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:29 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]': finished 2024-03-08T23:00:29.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:29 smithi046 ceph-mon[31859]: osdmap e35: 6 total, 5 up, 6 in 2024-03-08T23:00:29.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:29.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:29 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/2312487359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:29 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:29 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]': finished 2024-03-08T23:00:29.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:29 smithi192 ceph-mon[36700]: osdmap e35: 6 total, 5 up, 6 in 2024-03-08T23:00:29.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:29.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:29 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/2312487359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:29 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]: dispatch 2024-03-08T23:00:29.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:29 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0d68a52e-f5cd-4087-8d3d-0289330f4a9d"}]': finished 2024-03-08T23:00:29.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:29 smithi062 ceph-mon[35761]: osdmap e35: 6 total, 5 up, 6 in 2024-03-08T23:00:29.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:30 smithi046 ceph-mon[31859]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:30 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/3480042212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:30.794 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:30 smithi062 ceph-mon[35761]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:30.794 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:30 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/3480042212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:30.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:30 smithi192 ceph-mon[36700]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:30.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:30 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/3480042212' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:32.718 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:32 smithi192 ceph-mon[36700]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:32.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:32 smithi046 ceph-mon[31859]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:32.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:32 smithi062 ceph-mon[35761]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:34 smithi192 ceph-mon[36700]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:34 smithi046 ceph-mon[31859]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:34.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:34 smithi062 ceph-mon[35761]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:36.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:36 smithi046 ceph-mon[31859]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:36.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-08T23:00:36.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:36.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:36 smithi192 ceph-mon[36700]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:36.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-08T23:00:36.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:36.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:36 smithi062 ceph-mon[35761]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:36.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-08T23:00:36.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:37.724 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:37 smithi192 ceph-mon[36700]: Deploying daemon osd.5 on smithi192 2024-03-08T23:00:37.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:37 smithi062 ceph-mon[35761]: Deploying daemon osd.5 on smithi192 2024-03-08T23:00:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:37 smithi046 ceph-mon[31859]: Deploying daemon osd.5 on smithi192 2024-03-08T23:00:38.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:38 smithi192 ceph-mon[36700]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:38.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:38 smithi062 ceph-mon[35761]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:38.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:38 smithi046 ceph-mon[31859]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:40 smithi192 ceph-mon[36700]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:40.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:40 smithi062 ceph-mon[35761]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:40.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:40 smithi046 ceph-mon[31859]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:42.779 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:42 smithi046 ceph-mon[31859]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:42.779 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.780 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.780 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:42 smithi192 ceph-mon[36700]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:42.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:42 smithi062 ceph-mon[35761]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:42.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:42.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:44.764 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:44 smithi192 ceph-mon[36700]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:44.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:44 smithi062 ceph-mon[35761]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:44.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:44 smithi046 ceph-mon[31859]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:45.733 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.728+0000 7f8d82ffd700 1 -- 172.21.15.192:0/957576072 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8d8c062220 con 0x7f8d780a2890 2024-03-08T23:00:45.734 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-03-08T23:00:45.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 msgr2=0x7f8d780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 0x7f8d780a4d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8d7c009a10 tx=0x7f8d7c009190 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 msgr2=0x7f8d8c102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d88005910 tx=0x7f8d880047b0 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 shutdown_connections 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8d780a2890 0x7f8d780a4d50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d8c107af0 0x7f8d8c100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d8c102890 0x7f8d8c102180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 --2- 172.21.15.192:0/957576072 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d8c1051c0 0x7f8d8c100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.730+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 >> 172.21.15.192:0/957576072 conn(0x7f8d8c0fc2a0 msgr2=0x7f8d8c103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.731+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 shutdown_connections 2024-03-08T23:00:45.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:00:45.731+0000 7f8d9424c700 1 -- 172.21.15.192:0/957576072 wait complete. 2024-03-08T23:00:46.434 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.5.service 2024-03-08T23:00:46.436 INFO:tasks.cephadm:Deploying osd.6 on smithi046 with /dev/vg_nvme/lv_2... 2024-03-08T23:00:46.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-08T23:00:46.917 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:00:46.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:46.978 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:46 smithi046 ceph-mon[31859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:47.002 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:47.003 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:46 smithi192 ceph-mon[36700]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:47.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:47.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:47.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:47.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:46 smithi062 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-08T23:00:47.984 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-mon[36700]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-08T23:00:47.984 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-mon[36700]: osdmap e36: 6 total, 5 up, 6 in 2024-03-08T23:00:47.984 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:47.984 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-mon[36700]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:47.984 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-mon[36700]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:47.984 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:00:47.824+0000 7efec1f8b700 -1 osd.5 0 waiting for initial osdmap 2024-03-08T23:00:47.985 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:00:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:00:47.830+0000 7efebdd7e700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:00:48.116 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:47 smithi046 ceph-mon[31859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-08T23:00:48.116 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:47 smithi046 ceph-mon[31859]: osdmap e36: 6 total, 5 up, 6 in 2024-03-08T23:00:48.116 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:47 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:48.116 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:47 smithi046 ceph-mon[31859]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:48.116 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:47 smithi046 ceph-mon[31859]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:48.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:47 smithi062 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-08T23:00:48.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:47 smithi062 ceph-mon[35761]: osdmap e36: 6 total, 5 up, 6 in 2024-03-08T23:00:48.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:47 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:48.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:47 smithi062 ceph-mon[35761]: from='osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:48.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:47 smithi062 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:00:49.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:48 smithi192 ceph-mon[36700]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:49.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:48 smithi192 ceph-mon[36700]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:00:49.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:48 smithi192 ceph-mon[36700]: osdmap e37: 6 total, 5 up, 6 in 2024-03-08T23:00:49.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:48 smithi062 ceph-mon[35761]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:48 smithi062 ceph-mon[35761]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:00:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:48 smithi062 ceph-mon[35761]: osdmap e37: 6 total, 5 up, 6 in 2024-03-08T23:00:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:48 smithi046 ceph-mon[31859]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-03-08T23:00:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:48 smithi046 ceph-mon[31859]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:00:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:48 smithi046 ceph-mon[31859]: osdmap e37: 6 total, 5 up, 6 in 2024-03-08T23:00:49.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-08T23:00:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-08T23:00:49.801 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-08T23:00:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-08T23:00:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0334631 s, 313 MB/s 2024-03-08T23:00:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-08T23:00:49.896 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:00:49.896 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:00:49.896 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] boot 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: osdmap e38: 6 total, 6 up, 6 in 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: osdmap e39: 6 total, 6 up, 6 in 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:49.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:00:50.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:00:50.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] boot 2024-03-08T23:00:50.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: osdmap e38: 6 total, 6 up, 6 in 2024-03-08T23:00:50.145 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:50.145 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: osdmap e39: 6 total, 6 up, 6 in 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:50.146 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: osd.5 [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] boot 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: osdmap e38: 6 total, 6 up, 6 in 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: osdmap e39: 6 total, 6 up, 6 in 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:50.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:50.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.704 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_2 2024-03-08T23:00:50.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi192 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:50.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:50 smithi046 ceph-mon[31859]: osdmap e40: 6 total, 6 up, 6 in 2024-03-08T23:00:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi192 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:50 smithi062 ceph-mon[35761]: osdmap e40: 6 total, 6 up, 6 in 2024-03-08T23:00:51.149 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:00:51.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi192 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 8540M 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:00:51.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:50 smithi192 ceph-mon[36700]: osdmap e40: 6 total, 6 up, 6 in 2024-03-08T23:00:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:51 smithi192 ceph-mon[36700]: osdmap e41: 6 total, 6 up, 6 in 2024-03-08T23:00:52.332 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:51 smithi046 ceph-mon[31859]: osdmap e41: 6 total, 6 up, 6 in 2024-03-08T23:00:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:51 smithi062 ceph-mon[35761]: osdmap e41: 6 total, 6 up, 6 in 2024-03-08T23:00:53.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:52 smithi046 ceph-mon[31859]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-03-08T23:00:53.072 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:52 smithi046 ceph-mon[31859]: osdmap e42: 6 total, 6 up, 6 in 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 -- 172.21.15.46:0/2048013245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 msgr2=0x7f565c10c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 --2- 172.21.15.46:0/2048013245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c10c1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f565400b0a0 tx=0x7f565400b3b0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 -- 172.21.15.46:0/2048013245 shutdown_connections 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 --2- 172.21.15.46:0/2048013245 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c10c1a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 --2- 172.21.15.46:0/2048013245 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c075d90 0x7f565c0741f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 --2- 172.21.15.46:0/2048013245 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c0747c0 0x7f565c074c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 -- 172.21.15.46:0/2048013245 >> 172.21.15.46:0/2048013245 conn(0x7f565c0fd390 msgr2=0x7f565c0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 -- 172.21.15.46:0/2048013245 shutdown_connections 2024-03-08T23:00:53.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.242+0000 7f5661600700 1 -- 172.21.15.46:0/2048013245 wait complete. 2024-03-08T23:00:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.244+0000 7f5661600700 1 Processor -- start 2024-03-08T23:00:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.244+0000 7f5661600700 1 -- start start 2024-03-08T23:00:53.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.244+0000 7f5661600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.244+0000 7f5661600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c075d90 0x7f565c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f5661600700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f5661600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c10fbd0 con 0x7f565c0747c0 2024-03-08T23:00:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f5661600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c10fa50 con 0x7f565c075d90 2024-03-08T23:00:53.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f5661600700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c10fd50 con 0x7f565c109d30 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565aa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565aa00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36426/0 (socket says 172.21.15.46:36426) 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565a000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c075d90 0x7f565c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565aa00700 1 -- 172.21.15.46:0/266031094 learned_addr learned my addr 172.21.15.46:0/266031094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 -- 172.21.15.46:0/266031094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c075d90 msgr2=0x7f565c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c075d90 0x7f565c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 -- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 msgr2=0x7f565c19a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:00:53.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565b400700 1 -- 172.21.15.46:0/266031094 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c1a1dc0 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.245+0000 7f565aa00700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f565b400700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c1a1720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f565400b070 tx=0x7f5654004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654021070 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5654007b80 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c1a2050 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654004800 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.246+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c1a2500 con 0x7f565c109d30 2024-03-08T23:00:53.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.247+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c068aa0 con 0x7f565c109d30 2024-03-08T23:00:53.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.250+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5654003a00 con 0x7f565c109d30 2024-03-08T23:00:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.251+0000 7f5653400700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 0x7f56400a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:00:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.251+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f565400f020 con 0x7f565c109d30 2024-03-08T23:00:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.251+0000 7f565aa00700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 0x7f56400a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:00:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.252+0000 7f565aa00700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 0x7f56400a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f564c011630 tx=0x7f564c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:00:53.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.257+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f565402a020 con 0x7f565c109d30 2024-03-08T23:00:53.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:52 smithi192 ceph-mon[36700]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-03-08T23:00:53.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:52 smithi192 ceph-mon[36700]: osdmap e42: 6 total, 6 up, 6 in 2024-03-08T23:00:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:52 smithi062 ceph-mon[35761]: pgmap v157: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 175 KiB/s, 0 objects/s recovering 2024-03-08T23:00:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:52 smithi062 ceph-mon[35761]: osdmap e42: 6 total, 6 up, 6 in 2024-03-08T23:00:53.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:00:53.502+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f565c062220 con 0x7f56400a2890 2024-03-08T23:00:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:54.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:00:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:00:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:00:55.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:54 smithi046 ceph-mon[31859]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-08T23:00:55.249 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:54 smithi046 ceph-mon[31859]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:54 smithi192 ceph-mon[36700]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-08T23:00:55.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:54 smithi192 ceph-mon[36700]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:54 smithi062 ceph-mon[35761]: pgmap v159: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 139 KiB/s, 0 objects/s recovering 2024-03-08T23:00:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:54 smithi062 ceph-mon[35761]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:00:57.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:56 smithi192 ceph-mon[36700]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:00:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:56 smithi062 ceph-mon[35761]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:00:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:56 smithi046 ceph-mon[31859]: pgmap v160: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:00:58.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:58 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]: dispatch 2024-03-08T23:00:58.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:58 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]': finished 2024-03-08T23:00:58.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:58 smithi192 ceph-mon[36700]: osdmap e43: 7 total, 6 up, 7 in 2024-03-08T23:00:58.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:58 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:00:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:58 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]: dispatch 2024-03-08T23:00:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:58 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]': finished 2024-03-08T23:00:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:58 smithi062 ceph-mon[35761]: osdmap e43: 7 total, 6 up, 7 in 2024-03-08T23:00:58.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:58 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:00:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]: dispatch 2024-03-08T23:00:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2816478097' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b897988-ec0f-4984-bc95-ec8014984db8"}]': finished 2024-03-08T23:00:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:58 smithi046 ceph-mon[31859]: osdmap e43: 7 total, 6 up, 7 in 2024-03-08T23:00:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:58 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:00:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:59 smithi192 ceph-mon[36700]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:00:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:00:59 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/445599440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:59 smithi062 ceph-mon[35761]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:00:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:00:59 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/445599440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:00:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:59 smithi046 ceph-mon[31859]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:00:59.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:00:59 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/445599440' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:01 smithi192 ceph-mon[36700]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:01.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:01 smithi062 ceph-mon[35761]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:01.412 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:01 smithi046 ceph-mon[31859]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:02.318 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:02 smithi046 ceph-mon[31859]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:02.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:02 smithi062 ceph-mon[35761]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:02.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:02 smithi192 ceph-mon[36700]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:04.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:04 smithi046 ceph-mon[31859]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:04.484 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:04 smithi192 ceph-mon[36700]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:04.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:04 smithi062 ceph-mon[35761]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:06.423 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:06 smithi046 ceph-mon[31859]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:06.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:06 smithi192 ceph-mon[36700]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:06.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:06 smithi062 ceph-mon[35761]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:07.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-08T23:01:07.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:07.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:07 smithi046 ceph-mon[31859]: Deploying daemon osd.6 on smithi046 2024-03-08T23:01:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-08T23:01:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:07 smithi192 ceph-mon[36700]: Deploying daemon osd.6 on smithi046 2024-03-08T23:01:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-08T23:01:07.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:07.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:07 smithi062 ceph-mon[35761]: Deploying daemon osd.6 on smithi046 2024-03-08T23:01:08.205 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:08 smithi046 ceph-mon[31859]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:08.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:08 smithi192 ceph-mon[36700]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:08.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:08 smithi062 ceph-mon[35761]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:10.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:10 smithi046 ceph-mon[31859]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:10 smithi192 ceph-mon[36700]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:10.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:10 smithi062 ceph-mon[35761]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:12.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:12 smithi046 ceph-mon[31859]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:12.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:12 smithi192 ceph-mon[36700]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:12 smithi062 ceph-mon[35761]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:14.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:14 smithi046 ceph-mon[31859]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:14 smithi192 ceph-mon[36700]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:14.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:14 smithi062 ceph-mon[35761]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:16.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:16 smithi046 ceph-mon[31859]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:16.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:16.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:16 smithi192 ceph-mon[36700]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:16.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:16.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:16 smithi062 ceph-mon[35761]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:16.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:16.893 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:18.755 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:18 smithi046 ceph-mon[31859]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:18.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:18 smithi192 ceph-mon[36700]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:18.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:18 smithi062 ceph-mon[35761]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:19.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:19 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:19.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:19 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:19.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:19 smithi046 ceph-mon[31859]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:19.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:19 smithi046 ceph-mon[31859]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:20.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:19 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:20.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:19 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:20.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:19 smithi062 ceph-mon[35761]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:20.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:19 smithi062 ceph-mon[35761]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:20.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:19 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:19 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:19 smithi192 ceph-mon[36700]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:20.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:19 smithi192 ceph-mon[36700]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-08T23:01:20.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.599+0000 7f5653400700 1 -- 172.21.15.46:0/266031094 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f565c062220 con 0x7f56400a2890 2024-03-08T23:01:20.605 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 6 on host 'smithi046' 2024-03-08T23:01:20.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 msgr2=0x7f56400a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 0x7f56400a4d50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f564c011630 tx=0x7f564c009260 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 msgr2=0x7f565c1a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c1a1720 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f565400b070 tx=0x7f5654004ce0 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 shutdown_connections 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56400a2890 0x7f56400a4d50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f565c109d30 0x7f565c1a1720 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f565c0747c0 0x7f565c19a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 --2- 172.21.15.46:0/266031094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f565c075d90 0x7f565c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 >> 172.21.15.46:0/266031094 conn(0x7f565c0fd390 msgr2=0x7f565c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 shutdown_connections 2024-03-08T23:01:20.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:01:20.603+0000 7f5661600700 1 -- 172.21.15.46:0/266031094 wait complete. 2024-03-08T23:01:21.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: osdmap e44: 7 total, 6 up, 7 in 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:21.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:21.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: osdmap e44: 7 total, 6 up, 7 in 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: osdmap e44: 7 total, 6 up, 7 in 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:21.451 DEBUG:teuthology.orchestra.run.smithi046:osd.6> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.6.service 2024-03-08T23:01:21.456 INFO:tasks.cephadm:Deploying osd.7 on smithi062 with /dev/vg_nvme/lv_2... 2024-03-08T23:01:21.456 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-08T23:01:21.654 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:01:21.555+0000 7fab07800700 -1 osd.6 0 waiting for initial osdmap 2024-03-08T23:01:21.655 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:01:21.588+0000 7faafd600700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:01:21.715 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:01:21.941 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-mon[31859]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:01:21.941 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-mon[31859]: osdmap e45: 7 total, 6 up, 7 in 2024-03-08T23:01:21.941 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:21.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:21.942 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:21 smithi046 ceph-mon[31859]: from='osd.6 ' entity='osd.6' 2024-03-08T23:01:22.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:21 smithi062 ceph-mon[35761]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:01:22.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:21 smithi062 ceph-mon[35761]: osdmap e45: 7 total, 6 up, 7 in 2024-03-08T23:01:22.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:22.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:22.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:21 smithi062 ceph-mon[35761]: from='osd.6 ' entity='osd.6' 2024-03-08T23:01:22.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:21 smithi192 ceph-mon[36700]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:01:22.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:21 smithi192 ceph-mon[36700]: osdmap e45: 7 total, 6 up, 7 in 2024-03-08T23:01:22.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:22.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:22.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:21 smithi192 ceph-mon[36700]: from='osd.6 ' entity='osd.6' 2024-03-08T23:01:23.101 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:01:23.101 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:01:23.101 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:23.102 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.102 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] boot 2024-03-08T23:01:23.102 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: osdmap e46: 7 total, 7 up, 7 in 2024-03-08T23:01:23.102 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.216 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:01:23.216 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:01:23.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:23.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] boot 2024-03-08T23:01:23.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: osdmap e46: 7 total, 7 up, 7 in 2024-03-08T23:01:23.217 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: osd.6 [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] boot 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: osdmap e46: 7 total, 7 up, 7 in 2024-03-08T23:01:23.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112937 s, 928 MB/s 2024-03-08T23:01:23.790 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-03-08T23:01:24.412 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi062:vg_nvme/lv_2 2024-03-08T23:01:24.667 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:01:25.003 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:24 smithi062 ceph-mon[35761]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:25.003 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:24 smithi062 ceph-mon[35761]: osdmap e47: 7 total, 7 up, 7 in 2024-03-08T23:01:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:24 smithi046 ceph-mon[31859]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:24 smithi046 ceph-mon[31859]: osdmap e47: 7 total, 7 up, 7 in 2024-03-08T23:01:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:24 smithi192 ceph-mon[36700]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:24 smithi192 ceph-mon[36700]: osdmap e47: 7 total, 7 up, 7 in 2024-03-08T23:01:26.327 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 -- 172.21.15.62:0/2103236401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6441028b0 msgr2=0x7fd644109160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 --2- 172.21.15.62:0/2103236401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6441028b0 0x7fd644109160 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd63c0099c0 tx=0x7fd63c009cd0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 -- 172.21.15.62:0/2103236401 shutdown_connections 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 --2- 172.21.15.62:0/2103236401 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6441028b0 0x7fd644109160 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 --2- 172.21.15.62:0/2103236401 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd644100d10 0x7fd644101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 --2- 172.21.15.62:0/2103236401 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644101f10 0x7fd644102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 -- 172.21.15.62:0/2103236401 >> 172.21.15.62:0/2103236401 conn(0x7fd6440fc2b0 msgr2=0x7fd6440fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 -- 172.21.15.62:0/2103236401 shutdown_connections 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.321+0000 7fd649444700 1 -- 172.21.15.62:0/2103236401 wait complete. 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 Processor -- start 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 -- start start 2024-03-08T23:01:26.328 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644100d10 0x7fd644198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6441028b0 0x7fd64419f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64410dad0 con 0x7fd6441028b0 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64406a9c0 con 0x7fd644101f10 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd649444700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64410dc50 con 0x7fd644100d10 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd6427fc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd6427fc700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:52382/0 (socket says 172.21.15.62:52382) 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd6427fc700 1 -- 172.21.15.62:0/2353314111 learned_addr learned my addr 172.21.15.62:0/2353314111 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.322+0000 7fd642ffd700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644100d10 0x7fd644198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 -- 172.21.15.62:0/2353314111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644100d10 msgr2=0x7fd644198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644100d10 0x7fd644198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6437fe700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6441028b0 0x7fd64419f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:26.329 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 -- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6441028b0 msgr2=0x7fd64419f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6441028b0 0x7fd64419f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 -- 172.21.15.62:0/2353314111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd64419fcc0 con 0x7fd644101f10 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd6427fc700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c920 tx=0x7fd62c00cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c007b00 con 0x7fd644101f10 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64419ffb0 con 0x7fd644101f10 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd62c004d10 con 0x7fd644101f10 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c005680 con 0x7fd644101f10 2024-03-08T23:01:26.330 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.323+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6441a04e0 con 0x7fd644101f10 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.324+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd644068aa0 con 0x7fd644101f10 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.325+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd62c004750 con 0x7fd644101f10 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.326+0000 7fd63bfff700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 0x7fd6300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.326+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7fd62c0cd8b0 con 0x7fd644101f10 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.326+0000 7fd642ffd700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 0x7fd6300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:26.331 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.327+0000 7fd642ffd700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 0x7fd6300a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd644101d70 tx=0x7fd63400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:01:26.334 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.330+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd62c014030 con 0x7fd644101f10 2024-03-08T23:01:26.549 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:26.545+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd644062220 con 0x7fd6300a2890 2024-03-08T23:01:27.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:27.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:27.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:27.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:27.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:27.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:28.214 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:27 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi046 2024-03-08T23:01:28.214 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:27 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 5698M 2024-03-08T23:01:28.214 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:27 smithi062 ceph-mon[35761]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:28.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:27 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi046 2024-03-08T23:01:28.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:27 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 5698M 2024-03-08T23:01:28.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:27 smithi046 ceph-mon[31859]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:28.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:27 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi046 2024-03-08T23:01:28.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:27 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 5698M 2024-03-08T23:01:28.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:27 smithi192 ceph-mon[36700]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:29.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:28 smithi062 ceph-mon[35761]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:29.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:28 smithi046 ceph-mon[31859]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:29.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:28 smithi192 ceph-mon[36700]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:29 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/1840046338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:29 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:29 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]': finished 2024-03-08T23:01:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:29 smithi062 ceph-mon[35761]: osdmap e48: 8 total, 7 up, 8 in 2024-03-08T23:01:30.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/1840046338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:29 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:29 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]': finished 2024-03-08T23:01:30.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:29 smithi046 ceph-mon[31859]: osdmap e48: 8 total, 7 up, 8 in 2024-03-08T23:01:30.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:29 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/1840046338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:29 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]: dispatch 2024-03-08T23:01:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:29 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "da72fad7-cbc8-48e4-8941-79da443bfc2a"}]': finished 2024-03-08T23:01:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:29 smithi192 ceph-mon[36700]: osdmap e48: 8 total, 7 up, 8 in 2024-03-08T23:01:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:31.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:30 smithi046 ceph-mon[31859]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:31.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:30 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/1654814224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:31.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:30 smithi192 ceph-mon[36700]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:31.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:30 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/1654814224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:31.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:30 smithi062 ceph-mon[35761]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:31.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:30 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/1654814224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:33.171 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:32 smithi062 ceph-mon[35761]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:33.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:32 smithi046 ceph-mon[31859]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:32 smithi192 ceph-mon[36700]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:34 smithi062 ceph-mon[35761]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:34 smithi046 ceph-mon[31859]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:34 smithi192 ceph-mon[36700]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:37.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:36 smithi046 ceph-mon[31859]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:37.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-08T23:01:37.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:37.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:36 smithi192 ceph-mon[36700]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:37.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-08T23:01:37.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:37.353 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:36 smithi062 ceph-mon[35761]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:37.353 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-08T23:01:37.353 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:38.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:37 smithi046 ceph-mon[31859]: Deploying daemon osd.7 on smithi062 2024-03-08T23:01:38.264 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:37 smithi062 ceph-mon[35761]: Deploying daemon osd.7 on smithi062 2024-03-08T23:01:38.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:37 smithi192 ceph-mon[36700]: Deploying daemon osd.7 on smithi062 2024-03-08T23:01:39.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:38 smithi046 ceph-mon[31859]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:39.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:38 smithi192 ceph-mon[36700]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:39.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:38 smithi062 ceph-mon[35761]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:41.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:40 smithi046 ceph-mon[31859]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:41.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:40 smithi192 ceph-mon[36700]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:41.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:40 smithi062 ceph-mon[35761]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:43.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:42 smithi062 ceph-mon[35761]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:43.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.069 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:43.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:42 smithi046 ceph-mon[31859]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:43.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:43.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:42 smithi192 ceph-mon[36700]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:43.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:43.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:44.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:44 smithi046 ceph-mon[31859]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:45.079 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:44 smithi062 ceph-mon[35761]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:45.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:44 smithi192 ceph-mon[36700]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:46.127 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.122+0000 7fd63bfff700 1 -- 172.21.15.62:0/2353314111 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd644062220 con 0x7fd6300a2890 2024-03-08T23:01:46.127 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 7 on host 'smithi062' 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.124+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 msgr2=0x7fd6300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.124+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 0x7fd6300a4d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd644101d70 tx=0x7fd63400d040 comp rx=0 tx=0).stop 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 msgr2=0x7fd6441985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd62c00c920 tx=0x7fd62c00cc30 comp rx=0 tx=0).stop 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 shutdown_connections 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6300a2890 0x7fd6300a4d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6441028b0 0x7fd64419f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd644100d10 0x7fd644198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:46.129 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 --2- 172.21.15.62:0/2353314111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd644101f10 0x7fd6441985a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 >> 172.21.15.62:0/2353314111 conn(0x7fd6440fc2b0 msgr2=0x7fd6441052e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:01:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 shutdown_connections 2024-03-08T23:01:46.130 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:01:46.125+0000 7fd649444700 1 -- 172.21.15.62:0/2353314111 wait complete. 2024-03-08T23:01:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:46 smithi062 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:46 smithi192 ceph-mon[36700]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:46 smithi046 ceph-mon[31859]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-08T23:01:46.800 DEBUG:teuthology.orchestra.run.smithi062:osd.7> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.7.service 2024-03-08T23:01:46.802 INFO:tasks.cephadm:Deploying osd.8 on smithi192 with /dev/vg_nvme/lv_2... 2024-03-08T23:01:46.802 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-08T23:01:47.048 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: osdmap e49: 8 total, 7 up, 8 in 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:48.271 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:48.271 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:01:48.134+0000 7fce2e5f3700 -1 osd.7 0 waiting for initial osdmap 2024-03-08T23:01:48.271 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:01:48 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:01:48.144+0000 7fce2a3e6700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: osdmap e49: 8 total, 7 up, 8 in 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:48.394 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:48 smithi192 ceph-mon[36700]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-08T23:01:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: from='osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:48.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:48.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: osdmap e49: 8 total, 7 up, 8 in 2024-03-08T23:01:48.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:48.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:48 smithi046 ceph-mon[31859]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113035 s, 928 MB/s 2024-03-08T23:01:49.100 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-08T23:01:49.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: osdmap e50: 8 total, 7 up, 8 in 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] boot 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: osdmap e51: 8 total, 8 up, 8 in 2024-03-08T23:01:49.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: osdmap e50: 8 total, 7 up, 8 in 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] boot 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: osdmap e51: 8 total, 8 up, 8 in 2024-03-08T23:01:49.403 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:01:49.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: osdmap e50: 8 total, 7 up, 8 in 2024-03-08T23:01:49.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: osd.7 [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] boot 2024-03-08T23:01:49.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: osdmap e51: 8 total, 8 up, 8 in 2024-03-08T23:01:49.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:01:49.812 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-03-08T23:01:50.058 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:01:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:01:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:01:50.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi062 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 5693M 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-08T23:01:50.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:50 smithi062 ceph-mon[35761]: osdmap e52: 8 total, 8 up, 8 in 2024-03-08T23:01:50.404 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:01:50.404 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:01:50.404 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:50.404 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi062 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 5693M 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-08T23:01:50.405 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:50 smithi192 ceph-mon[36700]: osdmap e52: 8 total, 8 up, 8 in 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi062 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:01:50.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 5693M 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-08T23:01:50.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:50 smithi046 ceph-mon[31859]: osdmap e52: 8 total, 8 up, 8 in 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.676+0000 7fd5bb40a700 1 -- 172.21.15.192:0/3653974925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b4101f40 msgr2=0x7fd5b41023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.676+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/3653974925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b4101f40 0x7fd5b41023a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4009990 tx=0x7fd5a4009ca0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- 172.21.15.192:0/3653974925 shutdown_connections 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/3653974925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b41028e0 0x7fd5b4109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/3653974925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4100d40 0x7fd5b4101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/3653974925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b4101f40 0x7fd5b41023a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- 172.21.15.192:0/3653974925 >> 172.21.15.192:0/3653974925 conn(0x7fd5b40fc300 msgr2=0x7fd5b40fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:01:51.684 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- 172.21.15.192:0/3653974925 shutdown_connections 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- 172.21.15.192:0/3653974925 wait complete. 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 Processor -- start 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- start start 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b4100d40 0x7fd5b406b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4101f40 0x7fd5b419cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.677+0000 7fd5bb40a700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b410c980 con 0x7fd5b4100d40 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5bb40a700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b410c800 con 0x7fd5b4101f40 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5bb40a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b410cb00 con 0x7fd5b41028e0 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42626/0 (socket says 172.21.15.192:42626) 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 -- 172.21.15.192:0/1084798385 learned_addr learned my addr 172.21.15.192:0/1084798385 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T23:01:51.685 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b89a5700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4101f40 0x7fd5b419cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 -- 172.21.15.192:0/1084798385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4101f40 msgr2=0x7fd5b419cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4101f40 0x7fd5b419cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 -- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b4100d40 msgr2=0x7fd5b406b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b4100d40 0x7fd5b406b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 -- 172.21.15.192:0/1084798385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b419fc90 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5b99a7700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00b970 tx=0x7fd5ac00bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac019070 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5ac00ce10 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac007370 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b419ff80 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.678+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b41a0460 con 0x7fd5b41028e0 2024-03-08T23:01:51.686 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.679+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b4068aa0 con 0x7fd5b41028e0 2024-03-08T23:01:51.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.681+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd5ac01c070 con 0x7fd5b41028e0 2024-03-08T23:01:51.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.681+0000 7fd5aa7fc700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 0x7fd5a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:01:51.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.681+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7fd5ac0cd710 con 0x7fd5b41028e0 2024-03-08T23:01:51.687 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.682+0000 7fd5b91a6700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 0x7fd5a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:01:51.688 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.683+0000 7fd5b91a6700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 0x7fd5a00a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4101da0 tx=0x7fd5b0009470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:01:51.689 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.685+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd5ac09b2f0 con 0x7fd5b41028e0 2024-03-08T23:01:51.892 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:01:51.887+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5b4062220 con 0x7fd5a00a2890 2024-03-08T23:01:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:52 smithi192 ceph-mon[36700]: osdmap e53: 8 total, 8 up, 8 in 2024-03-08T23:01:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:52.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:52 smithi062 ceph-mon[35761]: osdmap e53: 8 total, 8 up, 8 in 2024-03-08T23:01:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:52.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:52 smithi046 ceph-mon[31859]: osdmap e53: 8 total, 8 up, 8 in 2024-03-08T23:01:52.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:01:52.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:01:52.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:01:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:53 smithi192 ceph-mon[36700]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:53 smithi192 ceph-mon[36700]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:53 smithi192 ceph-mon[36700]: osdmap e54: 8 total, 8 up, 8 in 2024-03-08T23:01:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:53 smithi062 ceph-mon[35761]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:53 smithi062 ceph-mon[35761]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:53 smithi062 ceph-mon[35761]: osdmap e54: 8 total, 8 up, 8 in 2024-03-08T23:01:53.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:53 smithi046 ceph-mon[31859]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:53.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:53 smithi046 ceph-mon[31859]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:01:53.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:53 smithi046 ceph-mon[31859]: osdmap e54: 8 total, 8 up, 8 in 2024-03-08T23:01:54.131 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:54 smithi192 ceph-mon[36700]: osdmap e55: 8 total, 8 up, 8 in 2024-03-08T23:01:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:54 smithi062 ceph-mon[35761]: osdmap e55: 8 total, 8 up, 8 in 2024-03-08T23:01:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:54 smithi046 ceph-mon[31859]: osdmap e55: 8 total, 8 up, 8 in 2024-03-08T23:01:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/651800321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]': finished 2024-03-08T23:01:55.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: osdmap e56: 9 total, 8 up, 9 in 2024-03-08T23:01:55.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:55 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:01:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/651800321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]': finished 2024-03-08T23:01:55.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: osdmap e56: 9 total, 8 up, 9 in 2024-03-08T23:01:55.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:55 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:01:55.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:55.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/651800321' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]: dispatch 2024-03-08T23:01:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a65ba870-b01f-4b9a-847c-d0aee0440027"}]': finished 2024-03-08T23:01:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: osdmap e56: 9 total, 8 up, 9 in 2024-03-08T23:01:55.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:55 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:01:56.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:56 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/4170702117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:56.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:56 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/4170702117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:56.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:56 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/4170702117' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:01:57.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:57 smithi192 ceph-mon[36700]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:57.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:57 smithi192 ceph-mon[36700]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-08T23:01:57.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:57 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:01:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:57 smithi062 ceph-mon[35761]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:57 smithi062 ceph-mon[35761]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-08T23:01:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:57 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:01:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:57 smithi046 ceph-mon[31859]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:57 smithi046 ceph-mon[31859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-08T23:01:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:57 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:01:58.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:01:58 smithi192 ceph-mon[36700]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:01:58 smithi062 ceph-mon[35761]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:01:58.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:01:58 smithi046 ceph-mon[31859]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:00.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:00 smithi046 ceph-mon[31859]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:00.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:00 smithi192 ceph-mon[36700]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:00.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:00 smithi062 ceph-mon[35761]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:02.444 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:02 smithi062 ceph-mon[35761]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:02.445 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:02 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-08T23:02:02.445 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:02 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:02.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:02 smithi046 ceph-mon[31859]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:02.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:02 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-08T23:02:02.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:02 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:02.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:02 smithi192 ceph-mon[36700]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:02.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:02 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-08T23:02:02.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:02 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:03.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:03 smithi046 ceph-mon[31859]: Deploying daemon osd.8 on smithi192 2024-03-08T23:02:03.491 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:03 smithi192 ceph-mon[36700]: Deploying daemon osd.8 on smithi192 2024-03-08T23:02:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:03 smithi062 ceph-mon[35761]: Deploying daemon osd.8 on smithi192 2024-03-08T23:02:04.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:04 smithi046 ceph-mon[31859]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:04.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:04 smithi192 ceph-mon[36700]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:04.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:04 smithi062 ceph-mon[35761]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:06.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:06 smithi046 ceph-mon[31859]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:06.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:06 smithi192 ceph-mon[36700]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:06.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:06 smithi062 ceph-mon[35761]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:08 smithi062 ceph-mon[35761]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:08 smithi046 ceph-mon[31859]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:09.238 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:08 smithi192 ceph-mon[36700]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:09.238 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.238 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:09.239 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:11.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:10 smithi192 ceph-mon[36700]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:11.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:10 smithi046 ceph-mon[31859]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:11.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:10 smithi062 ceph-mon[35761]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:11.830 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.826+0000 7fd5aa7fc700 1 -- 172.21.15.192:0/1084798385 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fd5b4062220 con 0x7fd5a00a2890 2024-03-08T23:02:11.831 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 8 on host 'smithi192' 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 msgr2=0x7fd5a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 0x7fd5a00a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4101da0 tx=0x7fd5b0009470 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 msgr2=0x7fd5b419f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac00b970 tx=0x7fd5ac00bc80 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 shutdown_connections 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5a00a2890 0x7fd5a00a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5b41028e0 0x7fd5b419f3c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5b4100d40 0x7fd5b406b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 --2- 172.21.15.192:0/1084798385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5b4101f40 0x7fd5b419cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 >> 172.21.15.192:0/1084798385 conn(0x7fd5b40fc300 msgr2=0x7fd5b41051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 shutdown_connections 2024-03-08T23:02:11.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:02:11.828+0000 7fd5bb40a700 1 -- 172.21.15.192:0/1084798385 wait complete. 2024-03-08T23:02:12.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:12.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:12.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.446 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:12.447 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.472 DEBUG:teuthology.orchestra.run.smithi192:osd.8> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.8.service 2024-03-08T23:02:12.475 INFO:tasks.cephadm:Deploying osd.9 on smithi046 with /dev/vg_nvme/lv_1... 2024-03-08T23:02:12.475 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-08T23:02:12.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:12.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:12.942 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:02:13.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:13 smithi062 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-08T23:02:13.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:13 smithi062 ceph-mon[35761]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:13 smithi062 ceph-mon[35761]: osdmap e57: 9 total, 8 up, 9 in 2024-03-08T23:02:13.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:13 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:13.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:13 smithi062 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.416 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:13 smithi046 ceph-mon[31859]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-08T23:02:13.416 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:13 smithi046 ceph-mon[31859]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.416 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:13 smithi046 ceph-mon[31859]: osdmap e57: 9 total, 8 up, 9 in 2024-03-08T23:02:13.416 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:13.417 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:13 smithi046 ceph-mon[31859]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-mon[36700]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-08T23:02:13.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-mon[36700]: from='osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-mon[36700]: osdmap e57: 9 total, 8 up, 9 in 2024-03-08T23:02:13.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:13.549 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-mon[36700]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:02:13.814 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:02:13.544+0000 7f5c7cab7700 -1 osd.8 0 waiting for initial osdmap 2024-03-08T23:02:13.814 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:02:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:02:13.557+0000 7f5c748a2700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: osdmap e58: 9 total, 8 up, 9 in 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: from='osd.8 ' entity='osd.8' 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] boot 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: osdmap e59: 9 total, 9 up, 9 in 2024-03-08T23:02:14.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:14 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.436 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: osdmap e58: 9 total, 8 up, 9 in 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: from='osd.8 ' entity='osd.8' 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] boot 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: osdmap e59: 9 total, 9 up, 9 in 2024-03-08T23:02:14.437 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:02:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 232 MiB used, 715 GiB / 715 GiB avail 2024-03-08T23:02:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: osdmap e58: 9 total, 8 up, 9 in 2024-03-08T23:02:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:14.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: from='osd.8 ' entity='osd.8' 2024-03-08T23:02:14.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: osd.8 [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] boot 2024-03-08T23:02:14.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: osdmap e59: 9 total, 9 up, 9 in 2024-03-08T23:02:14.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:14 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:02:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:15 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:02:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:15 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:02:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:15 smithi062 ceph-mon[35761]: osdmap e60: 9 total, 9 up, 9 in 2024-03-08T23:02:15.430 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:15 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:02:15.430 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:15 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:02:15.430 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:15 smithi192 ceph-mon[36700]: osdmap e60: 9 total, 9 up, 9 in 2024-03-08T23:02:15.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:15 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:02:15.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:15 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:02:15.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:15 smithi046 ceph-mon[31859]: osdmap e60: 9 total, 9 up, 9 in 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr: stderr: 10+0 records in 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:10+0 records out 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.03345 s, 313 MB/s 2024-03-08T23:02:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:--> Zapping successful for: 2024-03-08T23:02:16.911 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi046:vg_nvme/lv_1 2024-03-08T23:02:17.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:17.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi192 2024-03-08T23:02:17.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:16 smithi192 ceph-mon[36700]: osdmap e61: 9 total, 9 up, 9 in 2024-03-08T23:02:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:17.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi192 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:16 smithi062 ceph-mon[35761]: osdmap e61: 9 total, 9 up, 9 in 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi192 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 5693M 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:17.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:16 smithi046 ceph-mon[31859]: osdmap e61: 9 total, 9 up, 9 in 2024-03-08T23:02:17.345 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:02:18.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:18 smithi046 ceph-mon[31859]: osdmap e62: 9 total, 9 up, 9 in 2024-03-08T23:02:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:18 smithi192 ceph-mon[36700]: osdmap e62: 9 total, 9 up, 9 in 2024-03-08T23:02:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:18 smithi062 ceph-mon[35761]: osdmap e62: 9 total, 9 up, 9 in 2024-03-08T23:02:19.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:19 smithi192 ceph-mon[36700]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:19.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:19 smithi192 ceph-mon[36700]: osdmap e63: 9 total, 9 up, 9 in 2024-03-08T23:02:19.374 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:19 smithi046 ceph-mon[31859]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:19.375 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:19 smithi046 ceph-mon[31859]: osdmap e63: 9 total, 9 up, 9 in 2024-03-08T23:02:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:19 smithi062 ceph-mon[35761]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:19 smithi062 ceph-mon[35761]: osdmap e63: 9 total, 9 up, 9 in 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 -- 172.21.15.46:0/752593448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341051d0 msgr2=0x7fb3341075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 --2- 172.21.15.46:0/752593448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341051d0 0x7fb3341075c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb320009990 tx=0x7fb320009ca0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 -- 172.21.15.46:0/752593448 shutdown_connections 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 --2- 172.21.15.46:0/752593448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 0x7fb334109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 --2- 172.21.15.46:0/752593448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341028a0 0x7fb334104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 --2- 172.21.15.46:0/752593448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341051d0 0x7fb3341075c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.546+0000 7fb338e00700 1 -- 172.21.15.46:0/752593448 >> 172.21.15.46:0/752593448 conn(0x7fb3340fc2b0 msgr2=0x7fb3340fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:02:19.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.547+0000 7fb338e00700 1 -- 172.21.15.46:0/752593448 shutdown_connections 2024-03-08T23:02:19.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.547+0000 7fb338e00700 1 -- 172.21.15.46:0/752593448 wait complete. 2024-03-08T23:02:19.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.548+0000 7fb338e00700 1 Processor -- start 2024-03-08T23:02:19.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.548+0000 7fb338e00700 1 -- start start 2024-03-08T23:02:19.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 0x7fb33419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 0x7fb3341a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410d9f0 con 0x7fb3341051d0 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410d870 con 0x7fb334107b00 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb338e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33410db70 con 0x7fb3341028a0 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb331400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 0x7fb33419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb331e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb332800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 0x7fb3341a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb331400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 0x7fb33419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53434/0 (socket says 172.21.15.46:53434) 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb331400700 1 -- 172.21.15.46:0/1120176530 learned_addr learned my addr 172.21.15.46:0/1120176530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.549+0000 7fb331e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:40040/0 (socket says 172.21.15.46:40040) 2024-03-08T23:02:19.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 -- 172.21.15.46:0/1120176530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 msgr2=0x7fb3341a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 0x7fb3341a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 -- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 msgr2=0x7fb33419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 0x7fb33419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 -- 172.21.15.46:0/1120176530 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3341a1e90 con 0x7fb3341028a0 2024-03-08T23:02:19.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb331e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb32400ea20 tx=0x7fb32400ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32400cbd0 con 0x7fb3341028a0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb32400ee50 con 0x7fb3341028a0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324018660 con 0x7fb3341028a0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3341a2180 con 0x7fb3341028a0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.550+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3341a2710 con 0x7fb3341028a0 2024-03-08T23:02:19.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.551+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334068aa0 con 0x7fb3341028a0 2024-03-08T23:02:19.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.554+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb324010430 con 0x7fb3341028a0 2024-03-08T23:02:19.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.555+0000 7fb32aa00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 0x7fb3180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:19.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.556+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5829+0+0 (secure 0 0 0) 0x7fb3240ce610 con 0x7fb3341028a0 2024-03-08T23:02:19.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.556+0000 7fb331400700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 0x7fb3180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:19.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.556+0000 7fb331400700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 0x7fb3180a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb32000b2e0 tx=0x7fb32000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:02:19.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.557+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb32409bfc0 con 0x7fb3341028a0 2024-03-08T23:02:19.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:19.796+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fb334062220 con 0x7fb3180a2910 2024-03-08T23:02:20.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:20.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:20.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:20.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:20.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:20.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:20.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:20.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:20.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:21 smithi192 ceph-mon[36700]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:21 smithi192 ceph-mon[36700]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:21 smithi062 ceph-mon[35761]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:21 smithi062 ceph-mon[35761]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:21 smithi046 ceph-mon[31859]: pgmap v223: 1 pgs: 1 remapped+peering; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:21 smithi046 ceph-mon[31859]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi046:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:23.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:23 smithi192 ceph-mon[36700]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:02:23.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:23 smithi062 ceph-mon[35761]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:02:23.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:23 smithi046 ceph-mon[31859]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 263 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:02:24.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:02:24.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/861030104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.304 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]': finished 2024-03-08T23:02:24.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: osdmap e64: 10 total, 9 up, 10 in 2024-03-08T23:02:24.305 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:24.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:02:24.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/861030104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]': finished 2024-03-08T23:02:24.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: osdmap e64: 10 total, 9 up, 10 in 2024-03-08T23:02:24.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/861030104' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]: dispatch 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73f96e3-f0af-44cf-9faa-76a4a469b378"}]': finished 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: osdmap e64: 10 total, 9 up, 10 in 2024-03-08T23:02:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:25 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2544968242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:25 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2544968242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:25.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:25 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2544968242' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:26.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:26 smithi192 ceph-mon[36700]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:02:26.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:26 smithi062 ceph-mon[35761]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:02:26.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:26 smithi046 ceph-mon[31859]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-08T23:02:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:28 smithi046 ceph-mon[31859]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:02:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:28 smithi192 ceph-mon[36700]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:02:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:28 smithi062 ceph-mon[35761]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-08T23:02:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:30 smithi046 ceph-mon[31859]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-08T23:02:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:30 smithi192 ceph-mon[36700]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-08T23:02:30.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:30 smithi062 ceph-mon[35761]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-08T23:02:32.459 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:32 smithi062 ceph-mon[35761]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:32.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:32 smithi046 ceph-mon[31859]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:32.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:32 smithi192 ceph-mon[36700]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-08T23:02:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:33 smithi046 ceph-mon[31859]: Deploying daemon osd.9 on smithi046 2024-03-08T23:02:33.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-08T23:02:33.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:33.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:33 smithi192 ceph-mon[36700]: Deploying daemon osd.9 on smithi046 2024-03-08T23:02:33.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-08T23:02:33.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:33.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:33 smithi062 ceph-mon[35761]: Deploying daemon osd.9 on smithi046 2024-03-08T23:02:34.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:34 smithi046 ceph-mon[31859]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:34 smithi192 ceph-mon[36700]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:34.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:34 smithi062 ceph-mon[35761]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:36.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:36 smithi046 ceph-mon[31859]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:36.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:36 smithi192 ceph-mon[36700]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:36.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:36 smithi062 ceph-mon[35761]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:38.453 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:38 smithi046 ceph-mon[31859]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:38.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:38 smithi192 ceph-mon[36700]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:38 smithi062 ceph-mon[35761]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:40.430 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:40 smithi046 ceph-mon[31859]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:40.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:40 smithi192 ceph-mon[36700]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:40 smithi062 ceph-mon[35761]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:42 smithi062 ceph-mon[35761]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:42.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:42 smithi046 ceph-mon[31859]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:42.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:42.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:42.646 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:42 smithi192 ceph-mon[36700]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:44 smithi062 ceph-mon[35761]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:44 smithi046 ceph-mon[31859]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:44 smithi192 ceph-mon[36700]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:46.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.055+0000 7fb32aa00700 1 -- 172.21.15.46:0/1120176530 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb334062220 con 0x7fb3180a2910 2024-03-08T23:02:46.061 INFO:teuthology.orchestra.run.smithi046.stdout:Created osd(s) 9 on host 'smithi046' 2024-03-08T23:02:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 msgr2=0x7fb3180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 0x7fb3180a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb32000b2e0 tx=0x7fb32000b880 comp rx=0 tx=0).stop 2024-03-08T23:02:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 msgr2=0x7fb33419a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb32400ea20 tx=0x7fb32400ed30 comp rx=0 tx=0).stop 2024-03-08T23:02:46.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 shutdown_connections 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3180a2910 0x7fb3180a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb334107b00 0x7fb3341a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3341028a0 0x7fb33419a230 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.058+0000 7fb338e00700 1 --2- 172.21.15.46:0/1120176530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3341051d0 0x7fb33419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.059+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 >> 172.21.15.46:0/1120176530 conn(0x7fb3340fc2b0 msgr2=0x7fb334101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.059+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 shutdown_connections 2024-03-08T23:02:46.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:02:46.059+0000 7fb338e00700 1 -- 172.21.15.46:0/1120176530 wait complete. 2024-03-08T23:02:46.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:46.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:46 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:46 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-08T23:02:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:46 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:46.894 DEBUG:teuthology.orchestra.run.smithi046:osd.9> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.9.service 2024-03-08T23:02:46.902 INFO:tasks.cephadm:Deploying osd.10 on smithi062 with /dev/vg_nvme/lv_1... 2024-03-08T23:02:46.902 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-08T23:02:47.162 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:02:47.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:47 smithi062 ceph-mon[35761]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-08T23:02:47.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:47 smithi062 ceph-mon[35761]: osdmap e65: 10 total, 9 up, 10 in 2024-03-08T23:02:47.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:47 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:47.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:47 smithi062 ceph-mon[35761]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.285 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:47 smithi062 ceph-mon[35761]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-mon[31859]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-08T23:02:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-mon[31859]: osdmap e65: 10 total, 9 up, 10 in 2024-03-08T23:02:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-mon[31859]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-mon[31859]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:47 smithi192 ceph-mon[36700]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-08T23:02:47.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:47 smithi192 ceph-mon[36700]: osdmap e65: 10 total, 9 up, 10 in 2024-03-08T23:02:47.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:47 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:47.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:47 smithi192 ceph-mon[36700]: from='osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:47 smithi192 ceph-mon[36700]: from='osd.9 ' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]: dispatch 2024-03-08T23:02:47.834 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:02:47.789+0000 7fe45e200700 -1 osd.9 0 waiting for initial osdmap 2024-03-08T23:02:47.835 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:02:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:02:47.828+0000 7fe454000700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: osdmap e66: 10 total, 9 up, 10 in 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:48.434 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:48 smithi062 ceph-mon[35761]: from='osd.9 ' entity='osd.9' 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: osdmap e66: 10 total, 9 up, 10 in 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:48.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:48 smithi046 ceph-mon[31859]: from='osd.9 ' entity='osd.9' 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi046", "root=default"]}]': finished 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: osdmap e66: 10 total, 9 up, 10 in 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 264 MiB used, 804 GiB / 805 GiB avail 2024-03-08T23:02:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:48 smithi192 ceph-mon[36700]: from='osd.9 ' entity='osd.9' 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111622 s, 939 MB/s 2024-03-08T23:02:49.217 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] boot 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: osdmap e67: 10 total, 10 up, 10 in 2024-03-08T23:02:49.354 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:49 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] boot 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: osdmap e67: 10 total, 10 up, 10 in 2024-03-08T23:02:49.526 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:49 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: osd.9 [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] boot 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: osdmap e67: 10 total, 10 up, 10 in 2024-03-08T23:02:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:49 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:02:49.853 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi062:vg_nvme/lv_1 2024-03-08T23:02:50.106 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:02:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:50 smithi192 ceph-mon[36700]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:50 smithi192 ceph-mon[36700]: osdmap e68: 10 total, 10 up, 10 in 2024-03-08T23:02:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:50 smithi062 ceph-mon[35761]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:50 smithi062 ceph-mon[35761]: osdmap e68: 10 total, 10 up, 10 in 2024-03-08T23:02:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:50 smithi046 ceph-mon[31859]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:50 smithi046 ceph-mon[31859]: osdmap e68: 10 total, 10 up, 10 in 2024-03-08T23:02:51.791 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- 172.21.15.62:0/1213007853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 msgr2=0x7fbb38104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 --2- 172.21.15.62:0/1213007853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb38104cf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fbb28009990 tx=0x7fbb28009ca0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- 172.21.15.62:0/1213007853 shutdown_connections 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 --2- 172.21.15.62:0/1213007853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 0x7fbb38109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 --2- 172.21.15.62:0/1213007853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb38104cf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 --2- 172.21.15.62:0/1213007853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38105230 0x7fbb38107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- 172.21.15.62:0/1213007853 >> 172.21.15.62:0/1213007853 conn(0x7fbb380fc2b0 msgr2=0x7fbb380fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- 172.21.15.62:0/1213007853 shutdown_connections 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- 172.21.15.62:0/1213007853 wait complete. 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 Processor -- start 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.786+0000 7fbb3e983700 1 -- start start 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:51.792 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38105230 0x7fbb3819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 0x7fbb381a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3810da80 con 0x7fbb38105230 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3810d900 con 0x7fbb38102900 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3e983700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb3810dc00 con 0x7fbb38107b60 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.62:49712/0 (socket says 172.21.15.62:49712) 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 -- 172.21.15.62:0/171712492 learned_addr learned my addr 172.21.15.62:0/171712492 (peer_addr_for_me v2:172.21.15.62:0/0) 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb3cf20700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 0x7fbb381a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 -- 172.21.15.62:0/171712492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 msgr2=0x7fbb381a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 0x7fbb381a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 -- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38105230 msgr2=0x7fbb3819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38105230 0x7fbb3819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:02:51.793 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 -- 172.21.15.62:0/171712492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb381a1f00 con 0x7fbb38102900 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb37fff700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098e0 tx=0x7fbb28009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.787+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb28027070 con 0x7fbb38102900 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.788+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb381a2190 con 0x7fbb38102900 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.788+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb28004150 con 0x7fbb38102900 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.788+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2800f460 con 0x7fbb38102900 2024-03-08T23:02:51.794 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.788+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb381a2670 con 0x7fbb38102900 2024-03-08T23:02:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.789+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb38068aa0 con 0x7fbb38102900 2024-03-08T23:02:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.790+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb2802a070 con 0x7fbb38102900 2024-03-08T23:02:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.791+0000 7fbb357fa700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:02:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.791+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fbb280cf1c0 con 0x7fbb38102900 2024-03-08T23:02:51.795 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.792+0000 7fbb377fe700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:02:51.796 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.793+0000 7fbb377fe700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 0x7fbb200a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0097b0 tx=0x7fbb2c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:02:51.798 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:51.795+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbb28013070 con 0x7fbb38102900 2024-03-08T23:02:52.016 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:02:52.012+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb38062220 con 0x7fbb200a2890 2024-03-08T23:02:52.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:52.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:52.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:52.998 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:52.999 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:53.000 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:53.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:53.066 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='client.24368 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:53.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:53.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:02:53.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:02:53.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:02:53.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:02:54.106 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:53 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi046 2024-03-08T23:02:54.106 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:53 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-08T23:02:54.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:53 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi046 2024-03-08T23:02:54.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:53 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-08T23:02:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:53 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi046 2024-03-08T23:02:54.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:53 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 4273M 2024-03-08T23:02:55.108 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:55.109 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/825067257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.109 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.109 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]': finished 2024-03-08T23:02:55.109 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: osdmap e69: 11 total, 10 up, 11 in 2024-03-08T23:02:55.109 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:54 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/825067257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]': finished 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: osdmap e69: 11 total, 10 up, 11 in 2024-03-08T23:02:55.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:54 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/825067257' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]: dispatch 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dce7acee-a4ac-4a8c-952e-a210943385e5"}]': finished 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: osdmap e69: 11 total, 10 up, 11 in 2024-03-08T23:02:55.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:54 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:02:56.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:55 smithi062 ceph-mon[35761]: from='client.? 172.21.15.62:0/1578506959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:56.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:55 smithi046 ceph-mon[31859]: from='client.? 172.21.15.62:0/1578506959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:56.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:55 smithi192 ceph-mon[36700]: from='client.? 172.21.15.62:0/1578506959' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:02:57.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:56 smithi062 ceph-mon[35761]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:56 smithi046 ceph-mon[31859]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:57.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:56 smithi192 ceph-mon[36700]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:59.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:02:58 smithi062 ceph-mon[35761]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:02:58 smithi046 ceph-mon[31859]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:02:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:02:58 smithi192 ceph-mon[36700]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:00.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:00 smithi062 ceph-mon[35761]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:01.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:00 smithi046 ceph-mon[31859]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:00 smithi192 ceph-mon[36700]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:03.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:02 smithi062 ceph-mon[35761]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:03.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:02 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-08T23:03:03.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:02 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:02 smithi046 ceph-mon[31859]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:02 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-08T23:03:03.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:02 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:02 smithi192 ceph-mon[36700]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:02 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-08T23:03:03.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:02 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:04.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:03 smithi062 ceph-mon[35761]: Deploying daemon osd.10 on smithi062 2024-03-08T23:03:04.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:03 smithi046 ceph-mon[31859]: Deploying daemon osd.10 on smithi062 2024-03-08T23:03:04.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:03 smithi192 ceph-mon[36700]: Deploying daemon osd.10 on smithi062 2024-03-08T23:03:05.091 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:04 smithi062 ceph-mon[35761]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:05.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:04 smithi046 ceph-mon[31859]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:05.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:04 smithi192 ceph-mon[36700]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:07.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:06 smithi062 ceph-mon[35761]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:07.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:06 smithi046 ceph-mon[31859]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:07.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:06 smithi192 ceph-mon[36700]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:08 smithi062 ceph-mon[35761]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:08 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:08 smithi046 ceph-mon[31859]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:08 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:09.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:08 smithi192 ceph-mon[36700]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:09.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:09.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:08 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:11.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:10 smithi062 ceph-mon[35761]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:10 smithi046 ceph-mon[31859]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:11.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:10 smithi192 ceph-mon[36700]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:11.700 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.696+0000 7fbb357fa700 1 -- 172.21.15.62:0/171712492 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fbb38062220 con 0x7fbb200a2890 2024-03-08T23:03:11.701 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 10 on host 'smithi062' 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 msgr2=0x7fbb200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 0x7fbb200a4d50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbb2c0097b0 tx=0x7fbb2c006d20 comp rx=0 tx=0).stop 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 msgr2=0x7fbb3819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbb280098e0 tx=0x7fbb28009ee0 comp rx=0 tx=0).stop 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 shutdown_connections 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbb200a2890 0x7fbb200a4d50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:11.703 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb38107b60 0x7fbb381a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb38102900 0x7fbb3819a2a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 --2- 172.21.15.62:0/171712492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb38105230 0x7fbb3819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 >> 172.21.15.62:0/171712492 conn(0x7fbb380fc2b0 msgr2=0x7fbb38101fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 shutdown_connections 2024-03-08T23:03:11.704 INFO:teuthology.orchestra.run.smithi062.stderr:2024-03-08T23:03:11.698+0000 7fbb3e983700 1 -- 172.21.15.62:0/171712492 wait complete. 2024-03-08T23:03:12.756 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:12 smithi062 ceph-mon[35761]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:12.756 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:12.756 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:12.756 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:12 smithi062 ceph-mon[35761]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:12.756 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:12 smithi062 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:12.784 DEBUG:teuthology.orchestra.run.smithi062:osd.10> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.10.service 2024-03-08T23:03:12.786 INFO:tasks.cephadm:Deploying osd.11 on smithi192 with /dev/vg_nvme/lv_1... 2024-03-08T23:03:12.787 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-08T23:03:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:12 smithi046 ceph-mon[31859]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:12 smithi046 ceph-mon[31859]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:12.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:12 smithi046 ceph-mon[31859]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:13.033 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:03:13.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:12 smithi192 ceph-mon[36700]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:13.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:13.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:13.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:12 smithi192 ceph-mon[36700]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:13.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:12 smithi192 ceph-mon[36700]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-08T23:03:13.977 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-08T23:03:13.977 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:13.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: osdmap e70: 11 total, 10 up, 11 in 2024-03-08T23:03:13.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:13.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:13.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:13.978 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:13.978 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:03:13.888+0000 7f4bb1424700 -1 osd.10 0 waiting for initial osdmap 2024-03-08T23:03:13.978 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:03:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:03:13.901+0000 7f4bad217700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: osdmap e70: 11 total, 10 up, 11 in 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:14.156 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:14.157 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:13 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:14.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-08T23:03:14.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:14.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: osdmap e70: 11 total, 10 up, 11 in 2024-03-08T23:03:14.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:14.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-03-08T23:03:14.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:14.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:15.048 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-08T23:03:15.048 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-08T23:03:15.048 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-03-08T23:03:15.048 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-03-08T23:03:15.049 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112282 s, 934 MB/s 2024-03-08T23:03:15.049 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: osdmap e71: 11 total, 10 up, 11 in 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] boot 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: osdmap e72: 11 total, 11 up, 11 in 2024-03-08T23:03:15.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:14 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: osdmap e71: 11 total, 10 up, 11 in 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] boot 2024-03-08T23:03:15.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: osdmap e72: 11 total, 11 up, 11 in 2024-03-08T23:03:15.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:14 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 295 MiB used, 894 GiB / 894 GiB avail 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: osdmap e71: 11 total, 10 up, 11 in 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: osd.10 [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] boot 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: osdmap e72: 11 total, 11 up, 11 in 2024-03-08T23:03:15.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:14 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:03:15.747 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-03-08T23:03:15.991 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:03:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:16 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:03:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:16 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:03:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:16 smithi046 ceph-mon[31859]: osdmap e73: 11 total, 11 up, 11 in 2024-03-08T23:03:16.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:16 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:03:16.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:16 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:03:16.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:16 smithi192 ceph-mon[36700]: osdmap e73: 11 total, 11 up, 11 in 2024-03-08T23:03:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:16 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:03:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:16 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:03:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:16 smithi062 ceph-mon[35761]: osdmap e73: 11 total, 11 up, 11 in 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: osdmap e74: 11 total, 11 up, 11 in 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.186 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.187 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:17.187 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.187 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.187 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:17 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: osdmap e74: 11 total, 11 up, 11 in 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:17 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: pgmap v261: 1 pgs: 1 remapped+peering; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: osdmap e74: 11 total, 11 up, 11 in 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:17.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:17.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:17 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 -- 172.21.15.192:0/1717781361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 msgr2=0x7f5510102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 --2- 172.21.15.192:0/1717781361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f5510102410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5500009990 tx=0x7f5500009ca0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 -- 172.21.15.192:0/1717781361 shutdown_connections 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 --2- 172.21.15.192:0/1717781361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5510102950 0x7f5510109200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 --2- 172.21.15.192:0/1717781361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510100db0 0x7f55101011d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 --2- 172.21.15.192:0/1717781361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f5510102410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 -- 172.21.15.192:0/1717781361 >> 172.21.15.192:0/1717781361 conn(0x7f55100fc370 msgr2=0x7f55100fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:17.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 -- 172.21.15.192:0/1717781361 shutdown_connections 2024-03-08T23:03:17.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.702+0000 7f5515bc3700 1 -- 172.21.15.192:0/1717781361 wait complete. 2024-03-08T23:03:17.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 Processor -- start 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 -- start start 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5510100db0 0x7f5510197090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 0x7f551019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551010ca20 con 0x7f5510100db0 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551010c8a0 con 0x7f5510102950 2024-03-08T23:03:17.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f5515bc3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551010cba0 con 0x7f5510101fb0 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f550effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f550effd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42882/0 (socket says 172.21.15.192:42882) 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.703+0000 7f550effd700 1 -- 172.21.15.192:0/489233378 learned_addr learned my addr 172.21.15.192:0/489233378 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550ffff700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 0x7f551019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 -- 172.21.15.192:0/489233378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 msgr2=0x7f551019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 0x7f551019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 -- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5510100db0 msgr2=0x7f5510197090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5510100db0 0x7f5510197090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 -- 172.21.15.192:0/489233378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551019ecf0 con 0x7f5510101fb0 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550ffff700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 0x7f551019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550effd700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f55000047b0 tx=0x7f5500003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:17.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500027070 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551019ef80 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5500022470 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000f6c0 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.704+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551019f400 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.705+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5510068aa0 con 0x7f5510101fb0 2024-03-08T23:03:17.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.706+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f550002a070 con 0x7f5510101fb0 2024-03-08T23:03:17.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.706+0000 7f550cff9700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 0x7f54f80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:17.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.707+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f55000cf520 con 0x7f5510101fb0 2024-03-08T23:03:17.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.707+0000 7f550f7fe700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 0x7f54f80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:17.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.708+0000 7f550f7fe700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 0x7f54f80a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5510101e10 tx=0x7f5504006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:17.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.711+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f550005e260 con 0x7f5510101fb0 2024-03-08T23:03:17.918 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:17.914+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5510062220 con 0x7f54f80a2890 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi062 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 4270M 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: osdmap e75: 11 total, 11 up, 11 in 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:03:18.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi062 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 4270M 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: osdmap e75: 11 total, 11 up, 11 in 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:03:18.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:18.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi062 2024-03-08T23:03:18.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 4270M 2024-03-08T23:03:18.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: osdmap e75: 11 total, 11 up, 11 in 2024-03-08T23:03:18.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:03:18.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:03:18.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:19.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:19 smithi192 ceph-mon[36700]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:19.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:19 smithi192 ceph-mon[36700]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:19.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:19 smithi192 ceph-mon[36700]: osdmap e76: 11 total, 11 up, 11 in 2024-03-08T23:03:19.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:19 smithi062 ceph-mon[35761]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:19 smithi062 ceph-mon[35761]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:19 smithi062 ceph-mon[35761]: osdmap e76: 11 total, 11 up, 11 in 2024-03-08T23:03:19.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:19 smithi046 ceph-mon[31859]: pgmap v263: 1 pgs: 1 remapped+peering; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:19.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:19 smithi046 ceph-mon[31859]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:19.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:19 smithi046 ceph-mon[31859]: osdmap e76: 11 total, 11 up, 11 in 2024-03-08T23:03:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-08T23:03:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/2931687807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]': finished 2024-03-08T23:03:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: osdmap e77: 12 total, 11 up, 12 in 2024-03-08T23:03:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/2931687807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]': finished 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: osdmap e77: 12 total, 11 up, 12 in 2024-03-08T23:03:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/2931687807' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]: dispatch 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a7095bc-14f0-4fe3-b8d4-717280a11bd2"}]': finished 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: osdmap e77: 12 total, 11 up, 12 in 2024-03-08T23:03:21.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:22 smithi192 ceph-mon[36700]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-08T23:03:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:22 smithi192 ceph-mon[36700]: from='client.? 172.21.15.192:0/3967017738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:03:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:22 smithi062 ceph-mon[35761]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-08T23:03:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:22 smithi062 ceph-mon[35761]: from='client.? 172.21.15.192:0/3967017738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:03:22.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:22 smithi046 ceph-mon[31859]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-08T23:03:22.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:22 smithi046 ceph-mon[31859]: from='client.? 172.21.15.192:0/3967017738' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:03:24.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:24 smithi046 ceph-mon[31859]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:03:24.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:24 smithi192 ceph-mon[36700]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:03:24.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:24 smithi062 ceph-mon[35761]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-08T23:03:26.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:26 smithi046 ceph-mon[31859]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-08T23:03:26.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:26 smithi192 ceph-mon[36700]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-08T23:03:26.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:26 smithi062 ceph-mon[35761]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-08T23:03:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:28 smithi046 ceph-mon[31859]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-08T23:03:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-08T23:03:28.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:28 smithi192 ceph-mon[36700]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-08T23:03:28.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-08T23:03:28.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:28 smithi062 ceph-mon[35761]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-03-08T23:03:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-08T23:03:28.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:29.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:29 smithi192 ceph-mon[36700]: Deploying daemon osd.11 on smithi192 2024-03-08T23:03:29.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:29 smithi062 ceph-mon[35761]: Deploying daemon osd.11 on smithi192 2024-03-08T23:03:29.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:29 smithi046 ceph-mon[31859]: Deploying daemon osd.11 on smithi192 2024-03-08T23:03:30.476 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:30 smithi192 ceph-mon[36700]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:30 smithi062 ceph-mon[35761]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:30 smithi046 ceph-mon[31859]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:32.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:32 smithi192 ceph-mon[36700]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:32.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:32 smithi062 ceph-mon[35761]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:32.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:32 smithi046 ceph-mon[31859]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:34 smithi192 ceph-mon[36700]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:34 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:34 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:34.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:34 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:34.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:34 smithi062 ceph-mon[35761]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:34.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:34 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:34.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:34 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:34.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:34 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:34 smithi046 ceph-mon[31859]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:34 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:34 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:34.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:34 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:36.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:36 smithi192 ceph-mon[36700]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:36.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:36 smithi062 ceph-mon[35761]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:36.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:36 smithi046 ceph-mon[31859]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:37.959 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.954+0000 7f550cff9700 1 -- 172.21.15.192:0/489233378 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f5510062220 con 0x7f54f80a2890 2024-03-08T23:03:37.959 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 11 on host 'smithi192' 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 msgr2=0x7f54f80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 0x7f54f80a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5510101e10 tx=0x7f5504006d20 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 msgr2=0x7f55101975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f55000047b0 tx=0x7f5500003680 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 shutdown_connections 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54f80a2890 0x7f54f80a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5510102950 0x7f551019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5510100db0 0x7f5510197090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 --2- 172.21.15.192:0/489233378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5510101fb0 0x7f55101975d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 >> 172.21.15.192:0/489233378 conn(0x7f55100fc370 msgr2=0x7f55101053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:37.961 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 shutdown_connections 2024-03-08T23:03:37.962 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:37.957+0000 7f5515bc3700 1 -- 172.21.15.192:0/489233378 wait complete. 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:38.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:38.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:38.561 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.589 DEBUG:teuthology.orchestra.run.smithi192:osd.11> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.11.service 2024-03-08T23:03:38.591 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-08T23:03:38.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-08T23:03:38.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:38.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:38.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:39.078 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:03:39.494 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:39 smithi046 ceph-mon[31859]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-08T23:03:39.495 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:39 smithi046 ceph-mon[31859]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:39.495 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:39 smithi046 ceph-mon[31859]: osdmap e78: 12 total, 11 up, 12 in 2024-03-08T23:03:39.495 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:39.495 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:39 smithi046 ceph-mon[31859]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-mon[36700]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-08T23:03:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-mon[36700]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-mon[36700]: osdmap e78: 12 total, 11 up, 12 in 2024-03-08T23:03:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-mon[36700]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:39.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:03:39.257+0000 7f958a380700 -1 osd.11 0 waiting for initial osdmap 2024-03-08T23:03:39.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:03:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:03:39.271+0000 7f9586173700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:03:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:39 smithi062 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-08T23:03:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:39 smithi062 ceph-mon[35761]: from='osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:39 smithi062 ceph-mon[35761]: osdmap e78: 12 total, 11 up, 12 in 2024-03-08T23:03:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:39 smithi062 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-03-08T23:03:40.311 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:40 smithi046 ceph-mon[31859]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:40.311 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:40 smithi046 ceph-mon[31859]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:03:40.311 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:40 smithi046 ceph-mon[31859]: osdmap e79: 12 total, 11 up, 12 in 2024-03-08T23:03:40.312 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:40.312 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:40.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:40 smithi192 ceph-mon[36700]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:40.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:40 smithi192 ceph-mon[36700]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:03:40.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:40 smithi192 ceph-mon[36700]: osdmap e79: 12 total, 11 up, 12 in 2024-03-08T23:03:40.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:40.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:40 smithi062 ceph-mon[35761]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:40 smithi062 ceph-mon[35761]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-03-08T23:03:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:40 smithi062 ceph-mon[35761]: osdmap e79: 12 total, 11 up, 12 in 2024-03-08T23:03:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:40.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 -- 172.21.15.46:0/718188474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd81051d0 msgr2=0x7fdcd81075c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 --2- 172.21.15.46:0/718188474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd81051d0 0x7fdcd81075c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fdcc4009990 tx=0x7fdcc4009ca0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 -- 172.21.15.46:0/718188474 shutdown_connections 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 --2- 172.21.15.46:0/718188474 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd8107b00 0x7fdcd8109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 --2- 172.21.15.46:0/718188474 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 0x7fdcd8104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 --2- 172.21.15.46:0/718188474 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd81051d0 0x7fdcd81075c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.245+0000 7fdcdec00700 1 -- 172.21.15.46:0/718188474 >> 172.21.15.46:0/718188474 conn(0x7fdcd80fc2b0 msgr2=0x7fdcd80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.246+0000 7fdcdec00700 1 -- 172.21.15.46:0/718188474 shutdown_connections 2024-03-08T23:03:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.246+0000 7fdcdec00700 1 -- 172.21.15.46:0/718188474 wait complete. 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 Processor -- start 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 -- start start 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 0x7fdcd819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd8107b00 0x7fdcd81a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd810da10 con 0x7fdcd8107b00 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd810d890 con 0x7fdcd81028a0 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdec00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd810db90 con 0x7fdcd81051d0 2024-03-08T23:03:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcd7400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcdcc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd8107b00 0x7fdcd81a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcd7400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43014/0 (socket says 172.21.15.46:43014) 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcd7e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 0x7fdcd819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.247+0000 7fdcd7400700 1 -- 172.21.15.46:0/2668748716 learned_addr learned my addr 172.21.15.46:0/2668748716 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 -- 172.21.15.46:0/2668748716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 msgr2=0x7fdcd819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 0x7fdcd819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 -- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd8107b00 msgr2=0x7fdcd81a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd8107b00 0x7fdcd81a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 -- 172.21.15.46:0/2668748716 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd81a1eb0 con 0x7fdcd81051d0 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.248+0000 7fdcd7400700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098e0 tx=0x7fdcc400bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.249+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc4027070 con 0x7fdcd81051d0 2024-03-08T23:03:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.249+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcc4022470 con 0x7fdcd81051d0 2024-03-08T23:03:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.249+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd81a2140 con 0x7fdcd81051d0 2024-03-08T23:03:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.249+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc400f6c0 con 0x7fdcd81051d0 2024-03-08T23:03:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.249+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd81a2590 con 0x7fdcd81051d0 2024-03-08T23:03:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.250+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8068aa0 con 0x7fdcd81051d0 2024-03-08T23:03:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.252+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdcc402a070 con 0x7fdcd81051d0 2024-03-08T23:03:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.253+0000 7fdcd4c00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.253+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdcc40cff80 con 0x7fdcd81051d0 2024-03-08T23:03:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.254+0000 7fdcd7e00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.254+0000 7fdcd7e00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8007c60 tx=0x7fdcc80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.255+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdcc4031020 con 0x7fdcd81051d0 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: purged_snaps scrub starts 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: purged_snaps scrub ok 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] boot 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: osdmap e80: 12 total, 12 up, 12 in 2024-03-08T23:03:41.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.487+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fdcd80629f0 con 0x7fdcd81051d0 2024-03-08T23:03:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.488+0000 7fdcd4c00700 1 -- 172.21.15.46:0/2668748716 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fdcc4013070 con 0x7fdcd81051d0 2024-03-08T23:03:41.491 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.490+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 msgr2=0x7fdcbc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.490+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8007c60 tx=0x7fdcc80073d0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.490+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 msgr2=0x7fdcd819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.490+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdcc40098e0 tx=0x7fdcc400bbb0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 shutdown_connections 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdcbc0a2890 0x7fdcbc0a4d50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdcd8107b00 0x7fdcd81a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdcd81028a0 0x7fdcd819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 --2- 172.21.15.46:0/2668748716 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdcd81051d0 0x7fdcd819a790 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 >> 172.21.15.46:0/2668748716 conn(0x7fdcd80fc2b0 msgr2=0x7fdcd8102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 shutdown_connections 2024-03-08T23:03:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:41.491+0000 7fdcdec00700 1 -- 172.21.15.46:0/2668748716 wait complete. 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: purged_snaps scrub starts 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: purged_snaps scrub ok 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] boot 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: osdmap e80: 12 total, 12 up, 12 in 2024-03-08T23:03:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: purged_snaps scrub starts 2024-03-08T23:03:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: purged_snaps scrub ok 2024-03-08T23:03:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: osd.11 [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] boot 2024-03-08T23:03:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: osdmap e80: 12 total, 12 up, 12 in 2024-03-08T23:03:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:03:42.261 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1709939020,"num_in_osds":12,"osd_in_since":1709939000,"num_remapped_pgs":0} 2024-03-08T23:03:42.262 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: osdmap e81: 12 total, 12 up, 12 in 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2668748716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: Detected new or changed devices on smithi192 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-08T23:03:42.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:42.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:42.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:42 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: osdmap e81: 12 total, 12 up, 12 in 2024-03-08T23:03:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2668748716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-08T23:03:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: Detected new or changed devices on smithi192 2024-03-08T23:03:42.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:42.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:42 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: osdmap e81: 12 total, 12 up, 12 in 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2668748716' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: Detected new or changed devices on smithi192 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:03:42.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 4270M 2024-03-08T23:03:42.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:42.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:42.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:42 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:42.701 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:03:44.487 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:44 smithi046 ceph-mon[31859]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:44.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:44 smithi192 ceph-mon[36700]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:44 smithi062 ceph-mon[35761]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 -- 172.21.15.46:0/1391423585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c1027f0 msgr2=0x7f9b1c10ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 --2- 172.21.15.46:0/1391423585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c10ec30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9b1400b0d0 tx=0x7f9b1400b3e0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 -- 172.21.15.46:0/1391423585 shutdown_connections 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 --2- 172.21.15.46:0/1391423585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c10ec30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 --2- 172.21.15.46:0/1391423585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c107e10 0x7f9b1c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 --2- 172.21.15.46:0/1391423585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c101e30 0x7f9b1c1022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 -- 172.21.15.46:0/1391423585 >> 172.21.15.46:0/1391423585 conn(0x7f9b1c078f10 msgr2=0x7f9b1c079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 -- 172.21.15.46:0/1391423585 shutdown_connections 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.783+0000 7f9b23e00700 1 -- 172.21.15.46:0/1391423585 wait complete. 2024-03-08T23:03:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.784+0000 7f9b23e00700 1 Processor -- start 2024-03-08T23:03:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.784+0000 7f9b23e00700 1 -- start start 2024-03-08T23:03:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c107e10 0x7f9b1c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c112100 con 0x7f9b1c101e30 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c111f80 con 0x7f9b1c107e10 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b23e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1c112280 con 0x7f9b1c1027f0 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b21400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b21400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34808/0 (socket says 172.21.15.46:34808) 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b21400700 1 -- 172.21.15.46:0/3597504760 learned_addr learned my addr 172.21.15.46:0/3597504760 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.785+0000 7f9b20a00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b21e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c107e10 0x7f9b1c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 -- 172.21.15.46:0/3597504760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c107e10 msgr2=0x7f9b1c19ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c107e10 0x7f9b1c19ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 -- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 msgr2=0x7f9b1c19a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:44.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 -- 172.21.15.46:0/3597504760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1c19f6b0 con 0x7f9b1c1027f0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b21400700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.786+0000 7f9b20a00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c19abe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800e9f0 tx=0x7f9b0800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.787+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b0800c9c0 con 0x7f9b1c1027f0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.787+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1c19f9a0 con 0x7f9b1c1027f0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.787+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9b0800cb60 con 0x7f9b1c1027f0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.787+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08018680 con 0x7f9b1c1027f0 2024-03-08T23:03:44.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.787+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1c1a7220 con 0x7f9b1c1027f0 2024-03-08T23:03:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.788+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b1c068aa0 con 0x7f9b1c1027f0 2024-03-08T23:03:44.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.790+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9b08010430 con 0x7f9b1c1027f0 2024-03-08T23:03:44.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.791+0000 7f9b1a000700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 0x7f9b040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:44.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.791+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9b080ce790 con 0x7f9b1c1027f0 2024-03-08T23:03:44.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.792+0000 7f9b21400700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 0x7f9b040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:44.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.792+0000 7f9b21400700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 0x7f9b040a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9b10011590 tx=0x7f9b10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:44.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:44.795+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9b0809bc30 con 0x7f9b1c1027f0 2024-03-08T23:03:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.024+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b1c1a7420 con 0x7f9b1c1027f0 2024-03-08T23:03:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.026+0000 7f9b1a000700 1 -- 172.21.15.46:0/3597504760 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18432 (secure 0 0 0) 0x7f9b0809b530 con 0x7f9b1c1027f0 2024-03-08T23:03:45.030 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:45.030 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:03:41.272757+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 msgr2=0x7f9b040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 0x7f9b040a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9b10011590 tx=0x7f9b10009260 comp rx=0 tx=0).stop 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 msgr2=0x7f9b1c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c19abe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800e9f0 tx=0x7f9b0800ed00 comp rx=0 tx=0).stop 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 shutdown_connections 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9b040a2890 0x7f9b040a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9b1c107e10 0x7f9b1c19ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9b1c101e30 0x7f9b1c19a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 --2- 172.21.15.46:0/3597504760 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9b1c1027f0 0x7f9b1c19abe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.029+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 >> 172.21.15.46:0/3597504760 conn(0x7f9b1c078f10 msgr2=0x7f9b1c074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.030+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 shutdown_connections 2024-03-08T23:03:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:45.030+0000 7f9b23e00700 1 -- 172.21.15.46:0/3597504760 wait complete. 2024-03-08T23:03:45.387 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:45 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3597504760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:45.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:45 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3597504760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:45.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:45 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3597504760' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:45.725 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T23:03:45.725 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-08T23:03:46.202 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:03:46.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:46 smithi192 ceph-mon[36700]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:46.602 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:46 smithi046 ceph-mon[31859]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:46.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:46 smithi062 ceph-mon[35761]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.158+0000 7f51e4000700 1 -- 172.21.15.46:0/316361525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06af30 msgr2=0x7f51dc106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 --2- 172.21.15.46:0/316361525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06af30 0x7f51dc106a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0099c0 tx=0x7f51cc009cd0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 -- 172.21.15.46:0/316361525 shutdown_connections 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 --2- 172.21.15.46:0/316361525 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc107160 0x7f51dc10ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 --2- 172.21.15.46:0/316361525 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc06a580 0x7f51dc06a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 --2- 172.21.15.46:0/316361525 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06af30 0x7f51dc106a30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 -- 172.21.15.46:0/316361525 >> 172.21.15.46:0/316361525 conn(0x7f51dc076d30 msgr2=0x7f51dc077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 -- 172.21.15.46:0/316361525 shutdown_connections 2024-03-08T23:03:48.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.159+0000 7f51e4000700 1 -- 172.21.15.46:0/316361525 wait complete. 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.160+0000 7f51e4000700 1 Processor -- start 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.160+0000 7f51e4000700 1 -- start start 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 0x7f51dc1a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc107160 0x7f51dc19a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc112120 con 0x7f51dc06a580 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc111fa0 con 0x7f51dc06af30 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e4000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51dc1122a0 con 0x7f51dc107160 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e1600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e0c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 0x7f51dc1a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:48.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e1600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34830/0 (socket says 172.21.15.46:34830) 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e1600700 1 -- 172.21.15.46:0/3685250550 learned_addr learned my addr 172.21.15.46:0/3685250550 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e2000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc107160 0x7f51dc19a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e0c00700 1 -- 172.21.15.46:0/3685250550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc107160 msgr2=0x7f51dc19a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e0c00700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc107160 0x7f51dc19a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e0c00700 1 -- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 msgr2=0x7f51dc1a0580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.161+0000 7f51e0c00700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51e0c00700 1 -- 172.21.15.46:0/3685250550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc19af30 con 0x7f51dc06af30 2024-03-08T23:03:48.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51e1600700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51e0c00700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 0x7f51dc1a0ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0098e0 tx=0x7f51cc003e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc009e60 con 0x7f51dc06af30 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51dc19b1c0 con 0x7f51dc06af30 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51cc003680 con 0x7f51dc06af30 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc00fd60 con 0x7f51dc06af30 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.162+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51dc1a7500 con 0x7f51dc06af30 2024-03-08T23:03:48.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.163+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51dc103510 con 0x7f51dc06af30 2024-03-08T23:03:48.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.165+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f51cc00f460 con 0x7f51dc06af30 2024-03-08T23:03:48.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.167+0000 7f51da000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 0x7f51c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:48.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.167+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f51cc0cf490 con 0x7f51dc06af30 2024-03-08T23:03:48.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.167+0000 7f51e1600700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 0x7f51c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:48.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.168+0000 7f51e1600700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 0x7f51c40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f51d0011590 tx=0x7f51d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:48.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.170+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f51cc013070 con 0x7f51dc06af30 2024-03-08T23:03:48.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.394+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f51dc19bb80 con 0x7f51dc06af30 2024-03-08T23:03:48.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.395+0000 7f51da000700 1 -- 172.21.15.46:0/3685250550 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f51cc09c9c0 con 0x7f51dc06af30 2024-03-08T23:03:48.398 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 msgr2=0x7f51c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 0x7f51c40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f51d0011590 tx=0x7f51d0009260 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 msgr2=0x7f51dc1a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 0x7f51dc1a0ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f51cc0098e0 tx=0x7f51cc003e00 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 shutdown_connections 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f51c40a2890 0x7f51c40a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f51dc107160 0x7f51dc19a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f51dc06a580 0x7f51dc1a0580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 --2- 172.21.15.46:0/3685250550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f51dc06af30 0x7f51dc1a0ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:48.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 >> 172.21.15.46:0/3685250550 conn(0x7f51dc076d30 msgr2=0x7f51dc0fed50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 shutdown_connections 2024-03-08T23:03:48.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:48.398+0000 7f51e4000700 1 -- 172.21.15.46:0/3685250550 wait complete. 2024-03-08T23:03:48.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:48 smithi192 ceph-mon[36700]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:48.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:48 smithi062 ceph-mon[35761]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:48 smithi046 ceph-mon[31859]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.a on smithi046 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.d on smithi046 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.g on smithi046 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.j on smithi046 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.b on smithi062 2024-03-08T23:03:49.191 INFO:tasks.cephadm:Adding mds.e on smithi062 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.h on smithi062 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.k on smithi062 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.c on smithi192 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.f on smithi192 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.i on smithi192 2024-03-08T23:03:49.192 INFO:tasks.cephadm:Adding mds.l on smithi192 2024-03-08T23:03:49.192 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph orch apply mds all '12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l' 2024-03-08T23:03:49.442 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:03:49.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:49 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3685250550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:49.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:49 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3685250550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:49.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:49 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3685250550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:50.578 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:50 smithi192 ceph-mon[36700]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:50 smithi062 ceph-mon[35761]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:50 smithi046 ceph-mon[31859]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1002340641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0100d00 msgr2=0x7f6ca0101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1002340641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0100d00 0x7f6ca0101120 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009990 tx=0x7f6c90009ca0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1002340641 shutdown_connections 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1002340641 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca01028a0 0x7f6ca0109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1002340641 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0100d00 0x7f6ca0101120 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1002340641 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ca0101f00 0x7f6ca0102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1002340641 >> 172.21.15.192:0/1002340641 conn(0x7f6ca00fc2e0 msgr2=0x7f6ca00fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1002340641 shutdown_connections 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1002340641 wait complete. 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.091+0000 7f6ca4c32700 1 Processor -- start 2024-03-08T23:03:51.097 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 -- start start 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca0100d00 0x7f6ca006b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ca01028a0 0x7f6ca019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca010c960 con 0x7f6ca01028a0 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca010c7e0 con 0x7f6ca0100d00 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6ca4c32700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ca010cae0 con 0x7f6ca0101f00 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:49590/0 (socket says 172.21.15.192:49590) 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 -- 172.21.15.192:0/1460981642 learned_addr learned my addr 172.21.15.192:0/1460981642 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9e59c700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca0100d00 0x7f6ca006b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 -- 172.21.15.192:0/1460981642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca0100d00 msgr2=0x7f6ca006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca0100d00 0x7f6ca006b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 -- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ca01028a0 msgr2=0x7f6ca019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ca01028a0 0x7f6ca019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 -- 172.21.15.192:0/1460981642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca019fd10 con 0x7f6ca0101f00 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c9dd9b700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b530 tx=0x7f6c9400b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94015020 con 0x7f6ca0101f00 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.092+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c94011d10 con 0x7f6ca0101f00 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.093+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ca019ffa0 con 0x7f6ca0101f00 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.093+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9400f2e0 con 0x7f6ca0101f00 2024-03-08T23:03:51.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.093+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ca01a0530 con 0x7f6ca0101f00 2024-03-08T23:03:51.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.094+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ca0068aa0 con 0x7f6ca0101f00 2024-03-08T23:03:51.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.095+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6c94018020 con 0x7f6ca0101f00 2024-03-08T23:03:51.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.095+0000 7f6c8f7fe700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 0x7f6c880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:51.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.095+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6c940cddd0 con 0x7f6ca0101f00 2024-03-08T23:03:51.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.096+0000 7f6c9e59c700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 0x7f6c880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:51.101 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.096+0000 7f6c9e59c700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 0x7f6c880a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6c90003890 tx=0x7f6c90005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:51.103 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.099+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6c9401f020 con 0x7f6ca0101f00 2024-03-08T23:03:51.308 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.304+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ca0062420 con 0x7f6c880a2890 2024-03-08T23:03:51.325 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.321+0000 7f6c8f7fe700 1 -- 172.21.15.192:0/1460981642 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f6ca0062420 con 0x7f6c880a2890 2024-03-08T23:03:51.325 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mds.all update... 2024-03-08T23:03:51.327 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 msgr2=0x7f6c880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:51.327 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 0x7f6c880a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6c90003890 tx=0x7f6c90005fd0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.327 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 msgr2=0x7f6ca019ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:51.327 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400b530 tx=0x7f6c9400b840 comp rx=0 tx=0).stop 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 shutdown_connections 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c880a2890 0x7f6c880a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6ca01028a0 0x7f6ca019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6ca0100d00 0x7f6ca006b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 --2- 172.21.15.192:0/1460981642 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ca0101f00 0x7f6ca019ce00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 >> 172.21.15.192:0/1460981642 conn(0x7f6ca00fc2e0 msgr2=0x7f6ca01052f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 shutdown_connections 2024-03-08T23:03:51.328 INFO:teuthology.orchestra.run.smithi192.stderr:2024-03-08T23:03:51.323+0000 7f6ca4c32700 1 -- 172.21.15.192:0/1460981642 wait complete. 2024-03-08T23:03:52.028 DEBUG:teuthology.orchestra.run.smithi046:mds.a> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.a.service 2024-03-08T23:03:52.034 DEBUG:teuthology.orchestra.run.smithi046:mds.d> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.d.service 2024-03-08T23:03:52.080 DEBUG:teuthology.orchestra.run.smithi046:mds.g> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.g.service 2024-03-08T23:03:52.125 DEBUG:teuthology.orchestra.run.smithi046:mds.j> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.j.service 2024-03-08T23:03:52.170 DEBUG:teuthology.orchestra.run.smithi062:mds.b> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.b.service 2024-03-08T23:03:52.172 DEBUG:teuthology.orchestra.run.smithi062:mds.e> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.e.service 2024-03-08T23:03:52.215 DEBUG:teuthology.orchestra.run.smithi062:mds.h> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.h.service 2024-03-08T23:03:52.259 DEBUG:teuthology.orchestra.run.smithi062:mds.k> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.k.service 2024-03-08T23:03:52.302 DEBUG:teuthology.orchestra.run.smithi192:mds.c> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.c.service 2024-03-08T23:03:52.305 DEBUG:teuthology.orchestra.run.smithi192:mds.f> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.f.service 2024-03-08T23:03:52.348 DEBUG:teuthology.orchestra.run.smithi192:mds.i> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.i.service 2024-03-08T23:03:52.391 DEBUG:teuthology.orchestra.run.smithi192:mds.l> sudo journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.l.service 2024-03-08T23:03:52.434 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-08T23:03:52.434 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:03:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:52 smithi046 ceph-mon[31859]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:52 smithi046 ceph-mon[31859]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:52 smithi046 ceph-mon[31859]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-08T23:03:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:52 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:52.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:52 smithi062 ceph-mon[35761]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:52.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:52 smithi062 ceph-mon[35761]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:52.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:52 smithi062 ceph-mon[35761]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-08T23:03:52.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:52.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:52 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:52.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 -- 172.21.15.46:0/2527502512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c109300 msgr2=0x7f860c10fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:52.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 --2- 172.21.15.46:0/2527502512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c109300 0x7f860c10fb90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8604009ca0 tx=0x7f860402c0f0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 -- 172.21.15.46:0/2527502512 shutdown_connections 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 --2- 172.21.15.46:0/2527502512 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c109300 0x7f860c10fb90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 --2- 172.21.15.46:0/2527502512 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 0x7f860c108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 --2- 172.21.15.46:0/2527502512 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c107750 0x7f860c107b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 -- 172.21.15.46:0/2527502512 >> 172.21.15.46:0/2527502512 conn(0x7f860c076700 msgr2=0x7f860c078b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.665+0000 7f860be00640 1 -- 172.21.15.46:0/2527502512 shutdown_connections 2024-03-08T23:03:52.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.666+0000 7f860be00640 1 -- 172.21.15.46:0/2527502512 wait complete. 2024-03-08T23:03:52.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 Processor -- start 2024-03-08T23:03:52.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 -- start start 2024-03-08T23:03:52.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:52.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 0x7f860c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 0x7f860c1a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c1133b0 con 0x7f860c107750 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c113230 con 0x7f860c109300 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.667+0000 7f860be00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c113530 con 0x7f860c108940 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860a000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 0x7f860c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34836/0 (socket says 172.21.15.46:34836) 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 -- 172.21.15.46:0/2899191683 learned_addr learned my addr 172.21.15.46:0/2899191683 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860b400640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 0x7f860c1a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 -- 172.21.15.46:0/2899191683 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 msgr2=0x7f860c19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 0x7f860c19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 -- 172.21.15.46:0/2899191683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 msgr2=0x7f860c1a4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:52.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 0x7f860c1a4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860aa00640 1 -- 172.21.15.46:0/2899191683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c1a55c0 con 0x7f860c107750 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.668+0000 7f860b400640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 0x7f860c1a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f860aa00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00e9c0 tx=0x7f85fc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc00cde0 con 0x7f860c107750 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f85fc010910 con 0x7f860c107750 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c1a58b0 con 0x7f860c107750 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85fc018a00 con 0x7f860c107750 2024-03-08T23:03:52.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.669+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c1a5e00 con 0x7f860c107750 2024-03-08T23:03:52.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.670+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c10ea10 con 0x7f860c107750 2024-03-08T23:03:52.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.672+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f85fc01f020 con 0x7f860c107750 2024-03-08T23:03:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.674+0000 7f8603400640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 0x7f85f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.675+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85fc0ce840 con 0x7f860c107750 2024-03-08T23:03:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.675+0000 7f860a000640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 0x7f85f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:52.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.675+0000 7f860a000640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 0x7f85f00a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f860c19eeb0 tx=0x7f85f8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:52.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.679+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f85fc09bce0 con 0x7f860c107750 2024-03-08T23:03:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:52 smithi192 ceph-mon[36700]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:52 smithi192 ceph-mon[36700]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:03:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:52 smithi192 ceph-mon[36700]: Saving service mds.all spec with placement smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-08T23:03:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:52 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:03:52.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.883+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f860c061ef0 con 0x7f860c107750 2024-03-08T23:03:52.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.886+0000 7f8603400640 1 -- 172.21.15.46:0/2899191683 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18432 (secure 0 0 0) 0x7f85fc09b5e0 con 0x7f860c107750 2024-03-08T23:03:52.890 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:52.890 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:03:41.272757+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:03:52.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 msgr2=0x7f85f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 0x7f85f00a4cc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f860c19eeb0 tx=0x7f85f8005f90 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 msgr2=0x7f860c19d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f85fc00e9c0 tx=0x7f85fc00ee80 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 shutdown_connections 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85f00a2800 0x7f85f00a4cc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f860c109300 0x7f860c1a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f860c108940 0x7f860c19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 --2- 172.21.15.46:0/2899191683 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f860c107750 0x7f860c19d9a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 >> 172.21.15.46:0/2899191683 conn(0x7f860c076700 msgr2=0x7f860c10bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 shutdown_connections 2024-03-08T23:03:52.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:52.889+0000 7f860be00640 1 -- 172.21.15.46:0/2899191683 wait complete. 2024-03-08T23:03:52.913 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T23:03:52.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:03:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 -- 172.21.15.46:0/2220185584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 msgr2=0x7f37a0104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 --2- 172.21.15.46:0/2220185584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0104b80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f378c009900 tx=0x7f378c02bc50 comp rx=0 tx=0).stop 2024-03-08T23:03:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 -- 172.21.15.46:0/2220185584 shutdown_connections 2024-03-08T23:03:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 --2- 172.21.15.46:0/2220185584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a01050c0 0x7f37a010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 --2- 172.21.15.46:0/2220185584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0104b80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 --2- 172.21.15.46:0/2220185584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a0103510 0x7f37a0103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 -- 172.21.15.46:0/2220185584 >> 172.21.15.46:0/2220185584 conn(0x7f37a00feaa0 msgr2=0x7f37a0100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 -- 172.21.15.46:0/2220185584 shutdown_connections 2024-03-08T23:03:53.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.188+0000 7f37a7e00640 1 -- 172.21.15.46:0/2220185584 wait complete. 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.189+0000 7f37a7e00640 1 Processor -- start 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.189+0000 7f37a7e00640 1 -- start start 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 0x7f37a0199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 0x7f37a01a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a010f1f0 con 0x7f37a0103510 2024-03-08T23:03:53.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a010f070 con 0x7f37a0104700 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a7e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a010f370 con 0x7f37a01050c0 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 0x7f37a01a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 0x7f37a0199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 0x7f37a01a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37510/0 (socket says 172.21.15.46:37510) 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a5e00640 1 -- 172.21.15.46:0/567519896 learned_addr learned my addr 172.21.15.46:0/567519896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.190+0000 7f37a4a00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 -- 172.21.15.46:0/567519896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 msgr2=0x7f37a01a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 0x7f37a01a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 -- 172.21.15.46:0/567519896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 msgr2=0x7f37a0199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 -- 172.21.15.46:0/567519896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a01a1490 con 0x7f37a0103510 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a5400640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 0x7f37a0199870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f379400e9c0 tx=0x7f379400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:53.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400ccc0 con 0x7f37a0103510 2024-03-08T23:03:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37940035f0 con 0x7f37a0103510 2024-03-08T23:03:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a01a1780 con 0x7f37a0103510 2024-03-08T23:03:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400ce60 con 0x7f37a0103510 2024-03-08T23:03:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.191+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a01a1c50 con 0x7f37a0103510 2024-03-08T23:03:53.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.192+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a0068530 con 0x7f37a0103510 2024-03-08T23:03:53.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.195+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f37940037a0 con 0x7f37a0103510 2024-03-08T23:03:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.196+0000 7f379e000640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 0x7f37880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.196+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f37940ceec0 con 0x7f37a0103510 2024-03-08T23:03:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.197+0000 7f37a4a00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 0x7f37880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.197+0000 7f37a4a00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 0x7f37880a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f378c005ab0 tx=0x7f378c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:53.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.199+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f379409c360 con 0x7f37a0103510 2024-03-08T23:03:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.399+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f37a002ce70 con 0x7f37a0103510 2024-03-08T23:03:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.400+0000 7f379e000640 1 -- 172.21.15.46:0/567519896 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f379409bc60 con 0x7f37a0103510 2024-03-08T23:03:53.404 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:03:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 msgr2=0x7f37880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 0x7f37880a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f378c005ab0 tx=0x7f378c000bc0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 msgr2=0x7f37a0199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 0x7f37a0199870 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f379400e9c0 tx=0x7f379400ee80 comp rx=0 tx=0).stop 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 shutdown_connections 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f37880a2800 0x7f37880a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37a01050c0 0x7f37a01a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37a0104700 0x7f37a0199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 --2- 172.21.15.46:0/567519896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37a0103510 0x7f37a0199870 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 >> 172.21.15.46:0/567519896 conn(0x7f37a00feaa0 msgr2=0x7f37a01079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 shutdown_connections 2024-03-08T23:03:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.403+0000 7f37a7e00640 1 -- 172.21.15.46:0/567519896 wait complete. 2024-03-08T23:03:53.426 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:03:53.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:53.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:53.482 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:53.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:53.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: Deploying daemon mds.l on smithi192 2024-03-08T23:03:53.483 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:53 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2899191683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: Deploying daemon mds.l on smithi192 2024-03-08T23:03:53.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:53 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2899191683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:53.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 -- 172.21.15.46:0/3335504082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 msgr2=0x7f6f34104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 --2- 172.21.15.46:0/3335504082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34104ba0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6f1c009860 tx=0x7f6f1c02bb70 comp rx=0 tx=0).stop 2024-03-08T23:03:53.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 -- 172.21.15.46:0/3335504082 shutdown_connections 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 --2- 172.21.15.46:0/3335504082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f341050e0 0x7f6f3410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 --2- 172.21.15.46:0/3335504082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34104ba0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 --2- 172.21.15.46:0/3335504082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f34103530 0x7f6f34103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 -- 172.21.15.46:0/3335504082 >> 172.21.15.46:0/3335504082 conn(0x7f6f340feaa0 msgr2=0x7f6f34100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 -- 172.21.15.46:0/3335504082 shutdown_connections 2024-03-08T23:03:53.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.646+0000 7f6f3a600640 1 -- 172.21.15.46:0/3335504082 wait complete. 2024-03-08T23:03:53.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 Processor -- start 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 -- start start 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f34103530 0x7f6f34199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3410f1c0 con 0x7f6f34103530 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3410f040 con 0x7f6f341050e0 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.648+0000 7f6f3a600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f3410f340 con 0x7f6f34104720 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f32a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f32a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37526/0 (socket says 172.21.15.46:37526) 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:60824/0 (socket says 172.21.15.46:60824) 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f32a00640 1 -- 172.21.15.46:0/2756754702 learned_addr learned my addr 172.21.15.46:0/2756754702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:53.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33400640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f34103530 0x7f6f34199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 -- 172.21.15.46:0/2756754702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 msgr2=0x7f6f34199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 -- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f34103530 msgr2=0x7f6f34199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f34103530 0x7f6f34199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f33e00640 1 -- 172.21.15.46:0/2756754702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f341a1470 con 0x7f6f341050e0 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.649+0000 7f6f32a00640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:03:53.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f33e00640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c005fd0 tx=0x7f6f2c00ed20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c00ccc0 con 0x7f6f341050e0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f2c0035f0 con 0x7f6f341050e0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f341a1760 con 0x7f6f341050e0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c00ce60 con 0x7f6f341050e0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.650+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f341a1c60 con 0x7f6f341050e0 2024-03-08T23:03:53.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.651+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f3410a8a0 con 0x7f6f341050e0 2024-03-08T23:03:53.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.653+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6f2c0037a0 con 0x7f6f341050e0 2024-03-08T23:03:53.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.654+0000 7f6f2be00640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:53.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.654+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f2c0ce8b0 con 0x7f6f341050e0 2024-03-08T23:03:53.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.654+0000 7f6f33400640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:53.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.655+0000 7f6f33400640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f34104580 tx=0x7f6f24009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:53.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.659+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6f2c09bd50 con 0x7f6f341050e0 2024-03-08T23:03:53.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.729 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:03:53.729 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:53.730 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:53.730 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:53.730 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:53.730 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: Deploying daemon mds.l on smithi192 2024-03-08T23:03:53.730 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:53 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2899191683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:03:53.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.901+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f3402ce70 con 0x7f6f341050e0 2024-03-08T23:03:53.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.902+0000 7f6f2be00640 1 -- 172.21.15.46:0/2756754702 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7f6f2c09b650 con 0x7f6f341050e0 2024-03-08T23:03:53.906 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:53.906 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-08T23:03:53.906 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 1 2024-03-08T23:03:53.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 msgr2=0x7f6f180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 0x7f6f180a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6f34104580 tx=0x7f6f24009500 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 msgr2=0x7f6f341a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c005fd0 tx=0x7f6f2c00ed20 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 shutdown_connections 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6f180a2800 0x7f6f180a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6f341050e0 0x7f6f341a0e00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6f34104720 0x7f6f34199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 --2- 172.21.15.46:0/2756754702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6f34103530 0x7f6f34199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 >> 172.21.15.46:0/2756754702 conn(0x7f6f340feaa0 msgr2=0x7f6f34107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 shutdown_connections 2024-03-08T23:03:53.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:53.905+0000 7f6f3a600640 1 -- 172.21.15.46:0/2756754702 wait complete. 2024-03-08T23:03:54.385 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:54 smithi192 ceph-mon[36700]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:54.385 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/567519896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:54.385 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2756754702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:54.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:54 smithi062 ceph-mon[35761]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:54.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/567519896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:54.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2756754702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:54.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:54 smithi046 ceph-mon[31859]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:54.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/567519896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:03:54.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2756754702' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:54.783 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:54 smithi192 systemd[1]: Starting Ceph mds.l for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:03:55.059 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:54 smithi192 podman[67175]: 2024-03-08 23:03:54.682389387 +0000 UTC m=+0.017524833 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:03:55.059 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:54 smithi192 podman[67175]: 2024-03-08 23:03:54.819449138 +0000 UTC m=+0.154584565 container create 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-08T23:03:55.059 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 podman[67175]: 2024-03-08 23:03:55.055804407 +0000 UTC m=+0.390939833 container init 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-08T23:03:55.314 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 podman[67175]: 2024-03-08 23:03:55.058423409 +0000 UTC m=+0.393558839 container start 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-08T23:03:55.314 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 ceph-mds[67193]: -- 172.21.15.192:0/1229068056 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f5b5c1a40 con 0x556f5b594800 2024-03-08T23:03:55.314 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l[67189]: starting mds.l at 2024-03-08T23:03:55.314 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 bash[67175]: 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e 2024-03-08T23:03:55.314 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:03:55 smithi192 systemd[1]: Started Ceph mds.l for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:03:55.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 -- 172.21.15.46:0/2575222254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 msgr2=0x7f4f00103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 --2- 172.21.15.46:0/2575222254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00103760 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0009900 tx=0x7f4ef002bc30 comp rx=0 tx=0).stop 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 -- 172.21.15.46:0/2575222254 shutdown_connections 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 --2- 172.21.15.46:0/2575222254 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104ef0 0x7f4f0010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 --2- 172.21.15.46:0/2575222254 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104530 0x7f4f001049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 --2- 172.21.15.46:0/2575222254 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00103760 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 -- 172.21.15.46:0/2575222254 >> 172.21.15.46:0/2575222254 conn(0x7f4f000fe8d0 msgr2=0x7f4f00100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 -- 172.21.15.46:0/2575222254 shutdown_connections 2024-03-08T23:03:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.162+0000 7f4f05600640 1 -- 172.21.15.46:0/2575222254 wait complete. 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.163+0000 7f4f05600640 1 Processor -- start 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 -- start start 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 0x7f4f00199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104ef0 0x7f4f001a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0010f020 con 0x7f4f00103340 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0010eea0 con 0x7f4f00104530 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f05600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0010f1a0 con 0x7f4f00104ef0 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4effe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4eff400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 0x7f4f00199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4effe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42550/0 (socket says 172.21.15.46:42550) 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4effe00640 1 -- 172.21.15.46:0/910227617 learned_addr learned my addr 172.21.15.46:0/910227617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.164+0000 7f4f04c00640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104ef0 0x7f4f001a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 -- 172.21.15.46:0/910227617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104ef0 msgr2=0x7f4f001a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104ef0 0x7f4f001a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 -- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 msgr2=0x7f4f00199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 -- 172.21.15.46:0/910227617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f001a1250 con 0x7f4f00104530 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4eff400640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 0x7f4f00199b70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00e9f0 tx=0x7f4eec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec00cd80 con 0x7f4f00104530 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4eec0035f0 con 0x7f4f00104530 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f001a1540 con 0x7f4f00104530 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eec010690 con 0x7f4f00104530 2024-03-08T23:03:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.165+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f001a1a90 con 0x7f4f00104530 2024-03-08T23:03:56.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.167+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0010a660 con 0x7f4f00104530 2024-03-08T23:03:56.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.169+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4eec003b30 con 0x7f4f00104530 2024-03-08T23:03:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.170+0000 7f4efcc00640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.170+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4eec0ce910 con 0x7f4f00104530 2024-03-08T23:03:56.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.171+0000 7f4effe00640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:56.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.172+0000 7f4effe00640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0006010 tx=0x7f4ef002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:56.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.175+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4eec09bdb0 con 0x7f4f00104530 2024-03-08T23:03:56.317 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6832/400907664,v1:172.21.15.192:6833/400907664] up:boot 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: fsmap 1 up:standby 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:56.318 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:56 smithi062 ceph-mon[35761]: Deploying daemon mds.k on smithi062 2024-03-08T23:03:56.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.414+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f0002ce70 con 0x7f4f00104530 2024-03-08T23:03:56.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.415+0000 7f4efcc00640 1 -- 172.21.15.46:0/910227617 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7f4eec0bb070 con 0x7f4f00104530 2024-03-08T23:03:56.419 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:56.420 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-08T23:03:56.420 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 2 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 msgr2=0x7f4ee40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f4ef0006010 tx=0x7f4ef002df90 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 msgr2=0x7f4f00199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 0x7f4f00199b70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00e9f0 tx=0x7f4eec00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 shutdown_connections 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4ee40a2800 0x7f4ee40a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f00104ef0 0x7f4f001a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f00104530 0x7f4f00199b70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 --2- 172.21.15.46:0/910227617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f00103340 0x7f4f00199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 >> 172.21.15.46:0/910227617 conn(0x7f4f000fe8d0 msgr2=0x7f4f00107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 shutdown_connections 2024-03-08T23:03:56.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:56.419+0000 7f4f05600640 1 -- 172.21.15.46:0/910227617 wait complete. 2024-03-08T23:03:56.439 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6832/400907664,v1:172.21.15.192:6833/400907664] up:boot 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: fsmap 1 up:standby 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:56.440 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:56.441 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:56.441 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:56 smithi046 ceph-mon[31859]: Deploying daemon mds.k on smithi062 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6832/400907664,v1:172.21.15.192:6833/400907664] up:boot 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: fsmap 1 up:standby 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:56.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:56.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:56.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:56 smithi192 ceph-mon[36700]: Deploying daemon mds.k on smithi062 2024-03-08T23:03:57.365 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:57 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/910227617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:57.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/910227617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:57.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:57 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/910227617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:57.642 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:57 smithi062 systemd[1]: Starting Ceph mds.k for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:03:58.019 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:57 smithi062 podman[64859]: 2024-03-08 23:03:57.66255323 +0000 UTC m=+0.017946720 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:03:58.019 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:57 smithi062 podman[64859]: 2024-03-08 23:03:57.771945987 +0000 UTC m=+0.127339514 container create 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2) 2024-03-08T23:03:58.019 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 podman[64859]: 2024-03-08 23:03:58.016042917 +0000 UTC m=+0.371436400 container init 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-08T23:03:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:58 smithi062 ceph-mon[35761]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:58.392 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 podman[64859]: 2024-03-08 23:03:58.019297102 +0000 UTC m=+0.374690580 container start 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-08T23:03:58.392 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 ceph-mds[64878]: -- 172.21.15.62:0/660780283 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555563ae7a40 con 0x555563abac00 2024-03-08T23:03:58.392 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k[64874]: starting mds.k at 2024-03-08T23:03:58.392 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 bash[64859]: 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 2024-03-08T23:03:58.392 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:03:58 smithi062 systemd[1]: Started Ceph mds.k for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:03:58.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:03:58.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:58 smithi046 ceph-mon[31859]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:58.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:58 smithi192 ceph-mon[36700]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:03:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2313005338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed01070b0 msgr2=0x7f8ed01094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2313005338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed01070b0 0x7f8ed01094a0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8ec8009900 tx=0x7f8ec802bc30 comp rx=0 tx=0).stop 2024-03-08T23:03:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2313005338 shutdown_connections 2024-03-08T23:03:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2313005338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed01070b0 0x7f8ed01094a0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2313005338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed0104780 0x7f8ed0106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.655+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2313005338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0101e50 0x7f8ed0104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.656+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2313005338 >> 172.21.15.46:0/2313005338 conn(0x7f8ed00fb780 msgr2=0x7f8ed00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.656+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2313005338 shutdown_connections 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.656+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2313005338 wait complete. 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.657+0000 7f8ed7c00640 1 Processor -- start 2024-03-08T23:03:58.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.657+0000 7f8ed7c00640 1 -- start start 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0104780 0x7f8ed0199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed01070b0 0x7f8ed01a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010cfb0 con 0x7f8ed01070b0 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010ce30 con 0x7f8ed0101e50 2024-03-08T23:03:58.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed7c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed010d130 con 0x7f8ed0104780 2024-03-08T23:03:58.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:60846/0 (socket says 172.21.15.46:60846) 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 -- 172.21.15.46:0/2940086535 learned_addr learned my addr 172.21.15.46:0/2940086535 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ecfe00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0104780 0x7f8ed0199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed01070b0 0x7f8ed01a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 -- 172.21.15.46:0/2940086535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0104780 msgr2=0x7f8ed0199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0104780 0x7f8ed0199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 -- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed01070b0 msgr2=0x7f8ed01a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed01070b0 0x7f8ed01a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.658+0000 7f8ed5200640 1 -- 172.21.15.46:0/2940086535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed01a13e0 con 0x7f8ed0101e50 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ed5200640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400ebc0 tx=0x7f8ec400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4010630 con 0x7f8ed0101e50 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4010cb0 con 0x7f8ed0101e50 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed01a16d0 con 0x7f8ed0101e50 2024-03-08T23:03:58.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4018bd0 con 0x7f8ed0101e50 2024-03-08T23:03:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.659+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed01a1c20 con 0x7f8ed0101e50 2024-03-08T23:03:58.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.660+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0108ba0 con 0x7f8ed0101e50 2024-03-08T23:03:58.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.662+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8ec40107d0 con 0x7f8ed0101e50 2024-03-08T23:03:58.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.663+0000 7f8ecd600640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 0x7f8eb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:03:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.663+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8ec40cec40 con 0x7f8ed0101e50 2024-03-08T23:03:58.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.664+0000 7f8ecfe00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 0x7f8eb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:03:58.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.664+0000 7f8ecfe00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 0x7f8eb80a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ed019acd0 tx=0x7f8ebc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:03:58.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.668+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8ec409ba90 con 0x7f8ed0101e50 2024-03-08T23:03:58.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.912+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ed002ce70 con 0x7f8ed0101e50 2024-03-08T23:03:58.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.913+0000 7f8ecd600640 1 -- 172.21.15.46:0/2940086535 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f8ec409fb30 con 0x7f8ed0101e50 2024-03-08T23:03:58.916 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:03:58.916 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-08T23:03:58.916 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-03-08T23:03:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.915+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 msgr2=0x7f8eb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.915+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 0x7f8eb80a4cc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8ed019acd0 tx=0x7f8ebc005f90 comp rx=0 tx=0).stop 2024-03-08T23:03:58.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 msgr2=0x7f8ed01997c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400ebc0 tx=0x7f8ec400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 shutdown_connections 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8eb80a2800 0x7f8eb80a4cc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ed01070b0 0x7f8ed01a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ed0104780 0x7f8ed0199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 --2- 172.21.15.46:0/2940086535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ed0101e50 0x7f8ed01997c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 >> 172.21.15.46:0/2940086535 conn(0x7f8ed00fb780 msgr2=0x7f8ed0101570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 shutdown_connections 2024-03-08T23:03:58.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:03:58.916+0000 7f8ed7c00640 1 -- 172.21.15.46:0/2940086535 wait complete. 2024-03-08T23:03:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6832/3990698134,v1:172.21.15.62:6833/3990698134] up:boot 2024-03-08T23:03:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: fsmap 2 up:standby 2024-03-08T23:03:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:03:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: Deploying daemon mds.j on smithi046 2024-03-08T23:03:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:03:59 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2940086535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6832/3990698134,v1:172.21.15.62:6833/3990698134] up:boot 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: fsmap 2 up:standby 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: Deploying daemon mds.j on smithi046 2024-03-08T23:03:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:03:59 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2940086535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:03:59.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6832/3990698134,v1:172.21.15.62:6833/3990698134] up:boot 2024-03-08T23:03:59.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: fsmap 2 up:standby 2024-03-08T23:03:59.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:03:59.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:03:59.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: Deploying daemon mds.j on smithi046 2024-03-08T23:03:59.566 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:03:59 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2940086535' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:00.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:00 smithi062 ceph-mon[35761]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:00.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:00 smithi046 ceph-mon[31859]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:00.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:00 smithi192 ceph-mon[36700]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:00.940 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:01.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 -- 172.21.15.46:0/1776133773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20101310 msgr2=0x7fbd20101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:01.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 --2- 172.21.15.46:0/1776133773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20101310 0x7fbd20101730 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009900 tx=0x7fbd1002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:01.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 -- 172.21.15.46:0/1776133773 shutdown_connections 2024-03-08T23:04:01.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 --2- 172.21.15.46:0/1776133773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20102ec0 0x7fbd20109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 --2- 172.21.15.46:0/1776133773 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102500 0x7fbd20102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 --2- 172.21.15.46:0/1776133773 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20101310 0x7fbd20101730 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 -- 172.21.15.46:0/1776133773 >> 172.21.15.46:0/1776133773 conn(0x7fbd200fc8a0 msgr2=0x7fbd200fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:01.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.412+0000 7fbd24a00640 1 -- 172.21.15.46:0/1776133773 shutdown_connections 2024-03-08T23:04:01.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.413+0000 7fbd24a00640 1 -- 172.21.15.46:0/1776133773 wait complete. 2024-03-08T23:04:01.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.413+0000 7fbd24a00640 1 Processor -- start 2024-03-08T23:04:01.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 -- start start 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 0x7fbd201997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 0x7fbd201a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2010cff0 con 0x7fbd20102500 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2010ce70 con 0x7fbd20102ec0 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd24a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2010d170 con 0x7fbd20101310 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd1d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd1d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42590/0 (socket says 172.21.15.46:42590) 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd1d400640 1 -- 172.21.15.46:0/543510484 learned_addr learned my addr 172.21.15.46:0/543510484 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd1de00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 0x7fbd201997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:01.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.414+0000 7fbd1e800640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 0x7fbd201a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 -- 172.21.15.46:0/543510484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 msgr2=0x7fbd201a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 0x7fbd201a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 -- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 msgr2=0x7fbd20199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 -- 172.21.15.46:0/543510484 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd201a1410 con 0x7fbd20101310 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1d400640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1e800640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 0x7fbd201a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.415+0000 7fbd1de00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 0x7fbd201997f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009230 tx=0x7fbd1002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.416+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10043020 con 0x7fbd20101310 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.416+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd201a16a0 con 0x7fbd20101310 2024-03-08T23:04:01.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.416+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd100054e0 con 0x7fbd20101310 2024-03-08T23:04:01.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.416+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd10003c60 con 0x7fbd20101310 2024-03-08T23:04:01.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.416+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd201a1c10 con 0x7fbd20101310 2024-03-08T23:04:01.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.417+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd201a1e10 con 0x7fbd20101310 2024-03-08T23:04:01.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.419+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbd10046020 con 0x7fbd20101310 2024-03-08T23:04:01.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.421+0000 7fbd16a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 0x7fbd040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:01.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.421+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fbd100f0670 con 0x7fbd20101310 2024-03-08T23:04:01.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.421+0000 7fbd1d400640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 0x7fbd040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:01.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.422+0000 7fbd1d400640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 0x7fbd040a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0095f0 tx=0x7fbd0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:01.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.424+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbd1004d020 con 0x7fbd20101310 2024-03-08T23:04:01.692 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:01 smithi046 systemd[1]: Starting Ceph mds.j for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:01.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.702+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd201a2080 con 0x7fbd20101310 2024-03-08T23:04:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.703+0000 7fbd16a00640 1 -- 172.21.15.46:0/543510484 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7fbd100bdb10 con 0x7fbd20101310 2024-03-08T23:04:01.707 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:01.707 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-08T23:04:01.707 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 3 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 msgr2=0x7fbd040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 0x7fbd040a4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c0095f0 tx=0x7fbd0c005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 msgr2=0x7fbd201997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 0x7fbd201997f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbd10009230 tx=0x7fbd1002c6c0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 shutdown_connections 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbd040a2880 0x7fbd040a4d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbd20102ec0 0x7fbd201a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbd20102500 0x7fbd20199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 --2- 172.21.15.46:0/543510484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd20101310 0x7fbd201997f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 >> 172.21.15.46:0/543510484 conn(0x7fbd200fc8a0 msgr2=0x7fbd20073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 shutdown_connections 2024-03-08T23:04:01.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:01.706+0000 7fbd24a00640 1 -- 172.21.15.46:0/543510484 wait complete. 2024-03-08T23:04:01.976 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:01 smithi046 podman[69325]: 2024-03-08 23:04:01.596481502 +0000 UTC m=+0.044278093 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:01.977 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:01 smithi046 podman[69325]: 2024-03-08 23:04:01.721838784 +0000 UTC m=+0.169635430 container create ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:04:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:02 smithi046 ceph-mon[31859]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:02 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/543510484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:01 smithi046 podman[69325]: 2024-03-08 23:04:01.984387627 +0000 UTC m=+0.432184257 container init ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:01 smithi046 podman[69325]: 2024-03-08 23:04:01.995178297 +0000 UTC m=+0.442974927 container start ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:02 smithi046 ceph-mds[69346]: -- 172.21.15.46:0/4293373436 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562118e3da40 con 0x562118e10800 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j[69342]: starting mds.j at 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:02 smithi046 bash[69325]: ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 2024-03-08T23:04:02.476 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:04:02 smithi046 systemd[1]: Started Ceph mds.j for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:02.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:02 smithi192 ceph-mon[36700]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:02.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:02 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/543510484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:02.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:02 smithi062 ceph-mon[35761]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:02.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:02 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/543510484' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:03.522 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6834/2033972073,v1:172.21.15.46:6835/2033972073] up:boot 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: fsmap 3 up:standby 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:03.523 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:03.524 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:03.524 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:03 smithi046 ceph-mon[31859]: Deploying daemon mds.d on smithi046 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6834/2033972073,v1:172.21.15.46:6835/2033972073] up:boot 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: fsmap 3 up:standby 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:03.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:03.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:03.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:03 smithi192 ceph-mon[36700]: Deploying daemon mds.d on smithi046 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6834/2033972073,v1:172.21.15.46:6835/2033972073] up:boot 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: fsmap 3 up:standby 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:03.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:03 smithi062 ceph-mon[35761]: Deploying daemon mds.d on smithi046 2024-03-08T23:04:03.734 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:03.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 -- 172.21.15.46:0/3240232535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 msgr2=0x7fe3d8103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:03.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 --2- 172.21.15.46:0/3240232535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8103950 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8009900 tx=0x7fe3c802bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 -- 172.21.15.46:0/3240232535 shutdown_connections 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 --2- 172.21.15.46:0/3240232535 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d81050e0 0x7fe3d810b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 --2- 172.21.15.46:0/3240232535 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d8104720 0x7fe3d8104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 --2- 172.21.15.46:0/3240232535 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8103950 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 -- 172.21.15.46:0/3240232535 >> 172.21.15.46:0/3240232535 conn(0x7fe3d80feae0 msgr2=0x7fe3d8100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 -- 172.21.15.46:0/3240232535 shutdown_connections 2024-03-08T23:04:03.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.958+0000 7fe3dd400640 1 -- 172.21.15.46:0/3240232535 wait complete. 2024-03-08T23:04:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 Processor -- start 2024-03-08T23:04:03.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 -- start start 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 0x7fe3d8199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d81050e0 0x7fe3d81a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f210 con 0x7fe3d8103530 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f090 con 0x7fe3d8104720 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.959+0000 7fe3dd400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3d810f390 con 0x7fe3d81050e0 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58744/0 (socket says 172.21.15.46:58744) 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 -- 172.21.15.46:0/36571157 learned_addr learned my addr 172.21.15.46:0/36571157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d7200640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d81050e0 0x7fe3d81a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d5e00640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 0x7fe3d8199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 -- 172.21.15.46:0/36571157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d81050e0 msgr2=0x7fe3d81a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:03.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d81050e0 0x7fe3d81a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 -- 172.21.15.46:0/36571157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 msgr2=0x7fe3d8199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 0x7fe3d8199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 -- 172.21.15.46:0/36571157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d81a1460 con 0x7fe3d8103530 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d5e00640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 0x7fe3d8199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.960+0000 7fe3d6800640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8000c00 tx=0x7fe3c802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.961+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8043020 con 0x7fe3d8103530 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.961+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3d81a16f0 con 0x7fe3d8103530 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.961+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3c80054e0 con 0x7fe3d8103530 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.961+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8003c40 con 0x7fe3d8103530 2024-03-08T23:04:03.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.961+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3d81a1c40 con 0x7fe3d8103530 2024-03-08T23:04:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.962+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3d810a850 con 0x7fe3d8103530 2024-03-08T23:04:03.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.964+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe3c8046020 con 0x7fe3d8103530 2024-03-08T23:04:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.965+0000 7fe3cf400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.965+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe3c80f0af0 con 0x7fe3d8103530 2024-03-08T23:04:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.966+0000 7fe3d5e00640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:03.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.966+0000 7fe3d5e00640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3d819ad50 tx=0x7fe3c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:03.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:03.968+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe3c804d020 con 0x7fe3d8103530 2024-03-08T23:04:04.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.215+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3d802ce70 con 0x7fe3d8103530 2024-03-08T23:04:04.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.216+0000 7fe3cf400640 1 -- 172.21.15.46:0/36571157 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2907 (secure 0 0 0) 0x7fe3c80a6e20 con 0x7fe3d8103530 2024-03-08T23:04:04.219 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:04.219 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-08T23:04:04.220 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 4 2024-03-08T23:04:04.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.218+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 msgr2=0x7fe3bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:04.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.218+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fe3d819ad50 tx=0x7fe3c4009500 comp rx=0 tx=0).stop 2024-03-08T23:04:04.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.218+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 msgr2=0x7fe3d8199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:04.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.218+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe3c8000c00 tx=0x7fe3c802c650 comp rx=0 tx=0).stop 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 shutdown_connections 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe3bc0a2800 0x7fe3bc0a4cc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe3d81050e0 0x7fe3d81a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe3d8104720 0x7fe3d8199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 --2- 172.21.15.46:0/36571157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe3d8103530 0x7fe3d8199840 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 >> 172.21.15.46:0/36571157 conn(0x7fe3d80feae0 msgr2=0x7fe3d8107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 shutdown_connections 2024-03-08T23:04:04.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:04.219+0000 7fe3dd400640 1 -- 172.21.15.46:0/36571157 wait complete. 2024-03-08T23:04:04.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:04 smithi046 ceph-mon[31859]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:04.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:04 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/36571157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:04.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:04 smithi192 ceph-mon[36700]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:04.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:04 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/36571157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:04 smithi062 ceph-mon[35761]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:04.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:04 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/36571157' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:05.726 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:05 smithi046 systemd[1]: Starting Ceph mds.d for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:06.070 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:05 smithi046 podman[69670]: 2024-03-08 23:04:05.678019182 +0000 UTC m=+0.036901799 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:06.071 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:05 smithi046 podman[69670]: 2024-03-08 23:04:05.794352684 +0000 UTC m=+0.153235336 container create 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True) 2024-03-08T23:04:06.071 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 podman[69670]: 2024-03-08 23:04:06.065811313 +0000 UTC m=+0.424693968 container init 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-08T23:04:06.243 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:06.321 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:06 smithi046 ceph-mon[31859]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:06.322 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 podman[69670]: 2024-03-08 23:04:06.076823439 +0000 UTC m=+0.435706068 container start 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-08T23:04:06.322 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 ceph-mds[69689]: -- 172.21.15.46:0/3618324341 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ca52aeba40 con 0x55ca52abec00 2024-03-08T23:04:06.322 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d[69685]: starting mds.d at 2024-03-08T23:04:06.322 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 bash[69670]: 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 2024-03-08T23:04:06.322 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:04:06 smithi046 systemd[1]: Started Ceph mds.d for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.457+0000 7f87ba200640 1 -- 172.21.15.46:0/503207508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102340 msgr2=0x7f87b41027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.457+0000 7f87ba200640 1 --2- 172.21.15.46:0/503207508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102340 0x7f87b41027c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009900 tx=0x7f87a002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:06.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 -- 172.21.15.46:0/503207508 shutdown_connections 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 --2- 172.21.15.46:0/503207508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4102d00 0x7f87b4109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 --2- 172.21.15.46:0/503207508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102340 0x7f87b41027c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 --2- 172.21.15.46:0/503207508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4101150 0x7f87b4101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 -- 172.21.15.46:0/503207508 >> 172.21.15.46:0/503207508 conn(0x7f87b40fc6c0 msgr2=0x7f87b40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 -- 172.21.15.46:0/503207508 shutdown_connections 2024-03-08T23:04:06.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.458+0000 7f87ba200640 1 -- 172.21.15.46:0/503207508 wait complete. 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 Processor -- start 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 -- start start 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4102340 0x7f87b4199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102d00 0x7f87b41a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:06.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410ce30 con 0x7f87b4102d00 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410ccb0 con 0x7f87b4101150 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87ba200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87b410cfb0 con 0x7f87b4102340 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87b8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87b8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36174/0 (socket says 172.21.15.46:36174) 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87b8e00640 1 -- 172.21.15.46:0/982891324 learned_addr learned my addr 172.21.15.46:0/982891324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.459+0000 7f87b9800640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102d00 0x7f87b41a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b3e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4102340 0x7f87b4199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 -- 172.21.15.46:0/982891324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4102340 msgr2=0x7f87b4199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4102340 0x7f87b4199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 -- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102d00 msgr2=0x7f87b41a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102d00 0x7f87b41a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 -- 172.21.15.46:0/982891324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b41a1260 con 0x7f87b4101150 2024-03-08T23:04:06.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b8e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87a400e9c0 tx=0x7f87a400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.460+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a400cd80 con 0x7f87b4101150 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.461+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87b41a1550 con 0x7f87b4101150 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.461+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87a40035f0 con 0x7f87b4101150 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.461+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87a4010690 con 0x7f87b4101150 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.461+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87b41a1ac0 con 0x7f87b4101150 2024-03-08T23:04:06.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.462+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87b41a1d80 con 0x7f87b4101150 2024-03-08T23:04:06.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.464+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f87a40037a0 con 0x7f87b4101150 2024-03-08T23:04:06.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.465+0000 7f87b1600640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 0x7f87980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:06.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.465+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f87a40cea60 con 0x7f87b4101150 2024-03-08T23:04:06.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.465+0000 7f87b3e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 0x7f87980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:06.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.466+0000 7f87b3e00640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 0x7f87980a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f87a002c240 tx=0x7f87a002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:06.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.467+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f87a409bf90 con 0x7f87b4101150 2024-03-08T23:04:06.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:06 smithi192 ceph-mon[36700]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:06.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:06 smithi062 ceph-mon[35761]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.711+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87b41a1f50 con 0x7f87b4101150 2024-03-08T23:04:06.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.712+0000 7f87b1600640 1 -- 172.21.15.46:0/982891324 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3695 (secure 0 0 0) 0x7f87a409b890 con 0x7f87b4101150 2024-03-08T23:04:06.716 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:06.716 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-08T23:04:06.716 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 5 2024-03-08T23:04:06.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 msgr2=0x7f87980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 0x7f87980a4d40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f87a002c240 tx=0x7f87a002df90 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 msgr2=0x7f87b4199640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f87a400e9c0 tx=0x7f87a400ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 shutdown_connections 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87980a2880 0x7f87980a4d40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87b4102d00 0x7f87b41a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87b4102340 0x7f87b4199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 --2- 172.21.15.46:0/982891324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f87b4101150 0x7f87b4199640 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 >> 172.21.15.46:0/982891324 conn(0x7f87b40fc6c0 msgr2=0x7f87b4073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 shutdown_connections 2024-03-08T23:04:06.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:06.715+0000 7f87ba200640 1 -- 172.21.15.46:0/982891324 wait complete. 2024-03-08T23:04:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6836/950462730,v1:172.21.15.46:6837/950462730] up:boot 2024-03-08T23:04:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: fsmap 4 up:standby 2024-03-08T23:04:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:04:07.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: Deploying daemon mds.e on smithi062 2024-03-08T23:04:07.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:07 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/982891324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6836/950462730,v1:172.21.15.46:6837/950462730] up:boot 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: fsmap 4 up:standby 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:07.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:07.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:07.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: Deploying daemon mds.e on smithi062 2024-03-08T23:04:07.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:07 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/982891324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:07.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6836/950462730,v1:172.21.15.46:6837/950462730] up:boot 2024-03-08T23:04:07.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: fsmap 4 up:standby 2024-03-08T23:04:07.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:04:07.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: Deploying daemon mds.e on smithi062 2024-03-08T23:04:07.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:07 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/982891324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:08.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:08 smithi062 ceph-mon[35761]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:08.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:08 smithi046 ceph-mon[31859]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:08.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:08.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:08 smithi192 ceph-mon[36700]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:08.833 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:08 smithi062 systemd[1]: Starting Ceph mds.e for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:08.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 -- 172.21.15.46:0/1723239513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 msgr2=0x7f079c1049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 --2- 172.21.15.46:0/1723239513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c1049b0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0788009900 tx=0x7f078802bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 -- 172.21.15.46:0/1723239513 shutdown_connections 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 --2- 172.21.15.46:0/1723239513 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c104ef0 0x7f079c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 --2- 172.21.15.46:0/1723239513 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c1049b0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 --2- 172.21.15.46:0/1723239513 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f079c103340 0x7f079c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 -- 172.21.15.46:0/1723239513 >> 172.21.15.46:0/1723239513 conn(0x7f079c0fe8d0 msgr2=0x7f079c100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 -- 172.21.15.46:0/1723239513 shutdown_connections 2024-03-08T23:04:08.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.971+0000 7f07a2400640 1 -- 172.21.15.46:0/1723239513 wait complete. 2024-03-08T23:04:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 Processor -- start 2024-03-08T23:04:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 -- start start 2024-03-08T23:04:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c103340 0x7f079c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:08.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f079c104ef0 0x7f079c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c10f080 con 0x7f079c104530 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c10ef00 con 0x7f079c104ef0 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.972+0000 7f07a2400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c10f200 con 0x7f079c103340 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58784/0 (socket says 172.21.15.46:58784) 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 -- 172.21.15.46:0/1372519623 learned_addr learned my addr 172.21.15.46:0/1372519623 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f07a1000640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c103340 0x7f079c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 -- 172.21.15.46:0/1372519623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c103340 msgr2=0x7f079c1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c103340 0x7f079c1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 -- 172.21.15.46:0/1372519623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f079c104ef0 msgr2=0x7f079c1a0c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f079c104ef0 0x7f079c1a0c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 -- 172.21.15.46:0/1372519623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c1a1300 con 0x7f079c104530 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.973+0000 7f079be00640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f078802c8e0 tx=0x7f0788005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:08.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.974+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788043020 con 0x7f079c104530 2024-03-08T23:04:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.974+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f07880054e0 con 0x7f079c104530 2024-03-08T23:04:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.974+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c1a1590 con 0x7f079c104530 2024-03-08T23:04:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.974+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0788003da0 con 0x7f079c104530 2024-03-08T23:04:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.974+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c1a1b30 con 0x7f079c104530 2024-03-08T23:04:08.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.975+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c10a600 con 0x7f079c104530 2024-03-08T23:04:08.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.977+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0788046020 con 0x7f079c104530 2024-03-08T23:04:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.980+0000 7f0799600640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 0x7f07840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.980+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f07880f0d10 con 0x7f079c104530 2024-03-08T23:04:08.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.980+0000 7f07a1000640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 0x7f07840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:08.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.981+0000 7f07a1000640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 0x7f07840a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f079c104390 tx=0x7f0790009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:08.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:08.983+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f078804d020 con 0x7f079c104530 2024-03-08T23:04:09.090 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:08 smithi062 podman[65181]: 2024-03-08 23:04:08.732911258 +0000 UTC m=+0.018029823 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:09.091 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:08 smithi062 podman[65181]: 2024-03-08 23:04:08.843388068 +0000 UTC m=+0.128506667 container create cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-08T23:04:09.091 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 podman[65181]: 2024-03-08 23:04:09.087631647 +0000 UTC m=+0.372750201 container init cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:04:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.226+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f079c02ce70 con 0x7f079c104530 2024-03-08T23:04:09.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.227+0000 7f0799600640 1 -- 172.21.15.46:0/1372519623 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3695 (secure 0 0 0) 0x7f07880473b0 con 0x7f079c104530 2024-03-08T23:04:09.231 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:09.231 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[]} 2024-03-08T23:04:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 5 2024-03-08T23:04:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.230+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 msgr2=0x7f07840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.230+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 0x7f07840a4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f079c104390 tx=0x7f0790009110 comp rx=0 tx=0).stop 2024-03-08T23:04:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.230+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 msgr2=0x7f079c199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.230+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f078802c8e0 tx=0x7f0788005dd0 comp rx=0 tx=0).stop 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 shutdown_connections 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f07840a2880 0x7f07840a4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f079c104ef0 0x7f079c1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f079c104530 0x7f079c199c20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 --2- 172.21.15.46:0/1372519623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f079c103340 0x7f079c1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 >> 172.21.15.46:0/1372519623 conn(0x7f079c0fe8d0 msgr2=0x7f079c1077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 shutdown_connections 2024-03-08T23:04:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:09.231+0000 7f07a2400640 1 -- 172.21.15.46:0/1372519623 wait complete. 2024-03-08T23:04:09.365 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 podman[65181]: 2024-03-08 23:04:09.090417412 +0000 UTC m=+0.375535963 container start cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:04:09.365 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mds[65200]: -- 172.21.15.62:0/1128383627 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5653ad677a40 con 0x5653ad64a800 2024-03-08T23:04:09.365 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e[65196]: starting mds.e at 2024-03-08T23:04:09.365 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 bash[65181]: cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f 2024-03-08T23:04:09.365 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:04:09 smithi062 systemd[1]: Started Ceph mds.e for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:09.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1372519623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:09.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:09 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:09.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1372519623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:09.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:09 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1372519623' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:09.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:09 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:10 smithi192 ceph-mon[36700]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:10 smithi192 ceph-mon[36700]: Deploying daemon mds.f on smithi192 2024-03-08T23:04:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:10 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6834/213914904,v1:172.21.15.62:6835/213914904] up:boot 2024-03-08T23:04:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:10 smithi192 ceph-mon[36700]: fsmap 5 up:standby 2024-03-08T23:04:10.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:10 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:04:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:10 smithi062 ceph-mon[35761]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:10.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:10 smithi062 ceph-mon[35761]: Deploying daemon mds.f on smithi192 2024-03-08T23:04:10.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:10 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6834/213914904,v1:172.21.15.62:6835/213914904] up:boot 2024-03-08T23:04:10.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:10 smithi062 ceph-mon[35761]: fsmap 5 up:standby 2024-03-08T23:04:10.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:10 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:04:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:10 smithi046 ceph-mon[31859]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:10 smithi046 ceph-mon[31859]: Deploying daemon mds.f on smithi192 2024-03-08T23:04:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:10 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6834/213914904,v1:172.21.15.62:6835/213914904] up:boot 2024-03-08T23:04:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:10 smithi046 ceph-mon[31859]: fsmap 5 up:standby 2024-03-08T23:04:10.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:10 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:04:11.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.488+0000 7f32efe00640 1 -- 172.21.15.46:0/3443900499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 msgr2=0x7f32f0101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.488+0000 7f32efe00640 1 --2- 172.21.15.46:0/3443900499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f0101550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f32e0009900 tx=0x7f32e002bc50 comp rx=0 tx=0).stop 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.488+0000 7f32efe00640 1 -- 172.21.15.46:0/3443900499 shutdown_connections 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.488+0000 7f32efe00640 1 --2- 172.21.15.46:0/3443900499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102ce0 0x7f32f0109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.489+0000 7f32efe00640 1 --2- 172.21.15.46:0/3443900499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102320 0x7f32f01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.489+0000 7f32efe00640 1 --2- 172.21.15.46:0/3443900499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f0101550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.489+0000 7f32efe00640 1 -- 172.21.15.46:0/3443900499 >> 172.21.15.46:0/3443900499 conn(0x7f32f00fc680 msgr2=0x7f32f00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.489+0000 7f32efe00640 1 -- 172.21.15.46:0/3443900499 shutdown_connections 2024-03-08T23:04:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.489+0000 7f32efe00640 1 -- 172.21.15.46:0/3443900499 wait complete. 2024-03-08T23:04:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.490+0000 7f32efe00640 1 Processor -- start 2024-03-08T23:04:11.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.490+0000 7f32efe00640 1 -- start start 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102320 0x7f32f0199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102ce0 0x7f32f01a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f010cdc0 con 0x7f32f0102ce0 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f010cc40 con 0x7f32f0102320 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32efe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f010cf40 con 0x7f32f0101130 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48258/0 (socket says 172.21.15.46:48258) 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 -- 172.21.15.46:0/1810054196 learned_addr learned my addr 172.21.15.46:0/1810054196 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32e7e00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102320 0x7f32f0199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32ef400640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102ce0 0x7f32f01a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:11.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 -- 172.21.15.46:0/1810054196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102320 msgr2=0x7f32f0199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102320 0x7f32f0199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 -- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102ce0 msgr2=0x7f32f01a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102ce0 0x7f32f01a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.491+0000 7f32eea00640 1 -- 172.21.15.46:0/1810054196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f01a1370 con 0x7f32f0101130 2024-03-08T23:04:11.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32eea00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c00 tx=0x7f32e002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0042020 con 0x7f32f0101130 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f32e0004450 con 0x7f32f0101130 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f01a1600 con 0x7f32f0101130 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e003e460 con 0x7f32f0101130 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.492+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f01a1ba0 con 0x7f32f0101130 2024-03-08T23:04:11.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.493+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f01a1da0 con 0x7f32f0101130 2024-03-08T23:04:11.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.495+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f32e0045020 con 0x7f32f0101130 2024-03-08T23:04:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.497+0000 7f32e7400640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 0x7f32d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.497+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f32e00f0540 con 0x7f32f0101130 2024-03-08T23:04:11.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.497+0000 7f32e7e00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 0x7f32d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:11.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.498+0000 7f32e7e00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 0x7f32d40a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f32f019ac60 tx=0x7f32d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:11.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.501+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f32e004c020 con 0x7f32f0101130 2024-03-08T23:04:11.564 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 systemd[1]: Starting Ceph mds.f for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.746+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32f01a1f70 con 0x7f32f0101130 2024-03-08T23:04:11.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.747+0000 7f32e7400640 1 -- 172.21.15.46:0/1810054196 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4483 (secure 0 0 0) 0x7f32e00bda70 con 0x7f32f0101130 2024-03-08T23:04:11.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:11.751 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-08T23:04:11.751 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 6 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 msgr2=0x7f32d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 0x7f32d40a4d40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f32f019ac60 tx=0x7f32d8005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 msgr2=0x7f32f01997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f32e0000c00 tx=0x7f32e002c660 comp rx=0 tx=0).stop 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 shutdown_connections 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32d40a2880 0x7f32d40a4d40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f32f0102ce0 0x7f32f01a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f32f0102320 0x7f32f0199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 --2- 172.21.15.46:0/1810054196 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f32f0101130 0x7f32f01997a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 >> 172.21.15.46:0/1810054196 conn(0x7f32f00fc680 msgr2=0x7f32f0073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 shutdown_connections 2024-03-08T23:04:11.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:11.750+0000 7f32efe00640 1 -- 172.21.15.46:0/1810054196 wait complete. 2024-03-08T23:04:11.938 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 podman[67483]: 2024-03-08 23:04:11.584641392 +0000 UTC m=+0.017742355 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:11.938 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 podman[67483]: 2024-03-08 23:04:11.707620701 +0000 UTC m=+0.140721660 container create 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-08T23:04:11.938 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 podman[67483]: 2024-03-08 23:04:11.935414855 +0000 UTC m=+0.368515821 container init 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:04:12.314 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 podman[67483]: 2024-03-08 23:04:11.938210975 +0000 UTC m=+0.371311936 container start 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-08T23:04:12.314 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 ceph-mds[67502]: -- 172.21.15.192:0/921190642 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b92f7eba40 con 0x55b92f7bec00 2024-03-08T23:04:12.314 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: starting mds.f at 2024-03-08T23:04:12.314 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:12 smithi192 bash[67483]: 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 2024-03-08T23:04:12.314 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:04:12 smithi192 systemd[1]: Started Ceph mds.f for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1810054196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:12.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:12 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1810054196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:12.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:12.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:12 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1810054196' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:12.766 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:12 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:13.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:13 smithi046 ceph-mon[31859]: Deploying daemon mds.c on smithi192 2024-03-08T23:04:13.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:13 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6834/3258548049,v1:172.21.15.192:6835/3258548049] up:boot 2024-03-08T23:04:13.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:13 smithi046 ceph-mon[31859]: fsmap 6 up:standby 2024-03-08T23:04:13.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:13 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:04:13.775 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:13.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:13 smithi192 ceph-mon[36700]: Deploying daemon mds.c on smithi192 2024-03-08T23:04:13.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:13 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6834/3258548049,v1:172.21.15.192:6835/3258548049] up:boot 2024-03-08T23:04:13.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:13 smithi192 ceph-mon[36700]: fsmap 6 up:standby 2024-03-08T23:04:13.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:13 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:04:13.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:13 smithi062 ceph-mon[35761]: Deploying daemon mds.c on smithi192 2024-03-08T23:04:13.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:13 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6834/3258548049,v1:172.21.15.192:6835/3258548049] up:boot 2024-03-08T23:04:13.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:13 smithi062 ceph-mon[35761]: fsmap 6 up:standby 2024-03-08T23:04:13.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:13 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 -- 172.21.15.46:0/3162209113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c1044f0 msgr2=0x7fac2c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 --2- 172.21.15.46:0/3162209113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c1044f0 0x7fac2c1068e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac18009900 tx=0x7fac1802bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 -- 172.21.15.46:0/3162209113 shutdown_connections 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 --2- 172.21.15.46:0/3162209113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c106e20 0x7fac2c109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 --2- 172.21.15.46:0/3162209113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c1044f0 0x7fac2c1068e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 --2- 172.21.15.46:0/3162209113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 -- 172.21.15.46:0/3162209113 >> 172.21.15.46:0/3162209113 conn(0x7fac2c0fb5d0 msgr2=0x7fac2c0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 -- 172.21.15.46:0/3162209113 shutdown_connections 2024-03-08T23:04:14.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.009+0000 7fac2b200640 1 -- 172.21.15.46:0/3162209113 wait complete. 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.010+0000 7fac2b200640 1 Processor -- start 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.010+0000 7fac2b200640 1 -- start start 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 0x7fac2c1a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10ce20 con 0x7fac2c106e20 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10cca0 con 0x7fac2c101bc0 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac2c10cfa0 con 0x7fac2c1044f0 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 0x7fac2c1a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:14.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:58926/0 (socket says 172.21.15.46:58926) 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 -- 172.21.15.46:0/879650685 learned_addr learned my addr 172.21.15.46:0/879650685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 -- 172.21.15.46:0/879650685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 msgr2=0x7fac2c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 -- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 msgr2=0x7fac2c1a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 0x7fac2c1a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29400640 1 -- 172.21.15.46:0/879650685 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2c1a1290 con 0x7fac2c1044f0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac29e00640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.011+0000 7fac2a800640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 0x7fac2c1a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac29400640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fac18000c00 tx=0x7fac1802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18043020 con 0x7fac2c1044f0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac2c1a1520 con 0x7fac2c1044f0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac180054e0 con 0x7fac2c1044f0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac18003c40 con 0x7fac2c1044f0 2024-03-08T23:04:14.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.012+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2c1a1a70 con 0x7fac2c1044f0 2024-03-08T23:04:14.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.013+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2c108970 con 0x7fac2c1044f0 2024-03-08T23:04:14.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.015+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fac18046020 con 0x7fac2c1044f0 2024-03-08T23:04:14.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.017+0000 7fac22a00640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 0x7fac100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:14.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.017+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fac180f0a40 con 0x7fac2c1044f0 2024-03-08T23:04:14.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.017+0000 7fac29e00640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 0x7fac100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:14.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.018+0000 7fac29e00640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 0x7fac100a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fc0 tx=0x7fac1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:14.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.019+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fac1804d020 con 0x7fac2c1044f0 2024-03-08T23:04:14.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.262+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2c02ce70 con 0x7fac2c1044f0 2024-03-08T23:04:14.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.264+0000 7fac22a00640 1 -- 172.21.15.46:0/879650685 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5277 (secure 0 0 0) 0x7fac180bdee0 con 0x7fac2c1044f0 2024-03-08T23:04:14.268 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:14.268 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-08T23:04:14.269 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 7 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 msgr2=0x7fac100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 0x7fac100a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fac1c009fc0 tx=0x7fac1c005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 msgr2=0x7fac2c199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fac18000c00 tx=0x7fac1802c650 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 shutdown_connections 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fac100a2800 0x7fac100a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fac2c106e20 0x7fac2c1a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fac2c1044f0 0x7fac2c199bb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 --2- 172.21.15.46:0/879650685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fac2c101bc0 0x7fac2c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.266+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 >> 172.21.15.46:0/879650685 conn(0x7fac2c0fb5d0 msgr2=0x7fac2c101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:14.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.267+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 shutdown_connections 2024-03-08T23:04:14.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:14.267+0000 7fac2b200640 1 -- 172.21.15.46:0/879650685 wait complete. 2024-03-08T23:04:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:14 smithi192 ceph-mon[36700]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:14.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/879650685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:14.604 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:14 smithi046 ceph-mon[31859]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:14.604 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/879650685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:14 smithi062 ceph-mon[35761]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:14.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/879650685' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:15.314 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:04:15 smithi192 systemd[1]: Started Ceph mds.c for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:16.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: Deploying daemon mds.h on smithi062 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6836/3408320280,v1:172.21.15.192:6837/3408320280] up:boot 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: fsmap 7 up:standby 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:16 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: Deploying daemon mds.h on smithi062 2024-03-08T23:04:16.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:16.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6836/3408320280,v1:172.21.15.192:6837/3408320280] up:boot 2024-03-08T23:04:16.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: fsmap 7 up:standby 2024-03-08T23:04:16.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:16 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 -- 172.21.15.46:0/2071903972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8077890 msgr2=0x7fdbc8078130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 --2- 172.21.15.46:0/2071903972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8077890 0x7fdbc8078130 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0009ca0 tx=0x7fdbc002c100 comp rx=0 tx=0).stop 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 -- 172.21.15.46:0/2071903972 shutdown_connections 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 --2- 172.21.15.46:0/2071903972 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8077890 0x7fdbc8078130 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 --2- 172.21.15.46:0/2071903972 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8076ef0 0x7fdbc8077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 --2- 172.21.15.46:0/2071903972 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8075cb0 0x7fdbc80760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 -- 172.21.15.46:0/2071903972 >> 172.21.15.46:0/2071903972 conn(0x7fdbc80feaa0 msgr2=0x7fdbc8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:16.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 -- 172.21.15.46:0/2071903972 shutdown_connections 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.518+0000 7fdbccc00640 1 -- 172.21.15.46:0/2071903972 wait complete. 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 Processor -- start 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 -- start start 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 0x7fdbc819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8077890 0x7fdbc81a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc81135c0 con 0x7fdbc8076ef0 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8113440 con 0x7fdbc8075cb0 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbccc00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc8113740 con 0x7fdbc8077890 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.519+0000 7fdbc5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:16.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58828/0 (socket says 172.21.15.46:58828) 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 -- 172.21.15.46:0/117844368 learned_addr learned my addr 172.21.15.46:0/117844368 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5e00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 0x7fdbc819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc6800640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8077890 0x7fdbc81a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 -- 172.21.15.46:0/117844368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8077890 msgr2=0x7fdbc81a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8077890 0x7fdbc81a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 -- 172.21.15.46:0/117844368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 msgr2=0x7fdbc819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 0x7fdbc819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 -- 172.21.15.46:0/117844368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc81a5810 con 0x7fdbc8076ef0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5e00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 0x7fdbc819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbc5400640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d950 tx=0x7fdbb400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.520+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4017070 con 0x7fdbc8076ef0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.521+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb4009c30 con 0x7fdbc8076ef0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.521+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc81a5b00 con 0x7fdbc8076ef0 2024-03-08T23:04:16.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.521+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb400f6b0 con 0x7fdbc8076ef0 2024-03-08T23:04:16.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.521+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc81a6050 con 0x7fdbc8076ef0 2024-03-08T23:04:16.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.522+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc8110030 con 0x7fdbc8076ef0 2024-03-08T23:04:16.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.525+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdbb40109d0 con 0x7fdbc8076ef0 2024-03-08T23:04:16.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.526+0000 7fdbbea00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 0x7fdbac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:16.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.527+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdbb40ce1b0 con 0x7fdbc8076ef0 2024-03-08T23:04:16.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.527+0000 7fdbc5e00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 0x7fdbac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:16.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.527+0000 7fdbc5e00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 0x7fdbac0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fc0 tx=0x7fdbb8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:16.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.528+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdbb4009de0 con 0x7fdbc8076ef0 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:16.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: Deploying daemon mds.h on smithi062 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6836/3408320280,v1:172.21.15.192:6837/3408320280] up:boot 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: fsmap 7 up:standby 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:16 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:04:16.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.763+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbc802d070 con 0x7fdbc8076ef0 2024-03-08T23:04:16.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.764+0000 7fdbbea00640 1 -- 172.21.15.46:0/117844368 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6071 (secure 0 0 0) 0x7fdbb409b000 con 0x7fdbc8076ef0 2024-03-08T23:04:16.768 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:16.769 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-08T23:04:16.769 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 8 2024-03-08T23:04:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 msgr2=0x7fdbac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 0x7fdbac0a4cc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fc0 tx=0x7fdbb8005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 msgr2=0x7fdbc819e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:16.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d950 tx=0x7fdbb400de10 comp rx=0 tx=0).stop 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 shutdown_connections 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbac0a2800 0x7fdbac0a4cc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc8077890 0x7fdbc81a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc8076ef0 0x7fdbc819e130 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 --2- 172.21.15.46:0/117844368 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc8075cb0 0x7fdbc819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.766+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 >> 172.21.15.46:0/117844368 conn(0x7fdbc80feaa0 msgr2=0x7fdbc8100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.767+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 shutdown_connections 2024-03-08T23:04:16.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:16.767+0000 7fdbccc00640 1 -- 172.21.15.46:0/117844368 wait complete. 2024-03-08T23:04:17.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:17 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/117844368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:17.392 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 systemd[1]: Starting Ceph mds.h for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:17.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:17 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/117844368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:17.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:17 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/117844368' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:17.789 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 podman[65489]: 2024-03-08 23:04:17.419797714 +0000 UTC m=+0.018143135 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:17.789 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 podman[65489]: 2024-03-08 23:04:17.540173144 +0000 UTC m=+0.138518598 container create 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-08T23:04:17.789 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 podman[65489]: 2024-03-08 23:04:17.759362534 +0000 UTC m=+0.357707961 container init 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-08T23:04:17.789 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 podman[65489]: 2024-03-08 23:04:17.762043919 +0000 UTC m=+0.360389332 container start 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True) 2024-03-08T23:04:17.789 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 ceph-mds[65509]: -- 172.21.15.62:0/3495452464 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560eecd21a40 con 0x560eeccf4800 2024-03-08T23:04:18.142 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: starting mds.h at 2024-03-08T23:04:18.142 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 bash[65489]: 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 2024-03-08T23:04:18.142 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:04:17 smithi062 systemd[1]: Started Ceph mds.h for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:18.788 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:19.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.021+0000 7f5f8b200640 1 -- 172.21.15.46:0/997413348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c104540 msgr2=0x7f5f8c1049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.021+0000 7f5f8b200640 1 --2- 172.21.15.46:0/997413348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c104540 0x7f5f8c1049c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009900 tx=0x7f5f7802bc40 comp rx=0 tx=0).stop 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 -- 172.21.15.46:0/997413348 shutdown_connections 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 --2- 172.21.15.46:0/997413348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104f00 0x7f5f8c10b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 --2- 172.21.15.46:0/997413348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c104540 0x7f5f8c1049c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 --2- 172.21.15.46:0/997413348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c103350 0x7f5f8c103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 -- 172.21.15.46:0/997413348 >> 172.21.15.46:0/997413348 conn(0x7f5f8c0fe8c0 msgr2=0x7f5f8c100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 -- 172.21.15.46:0/997413348 shutdown_connections 2024-03-08T23:04:19.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.022+0000 7f5f8b200640 1 -- 172.21.15.46:0/997413348 wait complete. 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 Processor -- start 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 -- start start 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 0x7f5f8c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 0x7f5f8c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c10efe0 con 0x7f5f8c103350 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c10ee60 con 0x7f5f8c104540 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.023+0000 7f5f8b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f8c10f160 con 0x7f5f8c104f00 2024-03-08T23:04:19.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58846/0 (socket says 172.21.15.46:58846) 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 -- 172.21.15.46:0/3481314907 learned_addr learned my addr 172.21.15.46:0/3481314907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f8a800640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 0x7f5f8c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89400640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 0x7f5f8c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 -- 172.21.15.46:0/3481314907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 msgr2=0x7f5f8c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 0x7f5f8c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 -- 172.21.15.46:0/3481314907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 msgr2=0x7f5f8c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 0x7f5f8c199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 -- 172.21.15.46:0/3481314907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8c1a1280 con 0x7f5f8c103350 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f8a800640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 0x7f5f8c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:04:19.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89400640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 0x7f5f8c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.024+0000 7f5f89e00640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e9c0 tx=0x7f5f7c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.025+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00cd80 con 0x7f5f8c103350 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.025+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f7c0035f0 con 0x7f5f8c103350 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.025+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8c1a1570 con 0x7f5f8c103350 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.025+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c010690 con 0x7f5f8c103350 2024-03-08T23:04:19.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.025+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f8c1a1ac0 con 0x7f5f8c103350 2024-03-08T23:04:19.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.026+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8c10a670 con 0x7f5f8c103350 2024-03-08T23:04:19.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.028+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5f7c0037a0 con 0x7f5f8c103350 2024-03-08T23:04:19.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.029+0000 7f5f82a00640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 0x7f5f700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:19.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.030+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5f7c0ceb80 con 0x7f5f8c103350 2024-03-08T23:04:19.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.030+0000 7f5f89400640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 0x7f5f700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:19.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.031+0000 7f5f89400640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 0x7f5f700a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5f78036040 tx=0x7f5f78000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:19.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.034+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5f7c09c020 con 0x7f5f8c103350 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:19.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:19.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: Deploying daemon mds.i on smithi192 2024-03-08T23:04:19.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6836/2774760539,v1:172.21.15.62:6837/2774760539] up:boot 2024-03-08T23:04:19.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: fsmap 8 up:standby 2024-03-08T23:04:19.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:18 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:04:19.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.285+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f8c02ce70 con 0x7f5f8c103350 2024-03-08T23:04:19.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.287+0000 7f5f82a00640 1 -- 172.21.15.46:0/3481314907 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6862 (secure 0 0 0) 0x7f5f7c09b920 con 0x7f5f8c103350 2024-03-08T23:04:19.291 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:19.291 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-08T23:04:19.292 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 9 2024-03-08T23:04:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 msgr2=0x7f5f700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 0x7f5f700a4cc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5f78036040 tx=0x7f5f78000bc0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 msgr2=0x7f5f8c199660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:19.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00e9c0 tx=0x7f5f7c00ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 shutdown_connections 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f700a2800 0x7f5f700a4cc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f8c104f00 0x7f5f8c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f8c104540 0x7f5f8c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 --2- 172.21.15.46:0/3481314907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f8c103350 0x7f5f8c199660 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 >> 172.21.15.46:0/3481314907 conn(0x7f5f8c0fe8c0 msgr2=0x7f5f8c107840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 shutdown_connections 2024-03-08T23:04:19.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:19.290+0000 7f5f8b200640 1 -- 172.21.15.46:0/3481314907 wait complete. 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:19.302 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: Deploying daemon mds.i on smithi192 2024-03-08T23:04:19.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6836/2774760539,v1:172.21.15.62:6837/2774760539] up:boot 2024-03-08T23:04:19.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: fsmap 8 up:standby 2024-03-08T23:04:19.303 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:18 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: Deploying daemon mds.i on smithi192 2024-03-08T23:04:19.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6836/2774760539,v1:172.21.15.62:6837/2774760539] up:boot 2024-03-08T23:04:19.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: fsmap 8 up:standby 2024-03-08T23:04:19.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:18 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:04:20.008 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:19 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3481314907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:20.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:19 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3481314907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:20.315 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 systemd[1]: Starting Ceph mds.i for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:20.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:19 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3481314907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:20.662 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 podman[68097]: 2024-03-08 23:04:20.306534286 +0000 UTC m=+0.017759871 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:20.663 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 podman[68097]: 2024-03-08 23:04:20.431492265 +0000 UTC m=+0.142717845 container create eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-08T23:04:20.663 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 podman[68097]: 2024-03-08 23:04:20.659567563 +0000 UTC m=+0.370793152 container init eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-08T23:04:20.923 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 podman[68097]: 2024-03-08 23:04:20.662175611 +0000 UTC m=+0.373401191 container start eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-08T23:04:20.923 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mds[68116]: -- 172.21.15.192:0/1323196360 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56242e87ba40 con 0x56242e84e800 2024-03-08T23:04:20.923 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: starting mds.i at 2024-03-08T23:04:20.923 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 bash[68097]: eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a 2024-03-08T23:04:20.923 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:04:20 smithi192 systemd[1]: Started Ceph mds.i for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:21.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:21.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:20 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:21.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:21.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:20 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:21.315 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:21.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:21.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:21.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:20 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.534+0000 7f7e33200640 1 -- 172.21.15.46:0/3602893508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102340 msgr2=0x7f7e341027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.534+0000 7f7e33200640 1 --2- 172.21.15.46:0/3602893508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102340 0x7f7e341027c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7e20009900 tx=0x7f7e2002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 -- 172.21.15.46:0/3602893508 shutdown_connections 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 --2- 172.21.15.46:0/3602893508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102d00 0x7f7e34109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 --2- 172.21.15.46:0/3602893508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102340 0x7f7e341027c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 --2- 172.21.15.46:0/3602893508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 -- 172.21.15.46:0/3602893508 >> 172.21.15.46:0/3602893508 conn(0x7f7e340fc6c0 msgr2=0x7f7e340feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:21.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 -- 172.21.15.46:0/3602893508 shutdown_connections 2024-03-08T23:04:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.535+0000 7f7e33200640 1 -- 172.21.15.46:0/3602893508 wait complete. 2024-03-08T23:04:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 Processor -- start 2024-03-08T23:04:21.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 -- start start 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102340 0x7f7e34197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3410ce30 con 0x7f7e34102d00 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3410ccb0 con 0x7f7e34101150 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.536+0000 7f7e33200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3410cfb0 con 0x7f7e34102340 2024-03-08T23:04:21.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e31e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e31400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102340 0x7f7e34197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58870/0 (socket says 172.21.15.46:58870) 2024-03-08T23:04:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e31e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56724/0 (socket says 172.21.15.46:56724) 2024-03-08T23:04:21.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 -- 172.21.15.46:0/3511296788 learned_addr learned my addr 172.21.15.46:0/3511296788 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 -- 172.21.15.46:0/3511296788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102340 msgr2=0x7f7e34197960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102340 0x7f7e34197960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 -- 172.21.15.46:0/3511296788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 msgr2=0x7f7e34197420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e32800640 1 -- 172.21.15.46:0/3511296788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e3419f040 con 0x7f7e34102d00 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.537+0000 7f7e31e00640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:21.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e32800640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00eb10 tx=0x7f7e2c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c004b20 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e3419f330 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e2c004cc0 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e2c010760 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.538+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e3419f8a0 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.539+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e34108410 con 0x7f7e34102d00 2024-03-08T23:04:21.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.541+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7e2c005020 con 0x7f7e34102d00 2024-03-08T23:04:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.543+0000 7f7e2aa00640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 0x7f7e180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.543+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7e2c0cecb0 con 0x7f7e34102d00 2024-03-08T23:04:21.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.543+0000 7f7e31e00640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 0x7f7e180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:21.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.544+0000 7f7e31e00640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 0x7f7e180a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e341021a0 tx=0x7f7e24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:21.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.548+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7e2c09c150 con 0x7f7e34102d00 2024-03-08T23:04:21.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.828+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e3402ce70 con 0x7f7e34102d00 2024-03-08T23:04:21.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.829+0000 7f7e2aa00640 1 -- 172.21.15.46:0/3511296788 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7655 (secure 0 0 0) 0x7f7e2c09ba50 con 0x7f7e34102d00 2024-03-08T23:04:21.835 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:21.835 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-08T23:04:21.835 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 10 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 msgr2=0x7f7e180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 0x7f7e180a4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f7e341021a0 tx=0x7f7e24005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 msgr2=0x7f7e3419e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7e2c00eb10 tx=0x7f7e2c00c2d0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 shutdown_connections 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7e180a2880 0x7f7e180a4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e34102d00 0x7f7e3419e9d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e34102340 0x7f7e34197960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 --2- 172.21.15.46:0/3511296788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e34101150 0x7f7e34197420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 >> 172.21.15.46:0/3511296788 conn(0x7f7e340fc6c0 msgr2=0x7f7e341055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:21.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 shutdown_connections 2024-03-08T23:04:21.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:21.832+0000 7f7e33200640 1 -- 172.21.15.46:0/3511296788 wait complete. 2024-03-08T23:04:22.165 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:21 smithi046 ceph-mon[31859]: Deploying daemon mds.g on smithi046 2024-03-08T23:04:22.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:21 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:boot 2024-03-08T23:04:22.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:21 smithi046 ceph-mon[31859]: fsmap 9 up:standby 2024-03-08T23:04:22.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:21 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:04:22.166 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:21 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3511296788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:21 smithi192 ceph-mon[36700]: Deploying daemon mds.g on smithi046 2024-03-08T23:04:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:21 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:boot 2024-03-08T23:04:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:21 smithi192 ceph-mon[36700]: fsmap 9 up:standby 2024-03-08T23:04:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:21 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:04:22.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:21 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3511296788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:21 smithi062 ceph-mon[35761]: Deploying daemon mds.g on smithi046 2024-03-08T23:04:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:21 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:boot 2024-03-08T23:04:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:21 smithi062 ceph-mon[35761]: fsmap 9 up:standby 2024-03-08T23:04:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:21 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:04:22.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:21 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3511296788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:22.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:22 smithi046 ceph-mon[31859]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:23.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:22 smithi192 ceph-mon[36700]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:23.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:22 smithi062 ceph-mon[35761]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:23.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.065+0000 7f6083400640 1 -- 172.21.15.46:0/2419654392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 msgr2=0x7f607c104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.065+0000 7f6083400640 1 --2- 172.21.15.46:0/2419654392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c104ba0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6068009860 tx=0x7f606802bb70 comp rx=0 tx=0).stop 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 -- 172.21.15.46:0/2419654392 shutdown_connections 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 --2- 172.21.15.46:0/2419654392 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 0x7f607c10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 --2- 172.21.15.46:0/2419654392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c104ba0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 --2- 172.21.15.46:0/2419654392 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 -- 172.21.15.46:0/2419654392 >> 172.21.15.46:0/2419654392 conn(0x7f607c0feae0 msgr2=0x7f607c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:24.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 -- 172.21.15.46:0/2419654392 shutdown_connections 2024-03-08T23:04:24.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.066+0000 7f6083400640 1 -- 172.21.15.46:0/2419654392 wait complete. 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 Processor -- start 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 -- start start 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 0x7f607c1a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c10f1c0 con 0x7f607c103530 2024-03-08T23:04:24.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c10f040 con 0x7f607c1050e0 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.067+0000 7f6083400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c10f340 con 0x7f607c104720 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35950/0 (socket says 172.21.15.46:35950) 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 -- 172.21.15.46:0/2006083236 learned_addr learned my addr 172.21.15.46:0/2006083236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f607be00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6081400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 0x7f607c1a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 -- 172.21.15.46:0/2006083236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 msgr2=0x7f607c199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 -- 172.21.15.46:0/2006083236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 msgr2=0x7f607c1a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 0x7f607c1a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 -- 172.21.15.46:0/2006083236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f607c1a1460 con 0x7f607c103530 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f607be00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:24.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.068+0000 7f6080a00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f607000e9a0 tx=0x7f607000ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.069+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607000cd80 con 0x7f607c103530 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.069+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c1a1750 con 0x7f607c103530 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.069+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6070010910 con 0x7f607c103530 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.069+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c1a1c70 con 0x7f607c103530 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.069+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6070018a00 con 0x7f607c103530 2024-03-08T23:04:24.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.070+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c10a850 con 0x7f607c103530 2024-03-08T23:04:24.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.072+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6070010ac0 con 0x7f607c103530 2024-03-08T23:04:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.073+0000 7f6079600640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 0x7f60640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:24.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.074+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f60700ce950 con 0x7f607c103530 2024-03-08T23:04:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.074+0000 7f607be00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 0x7f60640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:24.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.075+0000 7f607be00640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 0x7f60640a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6068009830 tx=0x7f606802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:24.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.079+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6070053260 con 0x7f607c103530 2024-03-08T23:04:24.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.329+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f607c02ce70 con 0x7f607c103530 2024-03-08T23:04:24.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.331+0000 7f6079600640 1 -- 172.21.15.46:0/2006083236 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7655 (secure 0 0 0) 0x7f607009bdf0 con 0x7f607c103530 2024-03-08T23:04:24.336 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:24.336 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-08T23:04:24.336 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 10 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 msgr2=0x7f60640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 0x7f60640a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6068009830 tx=0x7f606802e040 comp rx=0 tx=0).stop 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 msgr2=0x7f607c199840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f607000e9a0 tx=0x7f607000ee60 comp rx=0 tx=0).stop 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 shutdown_connections 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60640a2800 0x7f60640a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f607c1050e0 0x7f607c1a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f607c104720 0x7f607c199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 --2- 172.21.15.46:0/2006083236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f607c103530 0x7f607c199840 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:24.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 >> 172.21.15.46:0/2006083236 conn(0x7f607c0feae0 msgr2=0x7f607c107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:24.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 shutdown_connections 2024-03-08T23:04:24.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:24.334+0000 7f6083400640 1 -- 172.21.15.46:0/2006083236 wait complete. 2024-03-08T23:04:24.955 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:04:24 smithi046 systemd[1]: Started Ceph mds.g for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2006083236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:25.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2006083236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:25.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:25.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:25.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2006083236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:25.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:26.161 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:25 smithi046 ceph-mon[31859]: Deploying daemon mds.a on smithi046 2024-03-08T23:04:26.161 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:25 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6838/2215569681,v1:172.21.15.46:6839/2215569681] up:boot 2024-03-08T23:04:26.161 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:25 smithi046 ceph-mon[31859]: fsmap 10 up:standby 2024-03-08T23:04:26.161 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:25 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:04:26.307 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:25 smithi192 ceph-mon[36700]: Deploying daemon mds.a on smithi046 2024-03-08T23:04:26.308 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:25 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6838/2215569681,v1:172.21.15.46:6839/2215569681] up:boot 2024-03-08T23:04:26.308 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:25 smithi192 ceph-mon[36700]: fsmap 10 up:standby 2024-03-08T23:04:26.308 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:25 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:04:26.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:26.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:25 smithi062 ceph-mon[35761]: Deploying daemon mds.a on smithi046 2024-03-08T23:04:26.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:25 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6838/2215569681,v1:172.21.15.46:6839/2215569681] up:boot 2024-03-08T23:04:26.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:25 smithi062 ceph-mon[35761]: fsmap 10 up:standby 2024-03-08T23:04:26.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:25 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:04:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 -- 172.21.15.46:0/2063032049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94103340 msgr2=0x7f8c94103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 --2- 172.21.15.46:0/2063032049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94103340 0x7f8c94103760 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8c84009900 tx=0x7f8c8402bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 -- 172.21.15.46:0/2063032049 shutdown_connections 2024-03-08T23:04:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 --2- 172.21.15.46:0/2063032049 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94104ef0 0x7f8c9410b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 --2- 172.21.15.46:0/2063032049 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104530 0x7f8c941049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 --2- 172.21.15.46:0/2063032049 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94103340 0x7f8c94103760 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.588+0000 7f8c93200640 1 -- 172.21.15.46:0/2063032049 >> 172.21.15.46:0/2063032049 conn(0x7f8c940fe8d0 msgr2=0x7f8c94100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.589+0000 7f8c93200640 1 -- 172.21.15.46:0/2063032049 shutdown_connections 2024-03-08T23:04:26.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.589+0000 7f8c93200640 1 -- 172.21.15.46:0/2063032049 wait complete. 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 Processor -- start 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 -- start start 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 0x7f8c94199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 0x7f8c94199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 0x7f8c941a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:26.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9410f020 con 0x7f8c94103340 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9410eea0 con 0x7f8c94104ef0 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c93200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c9410f1a0 con 0x7f8c94104530 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c91e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 0x7f8c94199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c91e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 0x7f8c94199670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35978/0 (socket says 172.21.15.46:35978) 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c91e00640 1 -- 172.21.15.46:0/1084157263 learned_addr learned my addr 172.21.15.46:0/1084157263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c8be00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 0x7f8c94199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.590+0000 7f8c92800640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 0x7f8c941a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 -- 172.21.15.46:0/1084157263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 msgr2=0x7f8c941a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 0x7f8c941a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 -- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 msgr2=0x7f8c94199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 0x7f8c94199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 -- 172.21.15.46:0/1084157263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c941a1290 con 0x7f8c94104530 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c92800640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 0x7f8c941a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:04:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8be00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 0x7f8c94199bb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00e9f0 tx=0x7f8c7c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c00ccc0 con 0x7f8c94104530 2024-03-08T23:04:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8c7c005020 con 0x7f8c94104530 2024-03-08T23:04:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c941a1580 con 0x7f8c94104530 2024-03-08T23:04:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c7c0049e0 con 0x7f8c94104530 2024-03-08T23:04:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.591+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c941a1ad0 con 0x7f8c94104530 2024-03-08T23:04:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.592+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9410a660 con 0x7f8c94104530 2024-03-08T23:04:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.594+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8c7c005560 con 0x7f8c94104530 2024-03-08T23:04:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.596+0000 7f8c8aa00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 0x7f8c780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.596+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8c7c0ce5f0 con 0x7f8c94104530 2024-03-08T23:04:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.597+0000 7f8c91e00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 0x7f8c780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.597+0000 7f8c91e00640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 0x7f8c780a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c84006010 tx=0x7f8c8402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.598+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8c7c0051d0 con 0x7f8c94104530 2024-03-08T23:04:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.844+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c9402d0c0 con 0x7f8c94104530 2024-03-08T23:04:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.845+0000 7f8c8aa00640 1 -- 172.21.15.46:0/1084157263 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8447 (secure 0 0 0) 0x7f8c7c09ba90 con 0x7f8c94104530 2024-03-08T23:04:26.850 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:26.850 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-08T23:04:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 11 2024-03-08T23:04:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.848+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 msgr2=0x7f8c780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.848+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 0x7f8c780a4d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c84006010 tx=0x7f8c8402df90 comp rx=0 tx=0).stop 2024-03-08T23:04:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.848+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 msgr2=0x7f8c94199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.848+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 0x7f8c94199bb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8c7c00e9f0 tx=0x7f8c7c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.848+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 shutdown_connections 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8c780a2880 0x7f8c780a4d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8c94104ef0 0x7f8c941a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c94104530 0x7f8c94199bb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 --2- 172.21.15.46:0/1084157263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8c94103340 0x7f8c94199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 >> 172.21.15.46:0/1084157263 conn(0x7f8c940fe8d0 msgr2=0x7f8c94107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 shutdown_connections 2024-03-08T23:04:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:26.849+0000 7f8c93200640 1 -- 172.21.15.46:0/1084157263 wait complete. 2024-03-08T23:04:27.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:26 smithi046 ceph-mon[31859]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:27.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:26 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1084157263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:26 smithi192 ceph-mon[36700]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:26 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1084157263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:26 smithi062 ceph-mon[35761]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:27.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:26 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1084157263' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:28.476 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 systemd[1]: Starting Ceph mds.a for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:28.757 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 podman[70657]: 2024-03-08 23:04:28.379292273 +0000 UTC m=+0.033645123 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:28.757 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 podman[70657]: 2024-03-08 23:04:28.498448801 +0000 UTC m=+0.152801681 container create ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-08T23:04:28.757 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 podman[70657]: 2024-03-08 23:04:28.752951518 +0000 UTC m=+0.407304404 container init ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-08T23:04:28.875 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.083+0000 7fedd6e00640 1 -- 172.21.15.46:0/3883902685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0076ef0 msgr2=0x7fedd0077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.083+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3883902685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0076ef0 0x7fedd0077350 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fedbc009900 tx=0x7fedbc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 -- 172.21.15.46:0/3883902685 shutdown_connections 2024-03-08T23:04:29.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3883902685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0077890 0x7fedd0078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3883902685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0076ef0 0x7fedd0077350 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3883902685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd00760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 -- 172.21.15.46:0/3883902685 >> 172.21.15.46:0/3883902685 conn(0x7fedd00feaa0 msgr2=0x7fedd0100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 -- 172.21.15.46:0/3883902685 shutdown_connections 2024-03-08T23:04:29.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.084+0000 7fedd6e00640 1 -- 172.21.15.46:0/3883902685 wait complete. 2024-03-08T23:04:29.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 Processor -- start 2024-03-08T23:04:29.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 -- start start 2024-03-08T23:04:29.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd019dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:29.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0076ef0 0x7fedd019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd0113600 con 0x7fedd0076ef0 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd0113480 con 0x7fedd0077890 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd6e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd0113780 con 0x7fedd0075cb0 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.085+0000 7fedd4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedcfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd019dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:59994/0 (socket says 172.21.15.46:59994) 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedcfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd019dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:47894/0 (socket says 172.21.15.46:47894) 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 -- 172.21.15.46:0/3545835571 learned_addr learned my addr 172.21.15.46:0/3545835571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedcf400640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0076ef0 0x7fedd019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 -- 172.21.15.46:0/3545835571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 msgr2=0x7fedd019dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd019dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 -- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0076ef0 msgr2=0x7fedd019e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:29.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0076ef0 0x7fedd019e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 -- 172.21.15.46:0/3545835571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd01a5830 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.086+0000 7fedd4e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fedc800eaa0 tx=0x7fedc800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.087+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc800cdb0 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.087+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedc8005020 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.087+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedc80116b0 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.087+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd01a5b20 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.087+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd01a6020 con 0x7fedd0077890 2024-03-08T23:04:29.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.088+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd010fff0 con 0x7fedd0077890 2024-03-08T23:04:29.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.091+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fedc8005560 con 0x7fedd0077890 2024-03-08T23:04:29.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.092+0000 7fedccc00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 0x7fedb40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:29.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.092+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fedc80ce830 con 0x7fedd0077890 2024-03-08T23:04:29.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.092+0000 7fedcfe00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 0x7fedb40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:29.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.093+0000 7fedcfe00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 0x7fedb40a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fedd019f0f0 tx=0x7fedc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:29.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.098+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fedc8016030 con 0x7fedd0077890 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 podman[70657]: 2024-03-08 23:04:28.766082373 +0000 UTC m=+0.420435273 container start ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 ceph-mds[70677]: -- 172.21.15.46:0/3775717379 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffe3497a40 con 0x55ffe346ac00 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: starting mds.a at 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 bash[70657]: ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:04:28 smithi046 systemd[1]: Started Ceph mds.a for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:29.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:28 smithi046 ceph-mon[31859]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:29.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:29.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:28 smithi192 ceph-mon[36700]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:29.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:29.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.347+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedd002ce70 con 0x7fedd0077890 2024-03-08T23:04:29.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.349+0000 7fedccc00640 1 -- 172.21.15.46:0/3545835571 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9239 (secure 0 0 0) 0x7fedc809b680 con 0x7fedd0077890 2024-03-08T23:04:29.353 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:29.354 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-08T23:04:29.354 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2024-03-08T23:04:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.351+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 msgr2=0x7fedb40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 0x7fedb40a4d40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fedd019f0f0 tx=0x7fedc0005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 msgr2=0x7fedd01a5190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fedc800eaa0 tx=0x7fedc800ef60 comp rx=0 tx=0).stop 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 shutdown_connections 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fedb40a2880 0x7fedb40a4d40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedd0077890 0x7fedd01a5190 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedd0076ef0 0x7fedd019e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 --2- 172.21.15.46:0/3545835571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedd0075cb0 0x7fedd019dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 >> 172.21.15.46:0/3545835571 conn(0x7fedd00feaa0 msgr2=0x7fedd0100130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 shutdown_connections 2024-03-08T23:04:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:29.352+0000 7fedd6e00640 1 -- 172.21.15.46:0/3545835571 wait complete. 2024-03-08T23:04:29.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:28 smithi062 ceph-mon[35761]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:29.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:boot 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: fsmap 11 up:standby 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:30.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: Deploying daemon mds.b on smithi062 2024-03-08T23:04:30.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:29 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3545835571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:boot 2024-03-08T23:04:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: fsmap 11 up:standby 2024-03-08T23:04:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:04:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:30.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:30.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:30.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: Deploying daemon mds.b on smithi062 2024-03-08T23:04:30.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:29 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3545835571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:30.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:boot 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: fsmap 11 up:standby 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: Deploying daemon mds.b on smithi062 2024-03-08T23:04:30.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3545835571' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:31.260 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:31 smithi062 ceph-mon[35761]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:31.260 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 systemd[1]: Starting Ceph mds.b for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:04:31.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:31 smithi192 ceph-mon[36700]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:31.377 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:31.471 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:31 smithi046 ceph-mon[31859]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 -- 172.21.15.46:0/3116508227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102340 msgr2=0x7f927c1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 --2- 172.21.15.46:0/3116508227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102340 0x7f927c1027c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9268009900 tx=0x7f926802bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:31.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 -- 172.21.15.46:0/3116508227 shutdown_connections 2024-03-08T23:04:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 --2- 172.21.15.46:0/3116508227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102d00 0x7f927c109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 --2- 172.21.15.46:0/3116508227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102340 0x7f927c1027c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 --2- 172.21.15.46:0/3116508227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 -- 172.21.15.46:0/3116508227 >> 172.21.15.46:0/3116508227 conn(0x7f927c0fc6c0 msgr2=0x7f927c0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:31.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 -- 172.21.15.46:0/3116508227 shutdown_connections 2024-03-08T23:04:31.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.608+0000 7f9282c00640 1 -- 172.21.15.46:0/3116508227 wait complete. 2024-03-08T23:04:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 Processor -- start 2024-03-08T23:04:31.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 -- start start 2024-03-08T23:04:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102340 0x7f927c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102d00 0x7f927c1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c10ce30 con 0x7f927c101150 2024-03-08T23:04:31.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.610+0000 7f9282c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c10ccb0 con 0x7f927c102d00 2024-03-08T23:04:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9282c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927c10cfb0 con 0x7f927c102340 2024-03-08T23:04:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36022/0 (socket says 172.21.15.46:36022) 2024-03-08T23:04:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 -- 172.21.15.46:0/3691504610 learned_addr learned my addr 172.21.15.46:0/3691504610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:31.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9280e00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102340 0x7f927c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9282200640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102d00 0x7f927c1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 -- 172.21.15.46:0/3691504610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102340 msgr2=0x7f927c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102340 0x7f927c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 -- 172.21.15.46:0/3691504610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102d00 msgr2=0x7f927c1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:31.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102d00 0x7f927c1a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 -- 172.21.15.46:0/3691504610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927c1a1250 con 0x7f927c101150 2024-03-08T23:04:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.611+0000 7f9281800640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f927000e9c0 tx=0x7f927000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.612+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f927000cd80 con 0x7f927c101150 2024-03-08T23:04:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.612+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92700035f0 con 0x7f927c101150 2024-03-08T23:04:31.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.612+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927c1a1540 con 0x7f927c101150 2024-03-08T23:04:31.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.612+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9270010690 con 0x7f927c101150 2024-03-08T23:04:31.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.612+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927c1a1a60 con 0x7f927c101150 2024-03-08T23:04:31.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.613+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927c1a1d20 con 0x7f927c101150 2024-03-08T23:04:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.616+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f92700037a0 con 0x7f927c101150 2024-03-08T23:04:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.617+0000 7f927a000640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 0x7f92640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:31.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.617+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92700ceba0 con 0x7f927c101150 2024-03-08T23:04:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.618+0000 7f9280e00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 0x7f92640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.618+0000 7f9280e00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 0x7f92640a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f926802c240 tx=0x7f926802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:31.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.620+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f927009c0d0 con 0x7f927c101150 2024-03-08T23:04:31.633 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 podman[65801]: 2024-03-08 23:04:31.37897322 +0000 UTC m=+0.018111800 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:04:31.633 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 podman[65801]: 2024-03-08 23:04:31.494835102 +0000 UTC m=+0.133973714 container create b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-08T23:04:31.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.865+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f927c1a1ef0 con 0x7f927c101150 2024-03-08T23:04:31.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.867+0000 7f927a000640 1 -- 172.21.15.46:0/3691504610 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9239 (secure 0 0 0) 0x7f927009b9d0 con 0x7f927c101150 2024-03-08T23:04:31.872 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:31.872 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[]} 2024-03-08T23:04:31.872 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 12 2024-03-08T23:04:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 msgr2=0x7f92640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 0x7f92640a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f926802c240 tx=0x7f926802df90 comp rx=0 tx=0).stop 2024-03-08T23:04:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 msgr2=0x7f927c199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:31.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f927000e9c0 tx=0x7f927000ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 shutdown_connections 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92640a2800 0x7f92640a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f927c102d00 0x7f927c1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f927c102340 0x7f927c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 --2- 172.21.15.46:0/3691504610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f927c101150 0x7f927c199630 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:31.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 >> 172.21.15.46:0/3691504610 conn(0x7f927c0fc6c0 msgr2=0x7f927c073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:31.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.870+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 shutdown_connections 2024-03-08T23:04:31.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:31.871+0000 7f9282c00640 1 -- 172.21.15.46:0/3691504610 wait complete. 2024-03-08T23:04:31.892 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 podman[65801]: 2024-03-08 23:04:31.772644705 +0000 UTC m=+0.411783287 container init b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-08T23:04:31.892 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 podman[65801]: 2024-03-08 23:04:31.775488623 +0000 UTC m=+0.414627185 container start b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-08T23:04:31.892 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 ceph-mds[65820]: -- 172.21.15.62:0/1783224949 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561c77ae9a40 con 0x561c77abd000 2024-03-08T23:04:31.892 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: starting mds.b at 2024-03-08T23:04:32.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:32 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3691504610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:32.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:32 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3691504610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:32.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:32 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3691504610' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:32.392 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:31 smithi062 bash[65801]: b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 2024-03-08T23:04:32.392 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:32 smithi062 systemd[1]: Started Ceph mds.b for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:boot 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: fsmap 12 up:standby 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:33 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:33.366 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:boot 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: fsmap 12 up:standby 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.367 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:33 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:33.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:boot 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: fsmap 12 up:standby 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:33.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:33 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:33.894 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 -- 172.21.15.46:0/1993340481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f40ff9b0 msgr2=0x7ff6f40ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 --2- 172.21.15.46:0/1993340481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f40ffdd0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009ca0 tx=0x7ff6e402c100 comp rx=0 tx=0).stop 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 -- 172.21.15.46:0/1993340481 shutdown_connections 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 --2- 172.21.15.46:0/1993340481 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f4100cb0 0x7ff6f4101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 --2- 172.21.15.46:0/1993340481 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 0x7ff6f4100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 --2- 172.21.15.46:0/1993340481 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f40ffdd0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 -- 172.21.15.46:0/1993340481 >> 172.21.15.46:0/1993340481 conn(0x7ff6f40fb5d0 msgr2=0x7ff6f40fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 -- 172.21.15.46:0/1993340481 shutdown_connections 2024-03-08T23:04:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.095+0000 7ff6f3200640 1 -- 172.21.15.46:0/1993340481 wait complete. 2024-03-08T23:04:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.096+0000 7ff6f3200640 1 Processor -- start 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.096+0000 7ff6f3200640 1 -- start start 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 0x7ff6f4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 0x7ff6f41a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f410efe0 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f410ee60 con 0x7ff6f4100310 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f3200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f410f160 con 0x7ff6f4100cb0 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41120/0 (socket says 172.21.15.46:41120) 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 -- 172.21.15.46:0/469611980 learned_addr learned my addr 172.21.15.46:0/469611980 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1400640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 0x7ff6f4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f2800640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 0x7ff6f41a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 -- 172.21.15.46:0/469611980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 msgr2=0x7ff6f41a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 0x7ff6f41a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 -- 172.21.15.46:0/469611980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 msgr2=0x7ff6f4199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 0x7ff6f4199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f1e00640 1 -- 172.21.15.46:0/469611980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f41a11f0 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.097+0000 7ff6f2800640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 0x7ff6f41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.098+0000 7ff6f1e00640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40325f0 tx=0x7ff6e4035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.099+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4039040 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.099+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e4044070 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.099+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4004540 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.099+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f41a1480 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.099+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f41a18f0 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.101+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f410a700 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.103+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff6e40046e0 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.104+0000 7ff6eaa00640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 0x7ff6d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.104+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff6e40f1670 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.104+0000 7ff6f1400640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 0x7ff6d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.105+0000 7ff6f1400640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 0x7ff6d80a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0095f0 tx=0x7ff6dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.107+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff6e40beb10 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.364+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6f402ce70 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.365+0000 7ff6eaa00640 1 -- 172.21.15.46:0/469611980 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10031 (secure 0 0 0) 0x7ff6e40be410 con 0x7ff6f40ff9b0 2024-03-08T23:04:34.370 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:34.371 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24578,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-08T23:04:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 13 2024-03-08T23:04:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 msgr2=0x7ff6d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 0x7ff6d80a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0095f0 tx=0x7ff6dc005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 msgr2=0x7ff6f41995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40325f0 tx=0x7ff6e4035d10 comp rx=0 tx=0).stop 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 shutdown_connections 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d80a2880 0x7ff6d80a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f4100cb0 0x7ff6f41a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f4100310 0x7ff6f4199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 --2- 172.21.15.46:0/469611980 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f40ff9b0 0x7ff6f41995d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.369+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 >> 172.21.15.46:0/469611980 conn(0x7ff6f40fb5d0 msgr2=0x7ff6f40fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.370+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 shutdown_connections 2024-03-08T23:04:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.370+0000 7ff6f3200640 1 -- 172.21.15.46:0/469611980 wait complete. 2024-03-08T23:04:34.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:34.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 -- 172.21.15.46:0/116459235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41050e0 msgr2=0x7f2ca410b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/116459235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41050e0 0x7f2ca410b970 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009ca0 tx=0x7f2c9c02c100 comp rx=0 tx=0).stop 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 -- 172.21.15.46:0/116459235 shutdown_connections 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/116459235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca41050e0 0x7f2ca410b970 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/116459235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 0x7f2ca4104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/116459235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca4103530 0x7f2ca4103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 -- 172.21.15.46:0/116459235 >> 172.21.15.46:0/116459235 conn(0x7f2ca40feaa0 msgr2=0x7f2ca4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 -- 172.21.15.46:0/116459235 shutdown_connections 2024-03-08T23:04:34.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.590+0000 7f2ca9a00640 1 -- 172.21.15.46:0/116459235 wait complete. 2024-03-08T23:04:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.591+0000 7f2ca9a00640 1 Processor -- start 2024-03-08T23:04:34.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 -- start start 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 0x7f2ca4199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca41050e0 0x7f2ca41a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca410f1c0 con 0x7f2ca4104720 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca410f040 con 0x7f2ca4103530 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca9a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca410f340 con 0x7f2ca41050e0 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:59840/0 (socket says 172.21.15.46:59840) 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca2a00640 1 -- 172.21.15.46:0/2638632217 learned_addr learned my addr 172.21.15.46:0/2638632217 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca2000640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 0x7f2ca4199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.592+0000 7f2ca3400640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca41050e0 0x7f2ca41a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 -- 172.21.15.46:0/2638632217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca41050e0 msgr2=0x7f2ca41a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca41050e0 0x7f2ca41a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 -- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 msgr2=0x7f2ca4199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 0x7f2ca4199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 -- 172.21.15.46:0/2638632217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca41a13b0 con 0x7f2ca4103530 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca2a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400d950 tx=0x7f2c9400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94017020 con 0x7f2ca4103530 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c94009c30 con 0x7f2ca4103530 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca41a16a0 con 0x7f2ca4103530 2024-03-08T23:04:34.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c9400f720 con 0x7f2ca4103530 2024-03-08T23:04:34.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.593+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca41a1c40 con 0x7f2ca4103530 2024-03-08T23:04:34.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.596+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca410a8a0 con 0x7f2ca4103530 2024-03-08T23:04:34.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.597+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2c9401a020 con 0x7f2ca4103530 2024-03-08T23:04:34.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.598+0000 7f2c9b400640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 0x7f2c880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:34.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.598+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2c940ce4a0 con 0x7f2ca4103530 2024-03-08T23:04:34.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.598+0000 7f2ca2000640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 0x7f2c880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:34.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.599+0000 7f2ca2000640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 0x7f2c880a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ca419aca0 tx=0x7f2c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.602+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2c94021020 con 0x7f2ca4103530 2024-03-08T23:04:34.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.829+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ca4061f20 con 0x7f2ca4103530 2024-03-08T23:04:34.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.830+0000 7f2c9b400640 1 -- 172.21.15.46:0/2638632217 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18432 (secure 0 0 0) 0x7f2c94009de0 con 0x7f2ca4103530 2024-03-08T23:04:34.835 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:34.835 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":81,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:03:41.272757+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.832+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 msgr2=0x7f2c880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.832+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 0x7f2c880a4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2ca419aca0 tx=0x7f2c90005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.832+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 msgr2=0x7f2ca4199790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.832+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2c9400d950 tx=0x7f2c9400de10 comp rx=0 tx=0).stop 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 shutdown_connections 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c880a2880 0x7f2c880a4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2ca41050e0 0x7f2ca41a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2ca4104720 0x7f2ca4199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 --2- 172.21.15.46:0/2638632217 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2ca4103530 0x7f2ca4199790 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 >> 172.21.15.46:0/2638632217 conn(0x7f2ca40feaa0 msgr2=0x7f2ca4107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 shutdown_connections 2024-03-08T23:04:34.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:34.833+0000 7f2ca9a00640 1 -- 172.21.15.46:0/2638632217 wait complete. 2024-03-08T23:04:34.857 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-08T23:04:34.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:04:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.059+0000 7f0f2b200640 1 -- 172.21.15.46:0/3833563152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c1011f0 msgr2=0x7f0f2c101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.059+0000 7f0f2b200640 1 --2- 172.21.15.46:0/3833563152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c101610 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c009900 tx=0x7f0f1c02bc50 comp rx=0 tx=0).stop 2024-03-08T23:04:35.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 -- 172.21.15.46:0/3833563152 shutdown_connections 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 --2- 172.21.15.46:0/3833563152 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c102da0 0x7f0f2c109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 --2- 172.21.15.46:0/3833563152 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 --2- 172.21.15.46:0/3833563152 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c101610 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 -- 172.21.15.46:0/3833563152 >> 172.21.15.46:0/3833563152 conn(0x7f0f2c0fc760 msgr2=0x7f0f2c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 -- 172.21.15.46:0/3833563152 shutdown_connections 2024-03-08T23:04:35.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.060+0000 7f0f2b200640 1 -- 172.21.15.46:0/3833563152 wait complete. 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 Processor -- start 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 -- start start 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c102da0 0x7f0f2c1a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c10cea0 con 0x7f0f2c1023e0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c10cd20 con 0x7f0f2c1011f0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.061+0000 7f0f2b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c10d020 con 0x7f0f2c102da0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41162/0 (socket says 172.21.15.46:41162) 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 -- 172.21.15.46:0/2337182067 learned_addr learned my addr 172.21.15.46:0/2337182067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29e00640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f2a800640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c102da0 0x7f0f2c1a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 -- 172.21.15.46:0/2337182067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c102da0 msgr2=0x7f0f2c1a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c102da0 0x7f0f2c1a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 -- 172.21.15.46:0/2337182067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 msgr2=0x7f0f2c199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 -- 172.21.15.46:0/2337182067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c1a1380 con 0x7f0f2c1023e0 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29e00640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.062+0000 7f0f29400640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800e9f0 tx=0x7f0f1800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.063+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f1800cd80 con 0x7f0f2c1023e0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.063+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f18010910 con 0x7f0f2c1023e0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.063+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c1a1670 con 0x7f0f2c1023e0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.063+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f18018990 con 0x7f0f2c1023e0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.063+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c1a1c10 con 0x7f0f2c1023e0 2024-03-08T23:04:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.064+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c1a1e70 con 0x7f0f2c1023e0 2024-03-08T23:04:35.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.066+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0f18010ac0 con 0x7f0f2c1023e0 2024-03-08T23:04:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.068+0000 7f0f22a00640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 0x7f0f100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.068+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0f180ce910 con 0x7f0f2c1023e0 2024-03-08T23:04:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.069+0000 7f0f29e00640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 0x7f0f100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.069+0000 7f0f29e00640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 0x7f0f100a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c02c190 tx=0x7f0f1c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:35.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.072+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0f1809b760 con 0x7f0f2c1023e0 2024-03-08T23:04:35.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:35.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/469611980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:35 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2638632217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/469611980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:35 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2638632217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.294+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f2c1a2040 con 0x7f0f2c1023e0 2024-03-08T23:04:35.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.294+0000 7f0f22a00640 1 -- 172.21.15.46:0/2337182067 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f180bb020 con 0x7f0f2c1023e0 2024-03-08T23:04:35.299 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:04:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 msgr2=0x7f0f100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 0x7f0f100a4d40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c02c190 tx=0x7f0f1c000bc0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 msgr2=0x7f0f2c199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f1800e9f0 tx=0x7f0f1800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 shutdown_connections 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0f100a2880 0x7f0f100a4d40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f2c102da0 0x7f0f2c1a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f2c1023e0 0x7f0f2c199ca0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 --2- 172.21.15.46:0/2337182067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f2c1011f0 0x7f0f2c199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 >> 172.21.15.46:0/2337182067 conn(0x7f0f2c0fc760 msgr2=0x7f0f2c073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 shutdown_connections 2024-03-08T23:04:35.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.297+0000 7f0f2b200640 1 -- 172.21.15.46:0/2337182067 wait complete. 2024-03-08T23:04:35.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/469611980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2638632217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:35.319 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 -- 172.21.15.46:0/2320879212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028101300 msgr2=0x7fa028101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 --2- 172.21.15.46:0/2320879212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028101300 0x7fa028101720 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa01c009900 tx=0x7fa01c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 -- 172.21.15.46:0/2320879212 shutdown_connections 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 --2- 172.21.15.46:0/2320879212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 --2- 172.21.15.46:0/2320879212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0281024f0 0x7fa028102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 --2- 172.21.15.46:0/2320879212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa028101300 0x7fa028101720 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 -- 172.21.15.46:0/2320879212 >> 172.21.15.46:0/2320879212 conn(0x7fa0280fc8d0 msgr2=0x7fa0280fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:35.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 -- 172.21.15.46:0/2320879212 shutdown_connections 2024-03-08T23:04:35.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.525+0000 7fa030600640 1 -- 172.21.15.46:0/2320879212 wait complete. 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.526+0000 7fa030600640 1 Processor -- start 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.526+0000 7fa030600640 1 -- start start 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 0x7fa0280750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02810cf90 con 0x7fa028101300 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02810ce10 con 0x7fa0281024f0 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa030600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02810d110 con 0x7fa028102eb0 2024-03-08T23:04:35.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 0x7fa0280750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02e600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:59872/0 (socket says 172.21.15.46:59872) 2024-03-08T23:04:35.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 0x7fa0280750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41186/0 (socket says 172.21.15.46:41186) 2024-03-08T23:04:35.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.527+0000 7fa02d200640 1 -- 172.21.15.46:0/1056989014 learned_addr learned my addr 172.21.15.46:0/1056989014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 -- 172.21.15.46:0/1056989014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 msgr2=0x7fa028073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 -- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 msgr2=0x7fa0280750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 0x7fa0280750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 -- 172.21.15.46:0/1056989014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0280744b0 con 0x7fa0281024f0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02e600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa02d200640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa01400e9f0 tx=0x7fa01400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01400cd80 con 0x7fa0281024f0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa014010910 con 0x7fa0281024f0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa014018990 con 0x7fa0281024f0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0281a1730 con 0x7fa0281024f0 2024-03-08T23:04:35.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.528+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0281a1c30 con 0x7fa0281024f0 2024-03-08T23:04:35.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.530+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa028108620 con 0x7fa0281024f0 2024-03-08T23:04:35.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.532+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa014010ac0 con 0x7fa0281024f0 2024-03-08T23:04:35.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.533+0000 7fa026000640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 0x7fa0100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:35.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.533+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa0140cea90 con 0x7fa0281024f0 2024-03-08T23:04:35.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.533+0000 7fa02dc00640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 0x7fa0100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:35.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.534+0000 7fa02dc00640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 0x7fa0100a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa01c000c00 tx=0x7fa01c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:35.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.535+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa01409b980 con 0x7fa0281024f0 2024-03-08T23:04:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.746+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa02802d0c0 con 0x7fa0281024f0 2024-03-08T23:04:35.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.746+0000 7fa026000640 1 -- 172.21.15.46:0/1056989014 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa01409fa20 con 0x7fa0281024f0 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout:[ 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout: "poolnum": 1, 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout: "poolname": ".mgr" 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:04:35.751 INFO:teuthology.orchestra.run.smithi046.stdout:] 2024-03-08T23:04:35.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 msgr2=0x7fa0100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 0x7fa0100a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa01c000c00 tx=0x7fa01c005fd0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 msgr2=0x7fa0280736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa01400e9f0 tx=0x7fa01400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 shutdown_connections 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0100a2800 0x7fa0100a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa028102eb0 0x7fa028073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0281024f0 0x7fa0280736e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 --2- 172.21.15.46:0/1056989014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa028101300 0x7fa0280750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 >> 172.21.15.46:0/1056989014 conn(0x7fa0280fc8d0 msgr2=0x7fa0281057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 shutdown_connections 2024-03-08T23:04:35.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.750+0000 7fa030600640 1 -- 172.21.15.46:0/1056989014 wait complete. 2024-03-08T23:04:35.775 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-08T23:04:35.775 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-08T23:04:35.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 -- 172.21.15.46:0/3617860583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 msgr2=0x7fb5dc104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 --2- 172.21.15.46:0/3617860583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc104040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc009900 tx=0x7fb5cc02bc40 comp rx=0 tx=0).stop 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 -- 172.21.15.46:0/3617860583 shutdown_connections 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 --2- 172.21.15.46:0/3617860583 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 --2- 172.21.15.46:0/3617860583 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc104580 0x7fb5dc106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 --2- 172.21.15.46:0/3617860583 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc104040 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 -- 172.21.15.46:0/3617860583 >> 172.21.15.46:0/3617860583 conn(0x7fb5dc0fb660 msgr2=0x7fb5dc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 -- 172.21.15.46:0/3617860583 shutdown_connections 2024-03-08T23:04:35.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.993+0000 7fb5e1600640 1 -- 172.21.15.46:0/3617860583 wait complete. 2024-03-08T23:04:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.994+0000 7fb5e1600640 1 Processor -- start 2024-03-08T23:04:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.994+0000 7fb5e1600640 1 -- start start 2024-03-08T23:04:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.994+0000 7fb5e1600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:36.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.994+0000 7fb5e1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc104580 0x7fb5dc199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e1600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e1600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc10ce60 con 0x7fb5dc104580 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e1600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc10cce0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e1600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5dc10cfe0 con 0x7fb5dc101c50 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5d3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc104580 0x7fb5dc199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5dbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:41874/0 (socket says 172.21.15.46:41874) 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5dbe00640 1 -- 172.21.15.46:0/349539925 learned_addr learned my addr 172.21.15.46:0/349539925 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 -- 172.21.15.46:0/349539925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 msgr2=0x7fb5dc1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc1996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 -- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc104580 msgr2=0x7fb5dc199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc104580 0x7fb5dc199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:36.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.995+0000 7fb5e0c00640 1 -- 172.21.15.46:0/349539925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5dc1a12f0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5e0c00640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc1a0c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009fd0 tx=0x7fb5d400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d400ce20 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5dc1a15e0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5d4005020 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4010720 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.996+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5dc1a1ae0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.997+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5dc108a00 con 0x7fb5dc106eb0 2024-03-08T23:04:36.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:35.999+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5d4005540 con 0x7fb5dc106eb0 2024-03-08T23:04:36.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.000+0000 7fb5d9600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 0x7fb5c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:36.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.000+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb5d40ced40 con 0x7fb5dc106eb0 2024-03-08T23:04:36.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.001+0000 7fb5dbe00640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 0x7fb5c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:36.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.001+0000 7fb5dbe00640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 0x7fb5c00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc000c00 tx=0x7fb5cc02c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:36.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.004+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb5d409c1e0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.202 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:36.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:36.219+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb5dc061ef0 con 0x7fb5dc106eb0 2024-03-08T23:04:36.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2337182067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:36.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:36 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1056989014' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-08T23:04:36.480 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:36.480 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.480 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.480 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:36.480 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:36.481 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.481 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2337182067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:36.481 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:36.481 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:36 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1056989014' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2337182067' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:04:36.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:36 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1056989014' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-08T23:04:37.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.220+0000 7fb5d9600640 1 -- 172.21.15.46:0/349539925 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb5d409bae0 con 0x7fb5dc106eb0 2024-03-08T23:04:37.228 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' created 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.225+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 msgr2=0x7fb5c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.225+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 0x7fb5c00a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb5cc000c00 tx=0x7fb5cc02c890 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.225+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 msgr2=0x7fb5dc1a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.225+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc1a0c80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009fd0 tx=0x7fb5d400ef10 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 shutdown_connections 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb5c00a2800 0x7fb5c00a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb5dc106eb0 0x7fb5dc1a0c80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb5dc104580 0x7fb5dc199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 --2- 172.21.15.46:0/349539925 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb5dc101c50 0x7fb5dc1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.227+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 >> 172.21.15.46:0/349539925 conn(0x7fb5dc0fb660 msgr2=0x7fb5dc101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.228+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 shutdown_connections 2024-03-08T23:04:37.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.228+0000 7fb5e1600640 1 -- 172.21.15.46:0/349539925 wait complete. 2024-03-08T23:04:37.260 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-08T23:04:37.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.467+0000 7f4b6f200640 1 -- 172.21.15.46:0/4129640091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b700752d0 msgr2=0x7f4b70073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.467+0000 7f4b6f200640 1 --2- 172.21.15.46:0/4129640091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b700752d0 0x7f4b70073720 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009900 tx=0x7f4b6002bcd0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 -- 172.21.15.46:0/4129640091 shutdown_connections 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 --2- 172.21.15.46:0/4129640091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70109080 0x7f4b7010b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 --2- 172.21.15.46:0/4129640091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70073cf0 0x7f4b70074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 --2- 172.21.15.46:0/4129640091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b700752d0 0x7f4b70073720 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 -- 172.21.15.46:0/4129640091 >> 172.21.15.46:0/4129640091 conn(0x7f4b700fc760 msgr2=0x7f4b700feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 -- 172.21.15.46:0/4129640091 shutdown_connections 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.468+0000 7f4b6f200640 1 -- 172.21.15.46:0/4129640091 wait complete. 2024-03-08T23:04:37.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 Processor -- start 2024-03-08T23:04:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 -- start start 2024-03-08T23:04:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70073cf0 0x7f4b70199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:37.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 0x7f4b70199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7010f0d0 con 0x7f4b70073cf0 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7010ef50 con 0x7f4b700752d0 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7010f250 con 0x7f4b70109080 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70073cf0 0x7f4b70199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 0x7f4b70199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:41896/0 (socket says 172.21.15.46:41896) 2024-03-08T23:04:37.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.469+0000 7f4b6e800640 1 -- 172.21.15.46:0/1046168241 learned_addr learned my addr 172.21.15.46:0/1046168241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 -- 172.21.15.46:0/1046168241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 msgr2=0x7f4b70199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 0x7f4b70199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 -- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70073cf0 msgr2=0x7f4b70199710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70073cf0 0x7f4b70199710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 -- 172.21.15.46:0/1046168241 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b701a1330 con 0x7f4b70109080 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6d400640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 0x7f4b70199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6e800640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800eb10 tx=0x7f4b6800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004b20 con 0x7f4b70109080 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b68004cc0 con 0x7f4b70109080 2024-03-08T23:04:37.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68010760 con 0x7f4b70109080 2024-03-08T23:04:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.470+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b701a1620 con 0x7f4b70109080 2024-03-08T23:04:37.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.471+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b701a1b20 con 0x7f4b70109080 2024-03-08T23:04:37.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:37 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/349539925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:37 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:37 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:37 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.472+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b70107e40 con 0x7f4b70109080 2024-03-08T23:04:37.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.473+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4b68005020 con 0x7f4b70109080 2024-03-08T23:04:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.475+0000 7f4b66a00640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 0x7f4b540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.475+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f4b680cecb0 con 0x7f4b70109080 2024-03-08T23:04:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.475+0000 7f4b6de00640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 0x7f4b540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:37.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.476+0000 7f4b6de00640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 0x7f4b540a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002c2e0 tx=0x7f4b6002c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:37.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.479+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4b6809bfe0 con 0x7f4b70109080 2024-03-08T23:04:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:37 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/349539925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:37 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:37 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:37 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:37 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/349539925' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:37 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-08T23:04:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:37 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:37 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:37.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:37.725+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f4b70061ef0 con 0x7f4b70109080 2024-03-08T23:04:38.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.313+0000 7f4b66a00640 1 -- 172.21.15.46:0/1046168241 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f4b6809b8e0 con 0x7f4b70109080 2024-03-08T23:04:38.318 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' created 2024-03-08T23:04:38.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.318+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 msgr2=0x7f4b540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:38.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.318+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 0x7f4b540a4d40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4b6002c2e0 tx=0x7f4b6002c920 comp rx=0 tx=0).stop 2024-03-08T23:04:38.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.318+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 msgr2=0x7f4b701a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:38.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.318+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800eb10 tx=0x7f4b6800c2d0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 shutdown_connections 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4b540a2880 0x7f4b540a4d40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70109080 0x7f4b701a0cc0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b700752d0 0x7f4b70199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 --2- 172.21.15.46:0/1046168241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70073cf0 0x7f4b70199710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.319+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 >> 172.21.15.46:0/1046168241 conn(0x7f4b700fc760 msgr2=0x7f4b70105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:38.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.320+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 shutdown_connections 2024-03-08T23:04:38.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.320+0000 7f4b6f200640 1 -- 172.21.15.46:0/1046168241 wait complete. 2024-03-08T23:04:38.357 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-08T23:04:38.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 -- 172.21.15.46:0/802872047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 msgr2=0x7f3280078130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:38.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 --2- 172.21.15.46:0/802872047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f3280078130 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3278009be0 tx=0x7f327802bef0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 -- 172.21.15.46:0/802872047 shutdown_connections 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 --2- 172.21.15.46:0/802872047 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f3280078130 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 --2- 172.21.15.46:0/802872047 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280076ef0 0x7f3280077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 --2- 172.21.15.46:0/802872047 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280075cb0 0x7f32800760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 -- 172.21.15.46:0/802872047 >> 172.21.15.46:0/802872047 conn(0x7f32800feaa0 msgr2=0x7f3280100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 -- 172.21.15.46:0/802872047 shutdown_connections 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.545+0000 7f3288600640 1 -- 172.21.15.46:0/802872047 wait complete. 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 Processor -- start 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 -- start start 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280075cb0 0x7f328019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f32801a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32801135c0 con 0x7f3280075cb0 2024-03-08T23:04:38.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280113440 con 0x7f3280077890 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.546+0000 7f3288600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3280113740 con 0x7f3280076ef0 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:41924/0 (socket says 172.21.15.46:41924) 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 -- 172.21.15.46:0/2229487833 learned_addr learned my addr 172.21.15.46:0/2229487833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285c00640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280075cb0 0x7f328019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3286600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f32801a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 -- 172.21.15.46:0/2229487833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 msgr2=0x7f32801a5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f32801a5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 -- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280075cb0 msgr2=0x7f328019db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280075cb0 0x7f328019db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:38.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 -- 172.21.15.46:0/2229487833 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32801a57b0 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.547+0000 7f3285200640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e9f0 tx=0x7f326c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.548+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c00ccc0 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.548+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32801a5aa0 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.548+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f326c010910 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.548+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c0188a0 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.548+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32801a6040 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.549+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328010fea0 con 0x7f3280076ef0 2024-03-08T23:04:38.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.552+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f326c010ac0 con 0x7f3280076ef0 2024-03-08T23:04:38.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.553+0000 7f327e000640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 0x7f32640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:38.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.553+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f326c0cf1d0 con 0x7f3280076ef0 2024-03-08T23:04:38.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.553+0000 7f3285c00640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 0x7f32640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:38.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.554+0000 7f3285c00640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 0x7f32640a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3270005fd0 tx=0x7f3270009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:38.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.554+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f326c09c380 con 0x7f3280076ef0 2024-03-08T23:04:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:38 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-08T23:04:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:38 smithi192 ceph-mon[36700]: osdmap e82: 12 total, 12 up, 12 in 2024-03-08T23:04:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:38 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-08T23:04:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:38 smithi062 ceph-mon[35761]: osdmap e82: 12 total, 12 up, 12 in 2024-03-08T23:04:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:38 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-08T23:04:38.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:38 smithi046 ceph-mon[31859]: osdmap e82: 12 total, 12 up, 12 in 2024-03-08T23:04:38.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:38.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:38.795+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f328002ce70 con 0x7f3280076ef0 2024-03-08T23:04:39.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.355+0000 7f327e000640 1 -- 172.21.15.46:0/2229487833 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f326c09bc80 con 0x7f3280076ef0 2024-03-08T23:04:39.361 INFO:teuthology.orchestra.run.smithi046.stderr:new fs with metadata pool 2 and data pool 3 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.360+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 msgr2=0x7f32640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.360+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 0x7f32640a4d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3270005fd0 tx=0x7f3270009260 comp rx=0 tx=0).stop 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.360+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 msgr2=0x7f328019e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.360+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f326c00e9f0 tx=0x7f326c00edf0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 shutdown_connections 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f32640a2880 0x7f32640a4d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3280077890 0x7f32801a5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3280076ef0 0x7f328019e0d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 --2- 172.21.15.46:0/2229487833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3280075cb0 0x7f328019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.361+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 >> 172.21.15.46:0/2229487833 conn(0x7f32800feaa0 msgr2=0x7f32801005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.362+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 shutdown_connections 2024-03-08T23:04:39.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.362+0000 7f3288600640 1 -- 172.21.15.46:0/2229487833 wait complete. 2024-03-08T23:04:39.389 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-08T23:04:39.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1046168241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: osdmap e83: 12 total, 12 up, 12 in 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2229487833' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-08T23:04:39.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:39 smithi192 ceph-mon[36700]: osdmap e84: 12 total, 12 up, 12 in 2024-03-08T23:04:39.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 -- 172.21.15.46:0/3456229448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc1050e0 msgr2=0x7fe1bc10b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 --2- 172.21.15.46:0/3456229448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc10b970 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009ca0 tx=0x7fe1b402c100 comp rx=0 tx=0).stop 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 -- 172.21.15.46:0/3456229448 shutdown_connections 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 --2- 172.21.15.46:0/3456229448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc10b970 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 --2- 172.21.15.46:0/3456229448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc104720 0x7fe1bc104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 --2- 172.21.15.46:0/3456229448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc103530 0x7fe1bc103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 -- 172.21.15.46:0/3456229448 >> 172.21.15.46:0/3456229448 conn(0x7fe1bc0feaa0 msgr2=0x7fe1bc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 -- 172.21.15.46:0/3456229448 shutdown_connections 2024-03-08T23:04:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.604+0000 7fe1c4000640 1 -- 172.21.15.46:0/3456229448 wait complete. 2024-03-08T23:04:39.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.605+0000 7fe1c4000640 1 Processor -- start 2024-03-08T23:04:39.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.605+0000 7fe1c4000640 1 -- start start 2024-03-08T23:04:39.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc104720 0x7fe1bc199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10f1c0 con 0x7fe1bc103530 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10f040 con 0x7fe1bc104720 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c4000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1bc10f340 con 0x7fe1bc1050e0 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41236/0 (socket says 172.21.15.46:41236) 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 -- 172.21.15.46:0/2271022651 learned_addr learned my addr 172.21.15.46:0/2271022651 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c0c00640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc104720 0x7fe1bc199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c2000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 -- 172.21.15.46:0/2271022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc1050e0 msgr2=0x7fe1bc1a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc1a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 -- 172.21.15.46:0/2271022651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc104720 msgr2=0x7fe1bc199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc104720 0x7fe1bc199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.606+0000 7fe1c1600640 1 -- 172.21.15.46:0/2271022651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc1a13d0 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1c1600640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000e9c0 tx=0x7fe1b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b000cd80 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1b00035f0 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b0010690 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1bc1a16c0 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.607+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1bc1a1bc0 con 0x7fe1bc103530 2024-03-08T23:04:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.608+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1bc10a850 con 0x7fe1bc103530 2024-03-08T23:04:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.610+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe1b00037a0 con 0x7fe1bc103530 2024-03-08T23:04:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.612+0000 7fe1ba000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.612+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe1b00cedb0 con 0x7fe1bc103530 2024-03-08T23:04:39.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.612+0000 7fe1c0c00640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.613+0000 7fe1c0c00640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 0x7fe1a40a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc19acc0 tx=0x7fe1a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.615+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe1b009bf90 con 0x7fe1bc103530 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1046168241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: osdmap e83: 12 total, 12 up, 12 in 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2229487833' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-08T23:04:39.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:39 smithi062 ceph-mon[35761]: osdmap e84: 12 total, 12 up, 12 in 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1046168241' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: osdmap e83: 12 total, 12 up, 12 in 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2229487833' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-08T23:04:39.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:39 smithi046 ceph-mon[31859]: osdmap e84: 12 total, 12 up, 12 in 2024-03-08T23:04:39.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:39.826+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fe1bc02ce70 con 0x7fe1bc103530 2024-03-08T23:04:40.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.059+0000 7fe1ba000640 1 -- 172.21.15.46:0/2271022651 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7fe1b009b890 con 0x7fe1bc103530 2024-03-08T23:04:40.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.064+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 msgr2=0x7fe1a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.064+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 0x7fe1a40a4d40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc19acc0 tx=0x7fe1a8005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.064+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 msgr2=0x7fe1bc1997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.064+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe1b000e9c0 tx=0x7fe1b000ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 shutdown_connections 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe1a40a2880 0x7fe1a40a4d40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1bc1050e0 0x7fe1bc1a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1bc104720 0x7fe1bc199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 --2- 172.21.15.46:0/2271022651 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1bc103530 0x7fe1bc1997b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.065+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 >> 172.21.15.46:0/2271022651 conn(0x7fe1bc0feaa0 msgr2=0x7fe1bc107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.066+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 shutdown_connections 2024-03-08T23:04:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.066+0000 7fe1c4000640 1 -- 172.21.15.46:0/2271022651 wait complete. 2024-03-08T23:04:40.094 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-08T23:04:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 -- 172.21.15.46:0/1299336099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48075cb0 msgr2=0x7f2c480760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 --2- 172.21.15.46:0/1299336099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48075cb0 0x7f2c480760d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2c38009860 tx=0x7f2c3802bb70 comp rx=0 tx=0).stop 2024-03-08T23:04:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 -- 172.21.15.46:0/1299336099 shutdown_connections 2024-03-08T23:04:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 --2- 172.21.15.46:0/1299336099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48077890 0x7f2c48078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 --2- 172.21.15.46:0/1299336099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48076ef0 0x7f2c48077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 --2- 172.21.15.46:0/1299336099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48075cb0 0x7f2c480760d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 -- 172.21.15.46:0/1299336099 >> 172.21.15.46:0/1299336099 conn(0x7f2c480fe8d0 msgr2=0x7f2c48100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.314+0000 7f2c47e00640 1 -- 172.21.15.46:0/1299336099 shutdown_connections 2024-03-08T23:04:40.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.315+0000 7f2c47e00640 1 -- 172.21.15.46:0/1299336099 wait complete. 2024-03-08T23:04:40.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.316+0000 7f2c47e00640 1 Processor -- start 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.316+0000 7f2c47e00640 1 -- start start 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.316+0000 7f2c47e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.316+0000 7f2c47e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48076ef0 0x7f2c4819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 0x7f2c481a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48113390 con 0x7f2c48075cb0 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48113210 con 0x7f2c48077890 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48113510 con 0x7f2c48076ef0 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:40.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41250/0 (socket says 172.21.15.46:41250) 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 -- 172.21.15.46:0/4180089775 learned_addr learned my addr 172.21.15.46:0/4180089775 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47400640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 0x7f2c481a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c3e000640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48076ef0 0x7f2c4819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 -- 172.21.15.46:0/4180089775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48076ef0 msgr2=0x7f2c4819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48076ef0 0x7f2c4819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 -- 172.21.15.46:0/4180089775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 msgr2=0x7f2c481a4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 0x7f2c481a4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c46a00640 1 -- 172.21.15.46:0/4180089775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c481a5630 con 0x7f2c48075cb0 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.317+0000 7f2c47400640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 0x7f2c481a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c46a00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2c380349e0 tx=0x7f2c38032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:40.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38041030 con 0x7f2c48075cb0 2024-03-08T23:04:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c481a58c0 con 0x7f2c48075cb0 2024-03-08T23:04:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c3802c510 con 0x7f2c48075cb0 2024-03-08T23:04:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c481a5e00 con 0x7f2c48075cb0 2024-03-08T23:04:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.318+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c38039040 con 0x7f2c48075cb0 2024-03-08T23:04:40.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.319+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4810fe60 con 0x7f2c48075cb0 2024-03-08T23:04:40.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.322+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2c38044070 con 0x7f2c48075cb0 2024-03-08T23:04:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.323+0000 7f2c4ca00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 0x7f2c2c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:40.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.323+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2c380f03a0 con 0x7f2c48075cb0 2024-03-08T23:04:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.323+0000 7f2c3e000640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 0x7f2c2c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:40.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.324+0000 7f2c3e000640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 0x7f2c2c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c300095f0 tx=0x7f2c30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:40.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.327+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2c380bd410 con 0x7f2c48075cb0 2024-03-08T23:04:40.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:40.536+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2c4802ce70 con 0x7f2c48075cb0 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: pgmap v314: 97 pgs: 8 creating+peering, 32 active+clean, 57 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-08T23:04:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:40 smithi062 ceph-mon[35761]: osdmap e85: 12 total, 12 up, 12 in 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: pgmap v314: 97 pgs: 8 creating+peering, 32 active+clean, 57 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-08T23:04:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:40 smithi046 ceph-mon[31859]: osdmap e85: 12 total, 12 up, 12 in 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: pgmap v314: 97 pgs: 8 creating+peering, 32 active+clean, 57 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2271022651' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-08T23:04:40.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:40 smithi192 ceph-mon[36700]: osdmap e85: 12 total, 12 up, 12 in 2024-03-08T23:04:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:41 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:active 2024-03-08T23:04:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:41 smithi062 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:41 smithi062 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-08T23:04:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:41 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:active 2024-03-08T23:04:41.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:41 smithi046 ceph-mon[31859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:41 smithi046 ceph-mon[31859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-08T23:04:41.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:41 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] up:active 2024-03-08T23:04:41.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:41 smithi192 ceph-mon[36700]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:41 smithi192 ceph-mon[36700]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:41.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-08T23:04:42.384 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:04:42 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: 2024-03-08T23:04:42.026+0000 7ff761276700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-08T23:04:42.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.393+0000 7f2c4ca00640 1 -- 172.21.15.46:0/4180089775 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7f2c380bcd10 con 0x7f2c48075cb0 2024-03-08T23:04:42.398 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' created 2024-03-08T23:04:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 msgr2=0x7f2c2c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 0x7f2c2c0a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2c300095f0 tx=0x7f2c30005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 msgr2=0x7f2c4819da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:42.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2c380349e0 tx=0x7f2c38032450 comp rx=0 tx=0).stop 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 shutdown_connections 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2c2c0a2800 0x7f2c2c0a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2c48077890 0x7f2c481a4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2c48076ef0 0x7f2c4819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 --2- 172.21.15.46:0/4180089775 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2c48075cb0 0x7f2c4819da10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.398+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 >> 172.21.15.46:0/4180089775 conn(0x7f2c480fe8d0 msgr2=0x7f2c480ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:42.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.399+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 shutdown_connections 2024-03-08T23:04:42.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.400+0000 7f2c47e00640 1 -- 172.21.15.46:0/4180089775 wait complete. 2024-03-08T23:04:42.432 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-08T23:04:42.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 -- 172.21.15.46:0/661635676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec104600 msgr2=0x7ff6ec104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:42.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 --2- 172.21.15.46:0/661635676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec104600 0x7ff6ec104a80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8009860 tx=0x7ff6d802bb70 comp rx=0 tx=0).stop 2024-03-08T23:04:42.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 -- 172.21.15.46:0/661635676 shutdown_connections 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 --2- 172.21.15.46:0/661635676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec10b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 --2- 172.21.15.46:0/661635676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec104600 0x7ff6ec104a80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 --2- 172.21.15.46:0/661635676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec103410 0x7ff6ec103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 -- 172.21.15.46:0/661635676 >> 172.21.15.46:0/661635676 conn(0x7ff6ec0fe9a0 msgr2=0x7ff6ec100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 -- 172.21.15.46:0/661635676 shutdown_connections 2024-03-08T23:04:42.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.623+0000 7ff6f1400640 1 -- 172.21.15.46:0/661635676 wait complete. 2024-03-08T23:04:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 Processor -- start 2024-03-08T23:04:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 -- start start 2024-03-08T23:04:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec103410 0x7ff6ec1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:42.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec10f0a0 con 0x7ff6ec104600 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec10ef20 con 0x7ff6ec104fc0 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.624+0000 7ff6f1400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6ec10f220 con 0x7ff6ec103410 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41266/0 (socket says 172.21.15.46:41266) 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 -- 172.21.15.46:0/751359408 learned_addr learned my addr 172.21.15.46:0/751359408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6ebe00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec103410 0x7ff6ec1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6f0a00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 -- 172.21.15.46:0/751359408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec103410 msgr2=0x7ff6ec1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec103410 0x7ff6ec1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 -- 172.21.15.46:0/751359408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 msgr2=0x7ff6ec1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:42.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 -- 172.21.15.46:0/751359408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ec1a1300 con 0x7ff6ec104600 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6f0a00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.625+0000 7ff6eb400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005940 tx=0x7ff6d8032470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.626+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8041020 con 0x7ff6ec104600 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.626+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6d802c9f0 con 0x7ff6ec104600 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.626+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6ec1a1590 con 0x7ff6ec104600 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.626+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6d8039040 con 0x7ff6ec104600 2024-03-08T23:04:42.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.626+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6ec1a1a80 con 0x7ff6ec104600 2024-03-08T23:04:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.627+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6ec10a730 con 0x7ff6ec104600 2024-03-08T23:04:42.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.629+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff6d8044020 con 0x7ff6ec104600 2024-03-08T23:04:42.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.630+0000 7ff6e8c00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 0x7ff6d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:42.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.630+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7ff6d80f0b30 con 0x7ff6ec104600 2024-03-08T23:04:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.631+0000 7ff6ebe00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 0x7ff6d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:42.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.631+0000 7ff6ebe00640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 0x7ff6d00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec104460 tx=0x7ff6dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:42.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.633+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff6d804b020 con 0x7ff6ec104600 2024-03-08T23:04:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:42 smithi062 ceph-mon[35761]: pgmap v316: 97 pgs: 8 creating+peering, 66 active+clean, 23 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-08T23:04:42.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:42 smithi062 ceph-mon[35761]: osdmap e86: 12 total, 12 up, 12 in 2024-03-08T23:04:42.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:42 smithi046 ceph-mon[31859]: pgmap v316: 97 pgs: 8 creating+peering, 66 active+clean, 23 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-08T23:04:42.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:42 smithi046 ceph-mon[31859]: osdmap e86: 12 total, 12 up, 12 in 2024-03-08T23:04:42.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:42 smithi192 ceph-mon[36700]: pgmap v316: 97 pgs: 8 creating+peering, 66 active+clean, 23 unknown; 578 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 2 op/s 2024-03-08T23:04:42.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:42 smithi192 ceph-mon[36700]: osdmap e86: 12 total, 12 up, 12 in 2024-03-08T23:04:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:42.837+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7ff6ec02ce70 con 0x7ff6ec104600 2024-03-08T23:04:43.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.404+0000 7ff6e8c00640 1 -- 172.21.15.46:0/751359408 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff6d80bd950 con 0x7ff6ec104600 2024-03-08T23:04:43.409 INFO:teuthology.orchestra.run.smithi046.stderr:set pool 4 allow_ec_overwrites to true 2024-03-08T23:04:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.408+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 msgr2=0x7ff6d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.408+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 0x7ff6d00a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec104460 tx=0x7ff6dc009260 comp rx=0 tx=0).stop 2024-03-08T23:04:43.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.408+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 msgr2=0x7ff6ec199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.408+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6d8005940 tx=0x7ff6d8032470 comp rx=0 tx=0).stop 2024-03-08T23:04:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 shutdown_connections 2024-03-08T23:04:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6d00a2800 0x7ff6d00a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6ec104fc0 0x7ff6ec1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6ec104600 0x7ff6ec199c20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 --2- 172.21.15.46:0/751359408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6ec103410 0x7ff6ec1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 >> 172.21.15.46:0/751359408 conn(0x7ff6ec0fe9a0 msgr2=0x7ff6ec107900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 shutdown_connections 2024-03-08T23:04:43.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.409+0000 7ff6f1400640 1 -- 172.21.15.46:0/751359408 wait complete. 2024-03-08T23:04:43.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 -- 172.21.15.46:0/1363890149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 msgr2=0x7fe5880741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 --2- 172.21.15.46:0/1363890149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5880741d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe570009900 tx=0x7fe57002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 -- 172.21.15.46:0/1363890149 shutdown_connections 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 --2- 172.21.15.46:0/1363890149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588109090 0x7fe58810b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 --2- 172.21.15.46:0/1363890149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5880741d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 --2- 172.21.15.46:0/1363890149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588075330 0x7fe588073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 -- 172.21.15.46:0/1363890149 >> 172.21.15.46:0/1363890149 conn(0x7fe5880fc760 msgr2=0x7fe5880feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 -- 172.21.15.46:0/1363890149 shutdown_connections 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.635+0000 7fe587e00640 1 -- 172.21.15.46:0/1363890149 wait complete. 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.636+0000 7fe587e00640 1 Processor -- start 2024-03-08T23:04:43.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.636+0000 7fe587e00640 1 -- start start 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5881996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 0x7fe5881a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58810f0a0 con 0x7fe588073d50 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58810ef20 con 0x7fe588075330 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe58810f220 con 0x7fe588109090 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe586a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5881996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37458/0 (socket says 172.21.15.46:37458) 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 -- 172.21.15.46:0/3641595966 learned_addr learned my addr 172.21.15.46:0/3641595966 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:43.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587400640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 0x7fe5881a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:43.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 -- 172.21.15.46:0/3641595966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 msgr2=0x7fe5881a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:43.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 0x7fe5881a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 -- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 msgr2=0x7fe5881996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:43.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5881996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:43.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe57e000640 1 -- 172.21.15.46:0/3641595966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5881a12e0 con 0x7fe588075330 2024-03-08T23:04:43.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe586a00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5881996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:43.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.637+0000 7fe587400640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 0x7fe5881a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:04:43.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe57e000640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe570005ab0 tx=0x7fe57002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:43.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570043020 con 0x7fe588075330 2024-03-08T23:04:43.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5700054e0 con 0x7fe588075330 2024-03-08T23:04:43.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5881a1570 con 0x7fe588075330 2024-03-08T23:04:43.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe570003ce0 con 0x7fe588075330 2024-03-08T23:04:43.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.638+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5881a1ac0 con 0x7fe588075330 2024-03-08T23:04:43.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.639+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe588107dc0 con 0x7fe588075330 2024-03-08T23:04:43.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.642+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe570046020 con 0x7fe588075330 2024-03-08T23:04:43.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.643+0000 7fe58ca00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 0x7fe56c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:43.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.643+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7fe5700f1270 con 0x7fe588075330 2024-03-08T23:04:43.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.643+0000 7fe586a00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 0x7fe56c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:43.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.644+0000 7fe586a00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 0x7fe56c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe578009fc0 tx=0x7fe578005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:43.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.645+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe57004d020 con 0x7fe588075330 2024-03-08T23:04:43.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:43 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-08T23:04:43.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:43 smithi046 ceph-mon[31859]: osdmap e87: 12 total, 12 up, 12 in 2024-03-08T23:04:43.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:43 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-08T23:04:43.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:43 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-08T23:04:43.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:43 smithi192 ceph-mon[36700]: osdmap e87: 12 total, 12 up, 12 in 2024-03-08T23:04:43.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:43 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-08T23:04:43.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:43.885+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7fe58802ce70 con 0x7fe588075330 2024-03-08T23:04:43.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:43 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4180089775' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-08T23:04:43.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:43 smithi062 ceph-mon[35761]: osdmap e87: 12 total, 12 up, 12 in 2024-03-08T23:04:43.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:43 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-08T23:04:44.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.411+0000 7fe58ca00640 1 -- 172.21.15.46:0/3641595966 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7fe5700be090 con 0x7fe588075330 2024-03-08T23:04:44.416 INFO:teuthology.orchestra.run.smithi046.stderr:added data pool 4 to fsmap 2024-03-08T23:04:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.416+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 msgr2=0x7fe56c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.416+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 0x7fe56c0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fe578009fc0 tx=0x7fe578005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.416+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 msgr2=0x7fe588199c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.416+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fe570005ab0 tx=0x7fe57002c6c0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 shutdown_connections 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe56c0a2800 0x7fe56c0a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe588109090 0x7fe5881a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe588075330 0x7fe588199c00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 --2- 172.21.15.46:0/3641595966 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe588073d50 0x7fe5881996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.417+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 >> 172.21.15.46:0/3641595966 conn(0x7fe5880fc760 msgr2=0x7fe588104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:44.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.418+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 shutdown_connections 2024-03-08T23:04:44.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.418+0000 7fe587e00640 1 -- 172.21.15.46:0/3641595966 wait complete. 2024-03-08T23:04:44.444 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: pgmap v319: 129 pgs: 8 creating+peering, 92 active+clean, 29 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: osdmap e88: 12 total, 12 up, 12 in 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3641595966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:44 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 -- 172.21.15.46:0/4064017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140103340 msgr2=0x7f4140103760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 --2- 172.21.15.46:0/4064017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140103340 0x7f4140103760 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4130009900 tx=0x7f413002bc50 comp rx=0 tx=0).stop 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 -- 172.21.15.46:0/4064017712 shutdown_connections 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 --2- 172.21.15.46:0/4064017712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104ef0 0x7f414010b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 --2- 172.21.15.46:0/4064017712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140104530 0x7f41401049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 --2- 172.21.15.46:0/4064017712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140103340 0x7f4140103760 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 -- 172.21.15.46:0/4064017712 >> 172.21.15.46:0/4064017712 conn(0x7f41400fe8d0 msgr2=0x7f4140100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 -- 172.21.15.46:0/4064017712 shutdown_connections 2024-03-08T23:04:44.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.663+0000 7f4145800640 1 -- 172.21.15.46:0/4064017712 wait complete. 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 Processor -- start 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 -- start start 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140103340 0x7f4140199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140104ef0 0x7f41401a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414010f020 con 0x7f4140104ef0 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414010eea0 con 0x7f4140103340 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4145800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414010f1a0 con 0x7f4140104530 2024-03-08T23:04:44.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.664+0000 7f4137e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55916/0 (socket says 172.21.15.46:55916) 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 -- 172.21.15.46:0/3039260140 learned_addr learned my addr 172.21.15.46:0/3039260140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f413fe00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140103340 0x7f4140199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4144e00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140104ef0 0x7f41401a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 -- 172.21.15.46:0/3039260140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140103340 msgr2=0x7f4140199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140103340 0x7f4140199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 -- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140104ef0 msgr2=0x7f41401a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140104ef0 0x7f41401a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 -- 172.21.15.46:0/3039260140 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41401a12a0 con 0x7f4140104530 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f4137e00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f412800ea60 tx=0x7f412800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.665+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412800cd20 con 0x7f4140104530 2024-03-08T23:04:44.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.666+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41280035f0 con 0x7f4140104530 2024-03-08T23:04:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.666+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41401a1590 con 0x7f4140104530 2024-03-08T23:04:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.666+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128010690 con 0x7f4140104530 2024-03-08T23:04:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.666+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41401a1ae0 con 0x7f4140104530 2024-03-08T23:04:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.667+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414010a6b0 con 0x7f4140104530 2024-03-08T23:04:44.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.669+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f41280037a0 con 0x7f4140104530 2024-03-08T23:04:44.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.670+0000 7f413d600640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 0x7f41240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:44.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.670+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f41280ceff0 con 0x7f4140104530 2024-03-08T23:04:44.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.670+0000 7f413fe00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 0x7f41240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:44.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.671+0000 7f413fe00640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 0x7f41240a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413002c190 tx=0x7f4130000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:44.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.675+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f412809be10 con 0x7f4140104530 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: pgmap v319: 129 pgs: 8 creating+peering, 92 active+clean, 29 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: osdmap e88: 12 total, 12 up, 12 in 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3641595966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:44 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: pgmap v319: 129 pgs: 8 creating+peering, 92 active+clean, 29 unknown; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 745 B/s wr, 2 op/s 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/751359408' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: osdmap e88: 12 total, 12 up, 12 in 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3641595966' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:44 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-08T23:04:44.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.906+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f414002ce70 con 0x7f4140104530 2024-03-08T23:04:44.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.908+0000 7f413d600640 1 -- 172.21.15.46:0/3039260140 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11274 (secure 0 0 0) 0x7f412809b710 con 0x7f4140104530 2024-03-08T23:04:44.912 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:44.912 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:04:43.889168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-08T23:04:44.913 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-03-08T23:04:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 msgr2=0x7f41240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 0x7f41240a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f413002c190 tx=0x7f4130000bc0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 msgr2=0x7f4140199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:44.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f412800ea60 tx=0x7f412800ee60 comp rx=0 tx=0).stop 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 shutdown_connections 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f41240a2800 0x7f41240a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4140104ef0 0x7f41401a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4140104530 0x7f4140199bc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 --2- 172.21.15.46:0/3039260140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4140103340 0x7f4140199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 >> 172.21.15.46:0/3039260140 conn(0x7f41400fe8d0 msgr2=0x7f4140107880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 shutdown_connections 2024-03-08T23:04:44.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:44.911+0000 7f4145800640 1 -- 172.21.15.46:0/3039260140 wait complete. 2024-03-08T23:04:44.936 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.133+0000 7f16fec00640 1 -- 172.21.15.46:0/3871421666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8106e20 msgr2=0x7f16f8109270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.133+0000 7f16fec00640 1 --2- 172.21.15.46:0/3871421666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8106e20 0x7f16f8109270 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f16f0009be0 tx=0x7f16f002bef0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 -- 172.21.15.46:0/3871421666 shutdown_connections 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 --2- 172.21.15.46:0/3871421666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8106e20 0x7f16f8109270 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 --2- 172.21.15.46:0/3871421666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f81044f0 0x7f16f81068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 --2- 172.21.15.46:0/3871421666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f8101bc0 0x7f16f8103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 -- 172.21.15.46:0/3871421666 >> 172.21.15.46:0/3871421666 conn(0x7f16f80fb5d0 msgr2=0x7f16f80fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 -- 172.21.15.46:0/3871421666 shutdown_connections 2024-03-08T23:04:45.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.134+0000 7f16fec00640 1 -- 172.21.15.46:0/3871421666 wait complete. 2024-03-08T23:04:45.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 Processor -- start 2024-03-08T23:04:45.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 -- start start 2024-03-08T23:04:45.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 0x7f16f80ffb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 0x7f16f81000b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810ce20 con 0x7f16f8101bc0 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810cca0 con 0x7f16f81044f0 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fec00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f810cfa0 con 0x7f16f8106e20 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fd800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fd800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57332/0 (socket says 172.21.15.46:57332) 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fd800640 1 -- 172.21.15.46:0/2198908045 learned_addr learned my addr 172.21.15.46:0/2198908045 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.135+0000 7f16fe200640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 0x7f16f81000b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fce00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 0x7f16f80ffb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 -- 172.21.15.46:0/2198908045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 msgr2=0x7f16f80ffb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 0x7f16f80ffb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 -- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 msgr2=0x7f16f81014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 -- 172.21.15.46:0/2198908045 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f81008d0 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fd800640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fce00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 0x7f16f80ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.136+0000 7f16fe200640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 0x7f16f81000b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16f002c430 tx=0x7f16f00337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.137+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0043020 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.137+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f81a14c0 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.137+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16f002ca60 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.137+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0034690 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.137+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f81a1a60 con 0x7f16f8106e20 2024-03-08T23:04:45.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.138+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f8068000 con 0x7f16f8106e20 2024-03-08T23:04:45.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.142+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f16f0046020 con 0x7f16f8106e20 2024-03-08T23:04:45.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.143+0000 7f16f6000640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 0x7f16e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.143+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f16f00f1810 con 0x7f16f8106e20 2024-03-08T23:04:45.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.143+0000 7f16fd800640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 0x7f16e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.144+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f16f004d030 con 0x7f16f8106e20 2024-03-08T23:04:45.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.144+0000 7f16fd800640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 0x7f16e00a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0096c0 tx=0x7f16ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.345+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16f8061ef0 con 0x7f16f8106e20 2024-03-08T23:04:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.347+0000 7f16f6000640 1 -- 172.21.15.46:0/2198908045 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f16f00be730 con 0x7f16f8106e20 2024-03-08T23:04:45.352 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:45.352 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 msgr2=0x7f16e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 0x7f16e00a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f16ec0096c0 tx=0x7f16ec009260 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 msgr2=0x7f16f81000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 0x7f16f81000b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f16f002c430 tx=0x7f16f00337b0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 shutdown_connections 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f16e00a2800 0x7f16e00a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f16f8106e20 0x7f16f81000b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f16f81044f0 0x7f16f80ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 --2- 172.21.15.46:0/2198908045 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f16f8101bc0 0x7f16f81014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.349+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 >> 172.21.15.46:0/2198908045 conn(0x7f16f80fb5d0 msgr2=0x7f16f8102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:45.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.350+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 shutdown_connections 2024-03-08T23:04:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.350+0000 7f16fec00640 1 -- 172.21.15.46:0/2198908045 wait complete. 2024-03-08T23:04:45.373 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:45.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 -- 172.21.15.46:0/247413265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8102ec0 msgr2=0x7ff6f8109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 --2- 172.21.15.46:0/247413265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8109750 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0009ca0 tx=0x7ff6f002c100 comp rx=0 tx=0).stop 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 -- 172.21.15.46:0/247413265 shutdown_connections 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 --2- 172.21.15.46:0/247413265 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8109750 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 --2- 172.21.15.46:0/247413265 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 0x7ff6f8102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 --2- 172.21.15.46:0/247413265 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8101310 0x7ff6f8101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 -- 172.21.15.46:0/247413265 >> 172.21.15.46:0/247413265 conn(0x7ff6f80fc8a0 msgr2=0x7ff6f80fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 -- 172.21.15.46:0/247413265 shutdown_connections 2024-03-08T23:04:45.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.567+0000 7ff6fca00640 1 -- 172.21.15.46:0/247413265 wait complete. 2024-03-08T23:04:45.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.568+0000 7ff6fca00640 1 Processor -- start 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.568+0000 7ff6fca00640 1 -- start start 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 0x7ff6f80736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f810cfa0 con 0x7ff6f8101310 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f810ce20 con 0x7ff6f8102ec0 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6fca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f810d120 con 0x7ff6f8102500 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57346/0 (socket says 172.21.15.46:57346) 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 -- 172.21.15.46:0/2704963781 learned_addr learned my addr 172.21.15.46:0/2704963781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f6800640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5400640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 0x7ff6f80736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 -- 172.21.15.46:0/2704963781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 msgr2=0x7ff6f80736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 0x7ff6f80736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 -- 172.21.15.46:0/2704963781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 msgr2=0x7ff6f8073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f5e00640 1 -- 172.21.15.46:0/2704963781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f80744b0 con 0x7ff6f8101310 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.569+0000 7ff6f6800640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6f5e00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800e9c0 tx=0x7ff6e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e800cd80 con 0x7ff6f8101310 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f81a1750 con 0x7ff6f8101310 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e80035f0 con 0x7ff6f8101310 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8010690 con 0x7ff6f8101310 2024-03-08T23:04:45.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.570+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f81a1cf0 con 0x7ff6f8101310 2024-03-08T23:04:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.571+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f81085d0 con 0x7ff6f8101310 2024-03-08T23:04:45.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.573+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff6e80037a0 con 0x7ff6f8101310 2024-03-08T23:04:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.574+0000 7ff6eea00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.575+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff6e80cf140 con 0x7ff6f8101310 2024-03-08T23:04:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.575+0000 7ff6f5400640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:45.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.575+0000 7ff6f5400640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8074e10 tx=0x7ff6e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:45.583 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:45 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-08T23:04:45.583 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:45 smithi046 ceph-mon[31859]: osdmap e89: 12 total, 12 up, 12 in 2024-03-08T23:04:45.583 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:45 smithi046 ceph-mon[31859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:45.583 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:45 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3039260140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:45.583 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:45 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2198908045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:45.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.579+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff6e809bf40 con 0x7ff6f8101310 2024-03-08T23:04:45.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.796+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f8061ef0 con 0x7ff6f8101310 2024-03-08T23:04:45.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.800+0000 7ff6eea00640 1 -- 172.21.15.46:0/2704963781 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7ff6e809b840 con 0x7ff6f8101310 2024-03-08T23:04:45.805 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:45.805 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:45.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 msgr2=0x7ff6dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8074e10 tx=0x7ff6e4005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 msgr2=0x7ff6f80750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800e9c0 tx=0x7ff6e800ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 shutdown_connections 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff6dc0a2800 0x7ff6dc0a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8102ec0 0x7ff6f8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8102500 0x7ff6f80736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 --2- 172.21.15.46:0/2704963781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f8101310 0x7ff6f80750a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 >> 172.21.15.46:0/2704963781 conn(0x7ff6f80fc8a0 msgr2=0x7ff6f81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 shutdown_connections 2024-03-08T23:04:45.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:45.803+0000 7ff6fca00640 1 -- 172.21.15.46:0/2704963781 wait complete. 2024-03-08T23:04:45.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:45 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-08T23:04:45.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:45 smithi192 ceph-mon[36700]: osdmap e89: 12 total, 12 up, 12 in 2024-03-08T23:04:45.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:45 smithi192 ceph-mon[36700]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:45.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:45 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3039260140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:45.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:45 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2198908045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:45.842 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:04:45.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:45 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-08T23:04:45.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:45 smithi062 ceph-mon[35761]: osdmap e89: 12 total, 12 up, 12 in 2024-03-08T23:04:45.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:45 smithi062 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:04:45.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:45 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3039260140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:45.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:45 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2198908045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 -- 172.21.15.46:0/1303166801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00101330 msgr2=0x7fdc00101750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 --2- 172.21.15.46:0/1303166801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00101330 0x7fdc00101750 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4009900 tx=0x7fdbf402bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 -- 172.21.15.46:0/1303166801 shutdown_connections 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 --2- 172.21.15.46:0/1303166801 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102ee0 0x7fdc00109770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 --2- 172.21.15.46:0/1303166801 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00102520 0x7fdc001029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 --2- 172.21.15.46:0/1303166801 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00101330 0x7fdc00101750 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 -- 172.21.15.46:0/1303166801 >> 172.21.15.46:0/1303166801 conn(0x7fdc000fc8a0 msgr2=0x7fdc000fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 -- 172.21.15.46:0/1303166801 shutdown_connections 2024-03-08T23:04:46.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.034+0000 7fdc08000640 1 -- 172.21.15.46:0/1303166801 wait complete. 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.035+0000 7fdc08000640 1 Processor -- start 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 -- start start 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102520 0x7fdc00199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 0x7fdc001a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0010cf90 con 0x7fdc00102ee0 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0010ce10 con 0x7fdc00101330 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc08000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0010d110 con 0x7fdc00102520 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc06000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 0x7fdc001a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc05600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc05600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37530/0 (socket says 172.21.15.46:37530) 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc05600640 1 -- 172.21.15.46:0/21139435 learned_addr learned my addr 172.21.15.46:0/21139435 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.036+0000 7fdc04c00640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102520 0x7fdc00199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 -- 172.21.15.46:0/21139435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102520 msgr2=0x7fdc00199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102520 0x7fdc00199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 -- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 msgr2=0x7fdc001a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 0x7fdc001a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 -- 172.21.15.46:0/21139435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc001a1490 con 0x7fdc00101330 2024-03-08T23:04:46.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc06000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 0x7fdc001a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc05600640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4000c00 tx=0x7fdbf4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4041070 con 0x7fdc00101330 2024-03-08T23:04:46.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbf40044d0 con 0x7fdc00101330 2024-03-08T23:04:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4034440 con 0x7fdc00101330 2024-03-08T23:04:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.037+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc001a1720 con 0x7fdc00101330 2024-03-08T23:04:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.038+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc001a1c10 con 0x7fdc00101330 2024-03-08T23:04:46.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.039+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc001a1e70 con 0x7fdc00101330 2024-03-08T23:04:46.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.041+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdbf4044070 con 0x7fdc00101330 2024-03-08T23:04:46.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.042+0000 7fdbfe000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 0x7fdbe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.042+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdbf40f09e0 con 0x7fdc00101330 2024-03-08T23:04:46.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.042+0000 7fdc04c00640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 0x7fdbe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.043+0000 7fdc04c00640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 0x7fdbe80a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdc0019ad80 tx=0x7fdbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.044+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdbf40bd1f0 con 0x7fdc00101330 2024-03-08T23:04:46.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.279+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc001a2040 con 0x7fdc00101330 2024-03-08T23:04:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.280+0000 7fdbfe000640 1 -- 172.21.15.46:0/21139435 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+11274 (secure 0 0 0) 0x7fdbf40dd0c0 con 0x7fdc00101330 2024-03-08T23:04:46.285 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:46.285 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24520,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24560,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:04:43.889168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24578},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-08T23:04:46.285 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 17 2024-03-08T23:04:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 msgr2=0x7fdbe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 0x7fdbe80a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fdc0019ad80 tx=0x7fdbec005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 msgr2=0x7fdc001998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdbf4000c00 tx=0x7fdbf4005d50 comp rx=0 tx=0).stop 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 shutdown_connections 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe80a2800 0x7fdbe80a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdc00102ee0 0x7fdc001a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdc00102520 0x7fdc00199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 --2- 172.21.15.46:0/21139435 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdc00101330 0x7fdc001998c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 >> 172.21.15.46:0/21139435 conn(0x7fdc000fc8a0 msgr2=0x7fdc00073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 shutdown_connections 2024-03-08T23:04:46.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.283+0000 7fdc08000640 1 -- 172.21.15.46:0/21139435 wait complete. 2024-03-08T23:04:46.306 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 -- 172.21.15.46:0/2618393111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620104720 msgr2=0x7fe620104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 --2- 172.21.15.46:0/2618393111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620104720 0x7fe620104ba0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009900 tx=0x7fe60c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 -- 172.21.15.46:0/2618393111 shutdown_connections 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 --2- 172.21.15.46:0/2618393111 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe6201050e0 0x7fe62010b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 --2- 172.21.15.46:0/2618393111 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe620104720 0x7fe620104ba0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 --2- 172.21.15.46:0/2618393111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe620103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 -- 172.21.15.46:0/2618393111 >> 172.21.15.46:0/2618393111 conn(0x7fe6200feaa0 msgr2=0x7fe620100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 -- 172.21.15.46:0/2618393111 shutdown_connections 2024-03-08T23:04:46.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.503+0000 7fe626000640 1 -- 172.21.15.46:0/2618393111 wait complete. 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 Processor -- start 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 -- start start 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe620104720 0x7fe620199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 0x7fe6201a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62010f1c0 con 0x7fe6201050e0 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62010f040 con 0x7fe620104720 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.505+0000 7fe626000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62010f340 con 0x7fe620103530 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55978/0 (socket says 172.21.15.46:55978) 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61f400640 1 -- 172.21.15.46:0/1485921018 learned_addr learned my addr 172.21.15.46:0/1485921018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 0x7fe6201a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61ea00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe620104720 0x7fe620199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 -- 172.21.15.46:0/1485921018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 msgr2=0x7fe6201997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 -- 172.21.15.46:0/1485921018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe620104720 msgr2=0x7fe620199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe620104720 0x7fe620199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 -- 172.21.15.46:0/1485921018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6201a1410 con 0x7fe6201050e0 2024-03-08T23:04:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61f400640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:46.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.506+0000 7fe61fe00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 0x7fe6201a0da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe618009fd0 tx=0x7fe61800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.507+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61800ce20 con 0x7fe6201050e0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.507+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe618005020 con 0x7fe6201050e0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.507+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6201a1700 con 0x7fe6201050e0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.507+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618010720 con 0x7fe6201050e0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.507+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6201a1bd0 con 0x7fe6201050e0 2024-03-08T23:04:46.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.508+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62010a850 con 0x7fe6201050e0 2024-03-08T23:04:46.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.510+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe618005540 con 0x7fe6201050e0 2024-03-08T23:04:46.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.512+0000 7fe617e00640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 0x7fe6040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.512+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe6180cf310 con 0x7fe6201050e0 2024-03-08T23:04:46.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.513+0000 7fe61f400640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 0x7fe6040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.513+0000 7fe61f400640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 0x7fe6040a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe6100058c0 tx=0x7fe610005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.514+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe61809c170 con 0x7fe6201050e0 2024-03-08T23:04:46.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.719+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe620061ef0 con 0x7fe6201050e0 2024-03-08T23:04:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:46 smithi046 ceph-mon[31859]: pgmap v322: 129 pgs: 3 creating+peering, 119 active+clean, 7 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-08T23:04:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:46 smithi046 ceph-mon[31859]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-08T23:04:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:46 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:04:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:46 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2704963781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:46 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/21139435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.724+0000 7fe617e00640 1 -- 172.21.15.46:0/1485921018 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7fe61809ba70 con 0x7fe6201050e0 2024-03-08T23:04:46.729 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:46.729 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.726+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 msgr2=0x7fe6040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.726+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 0x7fe6040a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fe6100058c0 tx=0x7fe610005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.726+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 msgr2=0x7fe6201a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.726+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 0x7fe6201a0da0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe618009fd0 tx=0x7fe61800ef10 comp rx=0 tx=0).stop 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 shutdown_connections 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe6040a2800 0x7fe6040a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe6201050e0 0x7fe6201a0da0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe620104720 0x7fe620199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 --2- 172.21.15.46:0/1485921018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe620103530 0x7fe6201997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 >> 172.21.15.46:0/1485921018 conn(0x7fe6200feaa0 msgr2=0x7fe620107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 shutdown_connections 2024-03-08T23:04:46.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.727+0000 7fe626000640 1 -- 172.21.15.46:0/1485921018 wait complete. 2024-03-08T23:04:46.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-08T23:04:38.798478+0000', 'modified': '2024-03-08T23:04:43.889168+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24578}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24578': {'gid': 24578, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6839/1938633866', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 1938633866}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 1938633866}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-08T23:04:46.751 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-08T23:04:46.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-08T23:04:46.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:46 smithi192 ceph-mon[36700]: pgmap v322: 129 pgs: 3 creating+peering, 119 active+clean, 7 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-08T23:04:46.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:46 smithi192 ceph-mon[36700]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-08T23:04:46.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:46 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:04:46.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:46 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2704963781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:46 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/21139435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:46 smithi062 ceph-mon[35761]: pgmap v322: 129 pgs: 3 creating+peering, 119 active+clean, 7 unknown; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-03-08T23:04:46.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:46 smithi062 ceph-mon[35761]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-08T23:04:46.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:46 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:04:46.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:46 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2704963781' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:46 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/21139435' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 -- 172.21.15.46:0/2331155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 msgr2=0x7fc32c104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 --2- 172.21.15.46:0/2331155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c104a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc318009900 tx=0x7fc31802bc40 comp rx=0 tx=0).stop 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 -- 172.21.15.46:0/2331155181 shutdown_connections 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 --2- 172.21.15.46:0/2331155181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c10b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 --2- 172.21.15.46:0/2331155181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c104a80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 --2- 172.21.15.46:0/2331155181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 0x7fc32c103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 -- 172.21.15.46:0/2331155181 >> 172.21.15.46:0/2331155181 conn(0x7fc32c0fe9a0 msgr2=0x7fc32c100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 -- 172.21.15.46:0/2331155181 shutdown_connections 2024-03-08T23:04:46.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.967+0000 7fc332000640 1 -- 172.21.15.46:0/2331155181 wait complete. 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 Processor -- start 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 -- start start 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 0x7fc32c1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c10f0f0 con 0x7fc32c104600 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c10ef70 con 0x7fc32c103410 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc332000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32c10f270 con 0x7fc32c104fc0 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.969+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57380/0 (socket says 172.21.15.46:57380) 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 -- 172.21.15.46:0/2508113406 learned_addr learned my addr 172.21.15.46:0/2508113406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc330c00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 0x7fc32c1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc331600640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 -- 172.21.15.46:0/2508113406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 msgr2=0x7fc32c1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 -- 172.21.15.46:0/2508113406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 msgr2=0x7fc32c1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 0x7fc32c1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 -- 172.21.15.46:0/2508113406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc32c1a1300 con 0x7fc32c104600 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc331600640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.970+0000 7fc32be00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc31802c8f0 tx=0x7fc31802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.971+0000 7fc329600640 1 -- 172.21.15.46:0/2508113406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318041070 con 0x7fc32c104600 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.971+0000 7fc329600640 1 -- 172.21.15.46:0/2508113406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc318004450 con 0x7fc32c104600 2024-03-08T23:04:46.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.971+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c1a1590 con 0x7fc32c104600 2024-03-08T23:04:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.971+0000 7fc329600640 1 -- 172.21.15.46:0/2508113406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc318034440 con 0x7fc32c104600 2024-03-08T23:04:46.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.971+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc32c1a1ad0 con 0x7fc32c104600 2024-03-08T23:04:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc329600640 1 -- 172.21.15.46:0/2508113406 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc318044070 con 0x7fc32c104600 2024-03-08T23:04:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 Processor -- start 2024-03-08T23:04:46.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 -- start start 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8aed150 0x5601d8af1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8af22c0 0x5601d8af4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8a516f0 con 0x5601d8af22c0 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d89fa9e0 con 0x5601d8aed150 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.974+0000 7fc3363ac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8a4d790 con 0x5601d8aec760 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56016/0 (socket says 172.21.15.46:56016) 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8af22c0 0x5601d8af4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 -- 172.21.15.46:0/2692582830 learned_addr learned my addr 172.21.15.46:0/2692582830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc330c00640 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8aed150 0x5601d8af1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 -- 172.21.15.46:0/2692582830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8aed150 msgr2=0x5601d8af1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8aed150 0x5601d8af1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 -- 172.21.15.46:0/2692582830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8af22c0 msgr2=0x5601d8af4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8af22c0 0x5601d8af4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 -- 172.21.15.46:0/2692582830 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601d89ee720 con 0x5601d8aec760 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc331600640 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3240117e0 tx=0x7fc324011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c60e114472ea66f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.975+0000 7fc329600640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 0x7fc3140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.976+0000 7fc329600640 1 -- 172.21.15.46:0/2508113406 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc3180f07e0 con 0x7fc32c104600 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.976+0000 7fc330c00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 0x7fc3140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.976+0000 7fc31dc00640 1 -- 172.21.15.46:0/2692582830 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324011310 con 0x5601d8aec760 2024-03-08T23:04:46.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.976+0000 7fc31dc00640 1 -- 172.21.15.46:0/2692582830 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc324003e50 con 0x5601d8aec760 2024-03-08T23:04:46.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.976+0000 7fc31dc00640 1 -- 172.21.15.46:0/2692582830 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324004620 con 0x5601d8aec760 2024-03-08T23:04:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- 172.21.15.46:0/2692582830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 msgr2=0x5601d8aecb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc3240117e0 tx=0x7fc324011ca0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc330c00640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 0x7fc3140a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5601d8aed550 tx=0x7fc320005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- 172.21.15.46:0/2692582830 shutdown_connections 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8af22c0 0x5601d8af4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8aed150 0x5601d8af1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 --2- 172.21.15.46:0/2692582830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8aecb80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- 172.21.15.46:0/2692582830 >> 172.21.15.46:0/2692582830 conn(0x5601d8ae7cd0 msgr2=0x5601d8ae80d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- 172.21.15.46:0/2692582830 shutdown_connections 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- 172.21.15.46:0/2692582830 wait complete. 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 -- start start 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.977+0000 7fc3363ac740 1 Processor -- start 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8af6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8afdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8af5270 con 0x5601d8aed150 2024-03-08T23:04:46.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8a0de20 con 0x5601d8af22c0 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8af53f0 con 0x5601d8aec760 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc3363ac740 10 client.0 _ll_register_callbacks cb 0x5601d8a488d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc331600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8afdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57394/0 (socket says 172.21.15.46:57394) 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc32be00640 1 -- 172.21.15.46:0/1668607689 learned_addr learned my addr 172.21.15.46:0/1668607689 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.978+0000 7fc330c00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8af6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 -- 172.21.15.46:0/1668607689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 msgr2=0x5601d8af6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8af6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 -- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 msgr2=0x5601d8afdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8afdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 -- 172.21.15.46:0/1668607689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601d87662d0 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc331600640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8afdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc32be00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c5150 tx=0x7fc3180b92f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a2d05273ac2d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc312000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31804b070 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc3180bd3f0 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc324003e50 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc318004c60 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc3363ac740 10 client.14580 fetch_fsmap learned FSMap version 17 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.979+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5601d88306b0 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc324003e50 con 0x5601d8aed150 2024-03-08T23:04:46.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc3180f0cd0 con 0x5601d8aed150 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 10 client.14580 fetch_fsmap finished waiting for FSMap version 17 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 10 client.14580 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 20 client.14580 populate_metadata read hostname 'smithi046' 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b034e0 0x5601d8b02470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 4 client.14580 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601d8a5ec30 con 0x5601d8b034e0 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.980+0000 7fc330c00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b034e0 0x5601d8b02470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.981+0000 7fc330c00640 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b034e0 0x5601d8b02470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=704e8e442983a719 server_cookie=c2977427c03b2202 in_seq=0 out_seq=0 2024-03-08T23:04:46.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.981+0000 7fc312000640 10 client.14580 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:04:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc312000640 1 -- 172.21.15.46:0/1668607689 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5601d8a5ec30 con 0x5601d8b034e0 2024-03-08T23:04:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc312000640 10 client.14580 handle_command_reply: tid=0 2024-03-08T23:04:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 client.14580 shutdown 2024-03-08T23:04:46.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 msgr2=0x5601d8af6f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc3180c5150 tx=0x7fc3180b92f0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 shutdown_connections 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b034e0 0x5601d8b02470 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8afdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.982+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8af6f40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc312000640 0 client.14580 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc3363ac740 1 --2- 172.21.15.46:0/1668607689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8af6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 >> 172.21.15.46:0/1668607689 conn(0x5601d8ae7cd0 msgr2=0x5601d8ae6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 shutdown_connections 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc3363ac740 1 -- 172.21.15.46:0/1668607689 wait complete. 2024-03-08T23:04:46.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:46.983+0000 7fc3363ac740 20 client.14580 trim_cache size 0 max 16384 2024-03-08T23:04:47.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 Processor -- start 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 -- start start 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8afdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8af6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b02d00 con 0x5601d8aed150 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b03360 con 0x5601d8af22c0 2024-03-08T23:04:47.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.000+0000 7fc3363ac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b039c0 con 0x5601d8aec760 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8afdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56040/0 (socket says 172.21.15.46:56040) 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 -- 172.21.15.46:0/375203084 learned_addr learned my addr 172.21.15.46:0/375203084 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc32be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8afdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57398/0 (socket says 172.21.15.46:57398) 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc331600640 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8af6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 -- 172.21.15.46:0/375203084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 msgr2=0x5601d8af6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:47.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8af6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 -- 172.21.15.46:0/375203084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 msgr2=0x5601d8afdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8afdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 -- 172.21.15.46:0/375203084 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601d8ae3080 con 0x5601d8aec760 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.001+0000 7fc330c00640 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5601d8aedbc0 tx=0x7fc32001a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86c562d2f6a2dfc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc31d200640 1 -- 172.21.15.46:0/375203084 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32002a460 con 0x5601d8aec760 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc31d200640 1 -- 172.21.15.46:0/375203084 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc320003ea0 con 0x5601d8aec760 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc31d200640 1 -- 172.21.15.46:0/375203084 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32001fa80 con 0x5601d8aec760 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 -- 172.21.15.46:0/375203084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 msgr2=0x5601d8b0b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x5601d8aedbc0 tx=0x7fc32001a000 comp rx=0 tx=0).stop 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 -- 172.21.15.46:0/375203084 shutdown_connections 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8af6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aed150 0x5601d8afdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 --2- 172.21.15.46:0/375203084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aec760 0x5601d8b0b980 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.002+0000 7fc3363ac740 1 -- 172.21.15.46:0/375203084 >> 172.21.15.46:0/375203084 conn(0x5601d8ae7cd0 msgr2=0x5601d8ae80b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.003+0000 7fc3363ac740 1 -- 172.21.15.46:0/375203084 shutdown_connections 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.003+0000 7fc3363ac740 1 -- 172.21.15.46:0/375203084 wait complete. 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.003+0000 7fc3363ac740 1 -- start start 2024-03-08T23:04:47.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.003+0000 7fc3363ac740 1 Processor -- start 2024-03-08T23:04:47.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aed150 0x5601d8afb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8b590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b04680 con 0x5601d8aec760 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b04020 con 0x5601d8af22c0 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601d8b04ce0 con 0x5601d8aed150 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc3363ac740 10 client.0 _ll_register_callbacks cb 0x7fc324009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc331600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8b590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57406/0 (socket says 172.21.15.46:57406) 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 -- 172.21.15.46:0/754885445 learned_addr learned my addr 172.21.15.46:0/754885445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc32be00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aed150 0x5601d8afb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 -- 172.21.15.46:0/754885445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aed150 msgr2=0x5601d8afb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aed150 0x5601d8afb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 -- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 msgr2=0x5601d8b590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8b590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.004+0000 7fc330c00640 1 -- 172.21.15.46:0/754885445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601d8afe180 con 0x5601d8aec760 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc330c00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc320019040 tx=0x7fc32001f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcafe15c8b2d4f91 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc312000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc320003bc0 con 0x5601d8aec760 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc32002a460 con 0x5601d8aec760 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc320003ea0 con 0x5601d8aec760 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc320048050 con 0x5601d8aec760 2024-03-08T23:04:47.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc3363ac740 10 client.14586 fetch_fsmap learned FSMap version 17 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5601d8b5a5d0 con 0x5601d8aec760 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.005+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc320003ea0 con 0x5601d8aec760 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 17) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc32003d3f0 con 0x5601d8aec760 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 10 client.14586 fetch_fsmap finished waiting for FSMap version 17 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 10 client.14586 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 20 client.14586 populate_metadata read hostname 'smithi046' 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b5ea80 0x5601d8b5a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 4 client.14586 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5601d8a58f80 con 0x5601d8b5ea80 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.006+0000 7fc32be00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b5ea80 0x5601d8b5a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc32be00640 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b5ea80 0x5601d8b5a7f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bb2e58b4d2d7d5c server_cookie=4b2558c3553906b1 in_seq=0 out_seq=0 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc312000640 10 client.14586 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc312000640 1 -- 172.21.15.46:0/754885445 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5601d8a58f80 con 0x5601d8b5ea80 2024-03-08T23:04:47.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc312000640 10 client.14586 handle_command_reply: tid=0 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 client.14586 shutdown 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 msgr2=0x5601d8afab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc320019040 tx=0x7fc32001f440 comp rx=0 tx=0).stop 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 shutdown_connections 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5601d8b5ea80 0x5601d8b5a7f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5601d8af22c0 0x5601d8b590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5601d8aed150 0x5601d8afb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc312000640 0 client.14586 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 --2- 172.21.15.46:0/754885445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5601d8aec760 0x5601d8afab70 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 >> 172.21.15.46:0/754885445 conn(0x5601d8ae7cd0 msgr2=0x5601d8b0b980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 shutdown_connections 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 1 -- 172.21.15.46:0/754885445 wait complete. 2024-03-08T23:04:47.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.007+0000 7fc3363ac740 20 client.14586 trim_cache size 0 max 16384 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24578, 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 7.6525169450000003, 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 17, 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 89, 2024-03-08T23:04:47.017 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 88, 2024-03-08T23:04:47.018 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 15.207048052999999 2024-03-08T23:04:47.018 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:04:47.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 msgr2=0x7fc3140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 0x7fc3140a4d40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x5601d8aed550 tx=0x7fc320005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 msgr2=0x7fc32c199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc31802c8f0 tx=0x7fc31802c660 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 shutdown_connections 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc3140a2880 0x7fc3140a4d40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc32c104fc0 0x7fc32c1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc32c104600 0x7fc32c199c20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 --2- 172.21.15.46:0/2508113406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc32c103410 0x7fc32c1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 >> 172.21.15.46:0/2508113406 conn(0x7fc32c0fe9a0 msgr2=0x7fc32c1078a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 shutdown_connections 2024-03-08T23:04:47.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:47.008+0000 7fc332000640 1 -- 172.21.15.46:0/2508113406 wait complete. 2024-03-08T23:04:47.031 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T23:04:47.031 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:47.031 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-08T23:04:47.486 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:04:47.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:47 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1485921018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:47.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:47 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1485921018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:47.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:47 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1485921018' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:48.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:48 smithi046 ceph-mon[31859]: pgmap v323: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-08T23:04:48.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:48 smithi192 ceph-mon[36700]: pgmap v323: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-08T23:04:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:48 smithi062 ceph-mon[35761]: pgmap v323: 129 pgs: 3 creating+peering, 126 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 6 op/s 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 -- 172.21.15.46:0/3138719139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 msgr2=0x7f84d81081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 --2- 172.21.15.46:0/3138719139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d81081f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f84cc0099c0 tx=0x7f84cc009cd0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 -- 172.21.15.46:0/3138719139 shutdown_connections 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 --2- 172.21.15.46:0/3138719139 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 --2- 172.21.15.46:0/3138719139 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d81081f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 --2- 172.21.15.46:0/3138719139 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 0x7f84d81022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 -- 172.21.15.46:0/3138719139 >> 172.21.15.46:0/3138719139 conn(0x7f84d8078f10 msgr2=0x7f84d8079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:49.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 -- 172.21.15.46:0/3138719139 shutdown_connections 2024-03-08T23:04:49.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.899+0000 7f84dfc00700 1 -- 172.21.15.46:0/3138719139 wait complete. 2024-03-08T23:04:49.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.900+0000 7f84dfc00700 1 Processor -- start 2024-03-08T23:04:49.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 -- start start 2024-03-08T23:04:49.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 0x7f84d819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d819ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d81120c0 con 0x7f84d8107e10 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8111f40 con 0x7f84d8101e30 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dfc00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d8112240 con 0x7f84d81027f0 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84ddc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84d7e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d819ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84ddc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:57424/0 (socket says 172.21.15.46:57424) 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84ddc00700 1 -- 172.21.15.46:0/31299638 learned_addr learned my addr 172.21.15.46:0/31299638 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.901+0000 7f84dd200700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 0x7f84d819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 -- 172.21.15.46:0/31299638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 msgr2=0x7f84d819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 0x7f84d819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 -- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 msgr2=0x7f84d819ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:49.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 -- 172.21.15.46:0/31299638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d819f6d0 con 0x7f84d81027f0 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84ddc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d7e00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d819ac50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84c400e9f0 tx=0x7f84c400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.902+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c400c9c0 con 0x7f84d81027f0 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.903+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84c400cb60 con 0x7f84d81027f0 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.903+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c4018680 con 0x7f84d81027f0 2024-03-08T23:04:49.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.903+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d819f9c0 con 0x7f84d81027f0 2024-03-08T23:04:49.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.903+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d81a7240 con 0x7f84d81027f0 2024-03-08T23:04:49.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.904+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d8068aa0 con 0x7f84d81027f0 2024-03-08T23:04:49.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.906+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f84c4047a60 con 0x7f84d81027f0 2024-03-08T23:04:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.907+0000 7f84d5600700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 0x7f84c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.907+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f84c40ced30 con 0x7f84d81027f0 2024-03-08T23:04:49.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.907+0000 7f84dd200700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 0x7f84c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:49.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.908+0000 7f84dd200700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 0x7f84c00a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000c00 tx=0x7f84cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:49.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:49.909+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f84c409bb30 con 0x7f84d81027f0 2024-03-08T23:04:50.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.187+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f84d819ba20 con 0x7f84d81027f0 2024-03-08T23:04:50.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.187+0000 7f84d5600700 1 -- 172.21.15.46:0/31299638 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7f84c409b430 con 0x7f84d81027f0 2024-03-08T23:04:50.192 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-08T23:04:50.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.190+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 msgr2=0x7f84c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:50.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.190+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 0x7f84c00a4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000c00 tx=0x7f84cc005fb0 comp rx=0 tx=0).stop 2024-03-08T23:04:50.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.190+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 msgr2=0x7f84d819ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:50.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.190+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d819ac50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f84c400e9f0 tx=0x7f84c400ed00 comp rx=0 tx=0).stop 2024-03-08T23:04:50.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 shutdown_connections 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f84c00a2910 0x7f84c00a4dd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f84d8107e10 0x7f84d819ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f84d8101e30 0x7f84d819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 --2- 172.21.15.46:0/31299638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f84d81027f0 0x7f84d819ac50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 >> 172.21.15.46:0/31299638 conn(0x7f84d8078f10 msgr2=0x7f84d8100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 shutdown_connections 2024-03-08T23:04:50.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:50.191+0000 7f84dfc00700 1 -- 172.21.15.46:0/31299638 wait complete. 2024-03-08T23:04:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:50 smithi046 ceph-mon[31859]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-08T23:04:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:50 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/31299638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:04:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:50 smithi192 ceph-mon[36700]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-08T23:04:50.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:50 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/31299638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:04:50.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:50 smithi062 ceph-mon[35761]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s wr, 5 op/s 2024-03-08T23:04:50.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:50 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/31299638' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T23:04:51.033 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-08T23:04:51.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:51.034 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-03-08T23:04:51.096 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:04:51.097 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-03-08T23:04:51.098 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41988sec preferred_lft 41988sec 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:04:51.099 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-03-08T23:04:51.101 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-08T23:04:51.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:51.101 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:04:51.101 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:04:51.101 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-03-08T23:04:51.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-03-08T23:04:51.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-03-08T23:04:51.102 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-08T23:04:51.102 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:04:51.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:51.577 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T23:04:51.659 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-03-08T23:04:51.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:51.663 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-03-08T23:04:51.713 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-03-08T23:04:51.714 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T23:04:51.714 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-08T23:04:51.714 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-08T23:04:51.714 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T23:04:51.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:51.714 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:04:51.715 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:04:51.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-08T23:04:51.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-08T23:04:51.715 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-08T23:04:51.715 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:04:52.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:52.457 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-08T23:04:52.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:52.515 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-03-08T23:04:52.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:52.572 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:04:52.572 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:04:52.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:52.572 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-08T23:04:52.572 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:04:52.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:52 smithi046 ceph-mon[31859]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-08T23:04:52.723 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-08T23:04:52.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:52.723 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-08T23:04:52.724 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:04:52.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:52 smithi192 ceph-mon[36700]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-08T23:04:52.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:52 smithi062 ceph-mon[35761]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-03-08T23:04:53.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:53.021 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:04:53.021 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:04:53.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-03-08T23:04:53.021 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-03-08T23:04:53.021 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:04:53.192 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-08T23:04:53.192 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T23:04:53.192 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:53.238 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-08T23:04:53.238 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T23:04:53.239 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:53.336 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-08T23:04:53.438 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-08T23:04:53.491 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T23:04:53.492 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/933c0916359251e737ef45f4626fdd6ecae6ab26e6441905006cf9283829024b/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f27f48c0e7f05a070407c3391fa4205e6d58f9e02c40512942023c725b3a59c4/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/67210dfb37a1a63597316cc2419676e7351097a827dcced6941208a3d9defc70/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/81a9313dde7a155bfca7830c57caf2edf016cecd1949ab30cf3e26316c3552ea/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/e05d57485413a0d17c0c71d68a6f458b624e79d02a32ffa2beefead960cecde3/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f9b4de30c8b87d0c8516771686d231c0d280907b5fcf0055d8e05d9b92db7e70/merged 2024-03-08T23:04:53.493 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f25a7368af15532b11c621a718e98399135a6a5bb515a258a289e32966024ae2/merged 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/29bbd26b3b7d9865602a7f6bb837bf755cf833bc0f822f59a394ddbf3b94f3c2/merged 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9ff39a474decd24856c6269cba84d10abfb749d79092e2b0351ad90ccfd99597/merged 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/52e0e826d0be1befdd51aa555e41dd85f5ee8bd0c810c05fa56f77edc2a94ab8/merged 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:53.494 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:53.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:53.495 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-08T23:04:53.542 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-08T23:04:53.542 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-08T23:04:53.591 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-08T23:04:53.676 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-08T23:04:53.718 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-08T23:04:53.719 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-08T23:04:53.720 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/933c0916359251e737ef45f4626fdd6ecae6ab26e6441905006cf9283829024b/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f27f48c0e7f05a070407c3391fa4205e6d58f9e02c40512942023c725b3a59c4/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/67210dfb37a1a63597316cc2419676e7351097a827dcced6941208a3d9defc70/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/81a9313dde7a155bfca7830c57caf2edf016cecd1949ab30cf3e26316c3552ea/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/e05d57485413a0d17c0c71d68a6f458b624e79d02a32ffa2beefead960cecde3/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f9b4de30c8b87d0c8516771686d231c0d280907b5fcf0055d8e05d9b92db7e70/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f25a7368af15532b11c621a718e98399135a6a5bb515a258a289e32966024ae2/merged 2024-03-08T23:04:53.721 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/29bbd26b3b7d9865602a7f6bb837bf755cf833bc0f822f59a394ddbf3b94f3c2/merged 2024-03-08T23:04:53.722 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9ff39a474decd24856c6269cba84d10abfb749d79092e2b0351ad90ccfd99597/merged 2024-03-08T23:04:53.722 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/52e0e826d0be1befdd51aa555e41dd85f5ee8bd0c810c05fa56f77edc2a94ab8/merged 2024-03-08T23:04:53.722 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-08T23:04:53.722 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:53.722 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:53.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:53.723 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-08T23:04:53.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:2024-03-08T23:04:53.805+0000 7fc4dee10480 -1 init, newargv = 0x563c56983ed0 newargc=15 2024-03-08T23:04:53.809 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[72841]: starting ceph client 2024-03-08T23:04:53.820 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[72841]: starting fuse 2024-03-08T23:04:54.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:54 smithi046 ceph-mon[31859]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-08T23:04:54.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:54 smithi192 ceph-mon[36700]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-08T23:04:54.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:54 smithi062 ceph-mon[35761]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 3 op/s 2024-03-08T23:04:54.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo modprobe fuse 2024-03-08T23:04:55.023 DEBUG:teuthology.orchestra.run.smithi046:> cat /proc/self/mounts | awk '{print $2}' 2024-03-08T23:04:55.108 INFO:teuthology.orchestra.run.smithi046.stdout:/proc 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/security 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/shm 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/pts 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/run 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/cgroup 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/pstore 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/bpf 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/ 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/selinux 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/mqueue 2024-03-08T23:04:55.109 INFO:teuthology.orchestra.run.smithi046.stdout:/dev/hugepages 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/debug 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/tracing 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/fs/fuse/connections 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/sys/kernel/config 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-sysctl.service 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/nfs/rpc_pipefs 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/1000 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/ceph 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/933c0916359251e737ef45f4626fdd6ecae6ab26e6441905006cf9283829024b/merged 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f27f48c0e7f05a070407c3391fa4205e6d58f9e02c40512942023c725b3a59c4/merged 2024-03-08T23:04:55.110 INFO:teuthology.orchestra.run.smithi046.stdout:/run/user/0 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/proc/sys/fs/binfmt_misc 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/67210dfb37a1a63597316cc2419676e7351097a827dcced6941208a3d9defc70/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/81a9313dde7a155bfca7830c57caf2edf016cecd1949ab30cf3e26316c3552ea/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/e05d57485413a0d17c0c71d68a6f458b624e79d02a32ffa2beefead960cecde3/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f9b4de30c8b87d0c8516771686d231c0d280907b5fcf0055d8e05d9b92db7e70/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/f25a7368af15532b11c621a718e98399135a6a5bb515a258a289e32966024ae2/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/29bbd26b3b7d9865602a7f6bb837bf755cf833bc0f822f59a394ddbf3b94f3c2/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/9ff39a474decd24856c6269cba84d10abfb749d79092e2b0351ad90ccfd99597/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/var/lib/containers/storage/overlay/52e0e826d0be1befdd51aa555e41dd85f5ee8bd0c810c05fa56f77edc2a94ab8/merged 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:04:55.111 INFO:teuthology.orchestra.run.smithi046.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:55.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:55.113 DEBUG:teuthology.orchestra.run.smithi046:> ls /sys/fs/fuse/connections 2024-03-08T23:04:55.164 INFO:teuthology.orchestra.run.smithi046.stdout:118 2024-03-08T23:04:55.165 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo stdin-killer -- python3 -c ' 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> import glob 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> import re 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> import os 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> import subprocess 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-08T23:04:55.165 DEBUG:teuthology.orchestra.run.smithi046:> def _find_admin_socket(client_name): 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> files = glob.glob(asok_path) 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> # Given a non-glob path, it better be there 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> if "*" not in asok_path: 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> assert(len(files) == 1) 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> return files[0] 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> for f in files: 2024-03-08T23:04:55.166 DEBUG:teuthology.orchestra.run.smithi046:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> contents = proc_f.read() 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> if mountpoint in contents: 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> return f 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> print(_find_admin_socket("client.admin")) 2024-03-08T23:04:55.167 DEBUG:teuthology.orchestra.run.smithi046:> ' 2024-03-08T23:04:55.337 INFO:teuthology.orchestra.run.smithi046.stdout:/var/run/ceph/ceph-client.admin.72841.asok 2024-03-08T23:04:55.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:55 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-08T23:04:55.355 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72841.asok 2024-03-08T23:04:55.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:55.355 DEBUG:teuthology.orchestra.run.smithi046:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72841.asok status 2024-03-08T23:04:55.601 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "metadata": { 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_sha1": "7d63482aa89628d733e298984a762389b625c761", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "entity_id": "admin", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": "smithi046", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "pid": "72841", 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "root": "/" 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_count": 0, 2024-03-08T23:04:55.602 INFO:teuthology.orchestra.run.smithi046.stdout: "dentry_pinned_count": 0, 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 14598, 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "inst": { 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": { 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "client", 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "num": 14598 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4100811640 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": { 2024-03-08T23:04:55.603 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "192.168.0.1:0", 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4100811640 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "inst_str": "client.14598 192.168.0.1:0/4100811640", 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "addr_str": "192.168.0.1:0/4100811640", 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "inode_count": 1, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_epoch": 17, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch": 89, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_epoch_barrier": 0, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "blocklisted": false, 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs" 2024-03-08T23:04:55.604 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:04:55.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:55.618 DEBUG:teuthology.orchestra.run.smithi046:> stat --file-system '--printf=%T 2024-03-08T23:04:55.618 DEBUG:teuthology.orchestra.run.smithi046:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:55.674 INFO:teuthology.orchestra.run.smithi046.stdout:fuseblk 2024-03-08T23:04:55.674 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:55.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:55.674 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:04:55.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:04:55.777 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-08T23:04:55.876 INFO:teuthology.orchestra.run.smithi046.stdout:# file: . 2024-03-08T23:04:55.876 INFO:teuthology.orchestra.run.smithi046.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-08T23:04:55.876 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:55.877 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:04:55.877 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T23:04:55.935 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 -- 172.21.15.46:0/1496590547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0102ce0 msgr2=0x7f7af0109570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1496590547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0102ce0 0x7f7af0109570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7ae8009be0 tx=0x7f7ae802bef0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 -- 172.21.15.46:0/1496590547 shutdown_connections 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1496590547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0102ce0 0x7f7af0109570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1496590547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1496590547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0101130 0x7f7af0101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 -- 172.21.15.46:0/1496590547 >> 172.21.15.46:0/1496590547 conn(0x7f7af00fc680 msgr2=0x7f7af00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 -- 172.21.15.46:0/1496590547 shutdown_connections 2024-03-08T23:04:56.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.152+0000 7f7aefe00640 1 -- 172.21.15.46:0/1496590547 wait complete. 2024-03-08T23:04:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.153+0000 7f7aefe00640 1 Processor -- start 2024-03-08T23:04:56.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.153+0000 7f7aefe00640 1 -- start start 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0101130 0x7f7af0199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0102ce0 0x7f7af01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af010cde0 con 0x7f7af0102320 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af010cc60 con 0x7f7af0101130 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aefe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af010cf60 con 0x7f7af0102ce0 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aeea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0101130 0x7f7af0199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49360/0 (socket says 172.21.15.46:49360) 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 -- 172.21.15.46:0/1464475206 learned_addr learned my addr 172.21.15.46:0/1464475206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:56.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7aef400640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0102ce0 0x7f7af01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 -- 172.21.15.46:0/1464475206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0102ce0 msgr2=0x7f7af01a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0102ce0 0x7f7af01a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 -- 172.21.15.46:0/1464475206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0101130 msgr2=0x7f7af0199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0101130 0x7f7af0199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.154+0000 7f7ae6000640 1 -- 172.21.15.46:0/1464475206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af01a1210 con 0x7f7af0102320 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7ae6000640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800e9c0 tx=0x7f7ad800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad800ccc0 con 0x7f7af0102320 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad8005500 con 0x7f7af0102320 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8010430 con 0x7f7af0102320 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af01a1500 con 0x7f7af0102320 2024-03-08T23:04:56.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.155+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af01a1a00 con 0x7f7af0102320 2024-03-08T23:04:56.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.156+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af01a1cc0 con 0x7f7af0102320 2024-03-08T23:04:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.160+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7ad80046c0 con 0x7f7af0102320 2024-03-08T23:04:56.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.161+0000 7f7ae7e00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.161+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7ad80cf1d0 con 0x7f7af0102320 2024-03-08T23:04:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.162+0000 7f7aeea00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.163+0000 7f7aeea00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7af0102180 tx=0x7f7ae0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.164+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7ad8053260 con 0x7f7af0102320 2024-03-08T23:04:56.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.366+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7af01a1e90 con 0x7f7af0102320 2024-03-08T23:04:56.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.369+0000 7f7ae7e00640 1 -- 172.21.15.46:0/1464475206 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f7ad809bfd0 con 0x7f7af0102320 2024-03-08T23:04:56.374 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:56.375 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 msgr2=0x7f7ad40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7af0102180 tx=0x7f7ae0005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 msgr2=0x7f7af0199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800e9c0 tx=0x7f7ad800ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 shutdown_connections 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ad40a2800 0x7f7ad40a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7af0102ce0 0x7f7af01a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7af0102320 0x7f7af0199bd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 --2- 172.21.15.46:0/1464475206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7af0101130 0x7f7af0199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 >> 172.21.15.46:0/1464475206 conn(0x7f7af00fc680 msgr2=0x7f7af0073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 shutdown_connections 2024-03-08T23:04:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.372+0000 7f7aefe00640 1 -- 172.21.15.46:0/1464475206 wait complete. 2024-03-08T23:04:56.399 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:04:56.399 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:04:56.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- 172.21.15.46:0/4057404194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 msgr2=0x7f0b3010b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- 172.21.15.46:0/4057404194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b3010b950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009be0 tx=0x7f0b2802bef0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- 172.21.15.46:0/4057404194 shutdown_connections 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- 172.21.15.46:0/4057404194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b3010b950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- 172.21.15.46:0/4057404194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b30104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- 172.21.15.46:0/4057404194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b30103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- 172.21.15.46:0/4057404194 >> 172.21.15.46:0/4057404194 conn(0x7f0b300feaa0 msgr2=0x7f0b30100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- 172.21.15.46:0/4057404194 shutdown_connections 2024-03-08T23:04:56.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- 172.21.15.46:0/4057404194 wait complete. 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 Processor -- start 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 -- start start 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.629+0000 7f0b38000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b300795e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b38000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b30075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b38000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3010f1f0 con 0x7f0b30103510 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b38000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3010f070 con 0x7f0b30104700 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b38000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3010f370 con 0x7f0b301050c0 2024-03-08T23:04:56.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49376/0 (socket says 172.21.15.46:49376) 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 -- 172.21.15.46:0/4281248855 learned_addr learned my addr 172.21.15.46:0/4281248855 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b34c00640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b300795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b36000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b30075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 -- 172.21.15.46:0/4281248855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 msgr2=0x7f0b30075c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b30075c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 -- 172.21.15.46:0/4281248855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 msgr2=0x7f0b300795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b300795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b35600640 1 -- 172.21.15.46:0/4281248855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300764d0 con 0x7f0b30103510 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.630+0000 7f0b34c00640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b300795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b35600640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400e9a0 tx=0x7f0b2400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2400cd80 con 0x7f0b30103510 2024-03-08T23:04:56.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b301a5bd0 con 0x7f0b30103510 2024-03-08T23:04:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b24010910 con 0x7f0b30103510 2024-03-08T23:04:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b24018a00 con 0x7f0b30103510 2024-03-08T23:04:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.631+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b301a6030 con 0x7f0b30103510 2024-03-08T23:04:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.632+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3010a7d0 con 0x7f0b30103510 2024-03-08T23:04:56.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.634+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0b24010ac0 con 0x7f0b30103510 2024-03-08T23:04:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.637+0000 7f0b2e000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 0x7f0b180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.637+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0b240ceff0 con 0x7f0b30103510 2024-03-08T23:04:56.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.637+0000 7f0b34c00640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 0x7f0b180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:56.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.637+0000 7f0b34c00640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 0x7f0b180a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b30077020 tx=0x7f0b1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:56.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.640+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0b24053260 con 0x7f0b30103510 2024-03-08T23:04:56.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:56 smithi046 ceph-mon[31859]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s wr, 0 op/s 2024-03-08T23:04:56.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:56 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1464475206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:56.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:56 smithi192 ceph-mon[36700]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s wr, 0 op/s 2024-03-08T23:04:56.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:56 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1464475206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:56.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.845+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b30061ef0 con 0x7f0b30103510 2024-03-08T23:04:56.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.845+0000 7f0b2e000640 1 -- 172.21.15.46:0/4281248855 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b2409bdf0 con 0x7f0b30103510 2024-03-08T23:04:56.849 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:04:56.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 msgr2=0x7f0b180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 0x7f0b180a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0b30077020 tx=0x7f0b1c005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:56.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 msgr2=0x7f0b300790a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:56.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400e9a0 tx=0x7f0b2400ee60 comp rx=0 tx=0).stop 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 shutdown_connections 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b180a2800 0x7f0b180a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0b301050c0 0x7f0b30075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0b30104700 0x7f0b300795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 --2- 172.21.15.46:0/4281248855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0b30103510 0x7f0b300790a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 >> 172.21.15.46:0/4281248855 conn(0x7f0b300feaa0 msgr2=0x7f0b301079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 shutdown_connections 2024-03-08T23:04:56.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:56.848+0000 7f0b38000640 1 -- 172.21.15.46:0/4281248855 wait complete. 2024-03-08T23:04:56.870 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:04:56.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:56 smithi062 ceph-mon[35761]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 95 B/s wr, 0 op/s 2024-03-08T23:04:56.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:56 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1464475206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.089+0000 7fd907e00640 1 -- 172.21.15.46:0/927415316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081023c0 msgr2=0x7fd908102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.089+0000 7fd907e00640 1 --2- 172.21.15.46:0/927415316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081023c0 0x7fd908102840 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009900 tx=0x7fd8f002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 -- 172.21.15.46:0/927415316 shutdown_connections 2024-03-08T23:04:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 --2- 172.21.15.46:0/927415316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd908102d80 0x7fd908109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 --2- 172.21.15.46:0/927415316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081023c0 0x7fd908102840 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 --2- 172.21.15.46:0/927415316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9081011d0 0x7fd9081015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 -- 172.21.15.46:0/927415316 >> 172.21.15.46:0/927415316 conn(0x7fd9080fc760 msgr2=0x7fd9080feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 -- 172.21.15.46:0/927415316 shutdown_connections 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.090+0000 7fd907e00640 1 -- 172.21.15.46:0/927415316 wait complete. 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.091+0000 7fd907e00640 1 Processor -- start 2024-03-08T23:04:57.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.091+0000 7fd907e00640 1 -- start start 2024-03-08T23:04:57.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9081023c0 0x7fd908197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd908102d80 0x7fd90819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90810ceb0 con 0x7fd908102d80 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90810cd30 con 0x7fd9081011d0 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd90810d030 con 0x7fd9081023c0 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd906a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd906a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:42196/0 (socket says 172.21.15.46:42196) 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd906a00640 1 -- 172.21.15.46:0/2242292490 learned_addr learned my addr 172.21.15.46:0/2242292490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd907400640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd908102d80 0x7fd90819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.092+0000 7fd8ffe00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9081023c0 0x7fd908197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 -- 172.21.15.46:0/2242292490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9081023c0 msgr2=0x7fd908197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9081023c0 0x7fd908197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 -- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd908102d80 msgr2=0x7fd90819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd908102d80 0x7fd90819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 -- 172.21.15.46:0/2242292490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90819f120 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd906a00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800d950 tx=0x7fd8f800de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f8017020 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8f8009c30 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90819f410 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f800f720 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.093+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90819f930 con 0x7fd9081011d0 2024-03-08T23:04:57.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.094+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd908108490 con 0x7fd9081011d0 2024-03-08T23:04:57.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.096+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd8f801a020 con 0x7fd9081011d0 2024-03-08T23:04:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.098+0000 7fd8ff400640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 0x7fd8ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.098+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd8f80cefb0 con 0x7fd9081011d0 2024-03-08T23:04:57.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.098+0000 7fd8ffe00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 0x7fd8ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.099+0000 7fd8ffe00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 0x7fd8ec0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0000c00 tx=0x7fd8f002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:57.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.102+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd8f8021020 con 0x7fd9081011d0 2024-03-08T23:04:57.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.305+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd908061ef0 con 0x7fd9081011d0 2024-03-08T23:04:57.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.306+0000 7fd8ff400640 1 -- 172.21.15.46:0/2242292490 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8f80bb020 con 0x7fd9081011d0 2024-03-08T23:04:57.310 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:04:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.308+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 msgr2=0x7fd8ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.308+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 0x7fd8ec0a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0000c00 tx=0x7fd8f002df90 comp rx=0 tx=0).stop 2024-03-08T23:04:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.308+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 msgr2=0x7fd9081975a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.308+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd8f800d950 tx=0x7fd8f800de10 comp rx=0 tx=0).stop 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 shutdown_connections 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd8ec0a2800 0x7fd8ec0a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd908102d80 0x7fd90819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9081023c0 0x7fd908197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 --2- 172.21.15.46:0/2242292490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9081011d0 0x7fd9081975a0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 >> 172.21.15.46:0/2242292490 conn(0x7fd9080fc760 msgr2=0x7fd908105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 shutdown_connections 2024-03-08T23:04:57.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.309+0000 7fd907e00640 1 -- 172.21.15.46:0/2242292490 wait complete. 2024-03-08T23:04:57.333 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 -- 172.21.15.46:0/3715390577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f4103530 msgr2=0x7f26f4103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 --2- 172.21.15.46:0/3715390577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f4103530 0x7f26f4103950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f26e4009be0 tx=0x7f26e402bef0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 -- 172.21.15.46:0/3715390577 shutdown_connections 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 --2- 172.21.15.46:0/3715390577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f41050e0 0x7f26f410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 --2- 172.21.15.46:0/3715390577 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 --2- 172.21.15.46:0/3715390577 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f4103530 0x7f26f4103950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 -- 172.21.15.46:0/3715390577 >> 172.21.15.46:0/3715390577 conn(0x7f26f40feaa0 msgr2=0x7f26f4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 -- 172.21.15.46:0/3715390577 shutdown_connections 2024-03-08T23:04:57.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.550+0000 7f26fc200640 1 -- 172.21.15.46:0/3715390577 wait complete. 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.551+0000 7f26fc200640 1 Processor -- start 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.551+0000 7f26fc200640 1 -- start start 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f4103530 0x7f26f4199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f41050e0 0x7f26f41a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410f210 con 0x7f26f4104720 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410f090 con 0x7f26f41050e0 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fc200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f410f390 con 0x7f26f4103530 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26f8e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26f8e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49418/0 (socket says 172.21.15.46:49418) 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26f8e00640 1 -- 172.21.15.46:0/2130420006 learned_addr learned my addr 172.21.15.46:0/2130420006 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26f9800640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f4103530 0x7f26f4199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.552+0000 7f26fa200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f41050e0 0x7f26f41a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 -- 172.21.15.46:0/2130420006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f4103530 msgr2=0x7f26f4199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f4103530 0x7f26f4199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 -- 172.21.15.46:0/2130420006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f41050e0 msgr2=0x7f26f41a0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f41050e0 0x7f26f41a0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 -- 172.21.15.46:0/2130420006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f41a1460 con 0x7f26f4104720 2024-03-08T23:04:57.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f8e00640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e9c0 tx=0x7f26e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e000ccc0 con 0x7f26f4104720 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f41a1750 con 0x7f26f4104720 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26e0005500 con 0x7f26f4104720 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e0018430 con 0x7f26f4104720 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.553+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f41a1cf0 con 0x7f26f4104720 2024-03-08T23:04:57.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.554+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f410a850 con 0x7f26f4104720 2024-03-08T23:04:57.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.558+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f26e00046c0 con 0x7f26f4104720 2024-03-08T23:04:57.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.558+0000 7f26f2000640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:57.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.559+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f26e00cf120 con 0x7f26f4104720 2024-03-08T23:04:57.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.559+0000 7f26f9800640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:57.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.560+0000 7f26f9800640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f26e4034fd0 tx=0x7f26e4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:57.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.563+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f26e009bf20 con 0x7f26f4104720 2024-03-08T23:04:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4281248855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2242292490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:04:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.760+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f26f4061ef0 con 0x7f26f4104720 2024-03-08T23:04:57.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.760+0000 7f26f2000640 1 -- 172.21.15.46:0/2130420006 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f26e009b820 con 0x7f26f4104720 2024-03-08T23:04:57.765 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:04:57.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 msgr2=0x7f26d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 0x7f26d80a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f26e4034fd0 tx=0x7f26e4000f40 comp rx=0 tx=0).stop 2024-03-08T23:04:57.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 msgr2=0x7f26f4199d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:57.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f26e000e9c0 tx=0x7f26e000ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:57.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 shutdown_connections 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f26d80a2800 0x7f26d80a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f26f41050e0 0x7f26f41a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f26f4104720 0x7f26f4199d80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 --2- 172.21.15.46:0/2130420006 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26f4103530 0x7f26f4199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 >> 172.21.15.46:0/2130420006 conn(0x7f26f40feaa0 msgr2=0x7f26f4107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 shutdown_connections 2024-03-08T23:04:57.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:57.763+0000 7f26fc200640 1 -- 172.21.15.46:0/2130420006 wait complete. 2024-03-08T23:04:57.786 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:04:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:57 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4281248855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:57.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:57 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2242292490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:04:57.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:57 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4281248855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:04:57.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:57 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2242292490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.007+0000 7f0cdc400640 1 -- 172.21.15.46:0/1020637158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4076ef0 msgr2=0x7f0cd4077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.007+0000 7f0cdc400640 1 --2- 172.21.15.46:0/1020637158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4076ef0 0x7f0cd4077350 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0009900 tx=0x7f0cc002bc30 comp rx=0 tx=0).stop 2024-03-08T23:04:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 -- 172.21.15.46:0/1020637158 shutdown_connections 2024-03-08T23:04:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 --2- 172.21.15.46:0/1020637158 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4077890 0x7f0cd4078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 --2- 172.21.15.46:0/1020637158 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4076ef0 0x7f0cd4077350 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 --2- 172.21.15.46:0/1020637158 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd40760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 -- 172.21.15.46:0/1020637158 >> 172.21.15.46:0/1020637158 conn(0x7f0cd40feaa0 msgr2=0x7f0cd4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:58.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 -- 172.21.15.46:0/1020637158 shutdown_connections 2024-03-08T23:04:58.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.008+0000 7f0cdc400640 1 -- 172.21.15.46:0/1020637158 wait complete. 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.009+0000 7f0cdc400640 1 Processor -- start 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.009+0000 7f0cdc400640 1 -- start start 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.009+0000 7f0cdc400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.009+0000 7f0cdc400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cdc400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 0x7f0cd41a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cdc400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4113600 con 0x7f0cd4076ef0 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cdc400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4113480 con 0x7f0cd4075cb0 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cdc400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4113780 con 0x7f0cd4077890 2024-03-08T23:04:58.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:42238/0 (socket says 172.21.15.46:42238) 2024-03-08T23:04:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49434/0 (socket says 172.21.15.46:49434) 2024-03-08T23:04:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9a00640 1 -- 172.21.15.46:0/222484445 learned_addr learned my addr 172.21.15.46:0/222484445 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:58.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cda400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 0x7f0cd41a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 -- 172.21.15.46:0/222484445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 msgr2=0x7f0cd41a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 0x7f0cd41a5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 -- 172.21.15.46:0/222484445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 msgr2=0x7f0cd419dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9000640 1 -- 172.21.15.46:0/222484445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd41a5830 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cda400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 0x7f0cd41a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.010+0000 7f0cd9a00640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cd9000640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0036040 tx=0x7f0cc002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0043020 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cc00054e0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd41a5ac0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cc0003c30 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.011+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd41a5fe0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.012+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd410fff0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.016+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0cc0046020 con 0x7f0cd4076ef0 2024-03-08T23:04:58.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.017+0000 7f0cd2000640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 0x7f0cbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.017+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0cc00f13a0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.018+0000 7f0cd9a00640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 0x7f0cbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.018+0000 7f0cd9a00640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 0x7f0cbc0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0cd419f0f0 tx=0x7f0cc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:58.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.020+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0cc004d020 con 0x7f0cd4076ef0 2024-03-08T23:04:58.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.228+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0cd402d000 con 0x7f0cd4076ef0 2024-03-08T23:04:58.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.230+0000 7f0cd2000640 1 -- 172.21.15.46:0/222484445 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0cc00be1a0 con 0x7f0cd4076ef0 2024-03-08T23:04:58.233 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 msgr2=0x7f0cbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 0x7f0cbc0a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0cd419f0f0 tx=0x7f0cc8005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 msgr2=0x7f0cd419e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0cc0036040 tx=0x7f0cc002c650 comp rx=0 tx=0).stop 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 shutdown_connections 2024-03-08T23:04:58.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0cbc0a2800 0x7f0cbc0a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0cd4077890 0x7f0cd41a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0cd4076ef0 0x7f0cd419e160 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 --2- 172.21.15.46:0/222484445 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0cd4075cb0 0x7f0cd419dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.232+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 >> 172.21.15.46:0/222484445 conn(0x7f0cd40feaa0 msgr2=0x7f0cd4100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.233+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 shutdown_connections 2024-03-08T23:04:58.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.233+0000 7f0cdc400640 1 -- 172.21.15.46:0/222484445 wait complete. 2024-03-08T23:04:58.254 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 -- 172.21.15.46:0/227216025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c108a00 msgr2=0x7fb05c108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 --2- 172.21.15.46:0/227216025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c108a00 0x7fb05c108e80 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb048009900 tx=0x7fb04802bc40 comp rx=0 tx=0).stop 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 -- 172.21.15.46:0/227216025 shutdown_connections 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 --2- 172.21.15.46:0/227216025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c1093c0 0x7fb05c10fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 --2- 172.21.15.46:0/227216025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c108a00 0x7fb05c108e80 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 --2- 172.21.15.46:0/227216025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c107810 0x7fb05c107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.480+0000 7fb062200640 1 -- 172.21.15.46:0/227216025 >> 172.21.15.46:0/227216025 conn(0x7fb05c0767c0 msgr2=0x7fb05c078be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.481+0000 7fb062200640 1 -- 172.21.15.46:0/227216025 shutdown_connections 2024-03-08T23:04:58.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.481+0000 7fb062200640 1 -- 172.21.15.46:0/227216025 wait complete. 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.481+0000 7fb062200640 1 Processor -- start 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 -- start start 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 0x7fb05c19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c1093c0 0x7fb05c1a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c1134e0 con 0x7fb05c1093c0 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c113360 con 0x7fb05c108a00 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb062200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05c113660 con 0x7fb05c107810 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb060e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb061800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c1093c0 0x7fb05c1a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb05be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 0x7fb05c19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb060e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56270/0 (socket says 172.21.15.46:56270) 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.482+0000 7fb060e00640 1 -- 172.21.15.46:0/2195575263 learned_addr learned my addr 172.21.15.46:0/2195575263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 -- 172.21.15.46:0/2195575263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 msgr2=0x7fb05c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 -- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c1093c0 msgr2=0x7fb05c1a5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c1093c0 0x7fb05c1a5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 -- 172.21.15.46:0/2195575263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c1a56e0 con 0x7fb05c108a00 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb060e00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb05be00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 0x7fb05c19e000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb048036040 tx=0x7fb048005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048041070 con 0x7fb05c108a00 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb05c1a5970 con 0x7fb05c108a00 2024-03-08T23:04:58.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0480044d0 con 0x7fb05c108a00 2024-03-08T23:04:58.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb048034440 con 0x7fb05c108a00 2024-03-08T23:04:58.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.483+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb05c1a5e80 con 0x7fb05c108a00 2024-03-08T23:04:58.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.485+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb05c10eb30 con 0x7fb05c108a00 2024-03-08T23:04:58.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.487+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb048044070 con 0x7fb05c108a00 2024-03-08T23:04:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.488+0000 7fb059600640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 0x7fb0440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.488+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb0480f0d50 con 0x7fb05c108a00 2024-03-08T23:04:58.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.488+0000 7fb060e00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 0x7fb0440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:58.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.489+0000 7fb060e00640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 0x7fb0440a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0500104e0 tx=0x7fb05000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:58.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.490+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb0480bd500 con 0x7fb05c108a00 2024-03-08T23:04:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.706+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb05c02ce70 con 0x7fb05c108a00 2024-03-08T23:04:58.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.707+0000 7fb059600640 1 -- 172.21.15.46:0/2195575263 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7fb0480c15a0 con 0x7fb05c108a00 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3556325343 2024-03-09T22:56:39.034402+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1288083028 2024-03-09T22:56:39.034402+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1166845583 2024-03-09T22:56:05.453593+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2084846526 2024-03-09T22:56:05.453593+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/300257767 2024-03-09T22:56:05.453593+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:04:58.711 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/256680715 2024-03-09T22:56:39.034402+0000 2024-03-08T23:04:58.712 INFO:teuthology.orchestra.run.smithi046.stderr:listed 10 entries 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.710+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 msgr2=0x7fb0440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.710+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 0x7fb0440a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb0500104e0 tx=0x7fb05000b410 comp rx=0 tx=0).stop 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.710+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 msgr2=0x7fb05c19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.710+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 0x7fb05c19e000 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb048036040 tx=0x7fb048005d50 comp rx=0 tx=0).stop 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 shutdown_connections 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb0440a2800 0x7fb0440a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb05c1093c0 0x7fb05c1a5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb05c108a00 0x7fb05c19e000 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 --2- 172.21.15.46:0/2195575263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb05c107810 0x7fb05c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 >> 172.21.15.46:0/2195575263 conn(0x7fb05c0767c0 msgr2=0x7fb05c10bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 shutdown_connections 2024-03-08T23:04:58.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:58.711+0000 7fb062200640 1 -- 172.21.15.46:0/2195575263 wait complete. 2024-03-08T23:04:58.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:58 smithi046 ceph-mon[31859]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:04:58.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2130420006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/222484445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.734 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:04:58.734 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T23:04:58.792 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:04:58.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:58 smithi192 ceph-mon[36700]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:04:58.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:58 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2130420006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.809 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:58 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/222484445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:58 smithi062 ceph-mon[35761]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:04:58.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:58 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2130420006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:04:58.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:58 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/222484445' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 -- 172.21.15.46:0/108993961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54109020 msgr2=0x7f1e5410b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 --2- 172.21.15.46:0/108993961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54109020 0x7f1e5410b4b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009860 tx=0x7f1e4c02bb70 comp rx=0 tx=0).stop 2024-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 -- 172.21.15.46:0/108993961 shutdown_connections 2024-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 --2- 172.21.15.46:0/108993961 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54109020 0x7f1e5410b4b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 --2- 172.21.15.46:0/108993961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e540741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 --2- 172.21.15.46:0/108993961 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54075330 0x7f1e54073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 -- 172.21.15.46:0/108993961 >> 172.21.15.46:0/108993961 conn(0x7f1e540fc6d0 msgr2=0x7f1e540feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 -- 172.21.15.46:0/108993961 shutdown_connections 2024-03-08T23:04:59.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.022+0000 7f1e53200640 1 -- 172.21.15.46:0/108993961 wait complete. 2024-03-08T23:04:59.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.023+0000 7f1e53200640 1 Processor -- start 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.023+0000 7f1e53200640 1 -- start start 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e541995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54075330 0x7f1e54199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5410efa0 con 0x7f1e54109020 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5410ee20 con 0x7f1e54075330 2024-03-08T23:04:59.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e53200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e5410f120 con 0x7f1e54073d50 2024-03-08T23:04:59.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49486/0 (socket says 172.21.15.46:49486) 2024-03-08T23:04:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e51e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e541995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 -- 172.21.15.46:0/1041714162 learned_addr learned my addr 172.21.15.46:0/1041714162 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:59.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e51400640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54075330 0x7f1e54199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 -- 172.21.15.46:0/1041714162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 msgr2=0x7f1e541995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e541995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 -- 172.21.15.46:0/1041714162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54075330 msgr2=0x7f1e54199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54075330 0x7f1e54199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e52800640 1 -- 172.21.15.46:0/1041714162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e541a1210 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.024+0000 7f1e51e00640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e541995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e52800640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c02c180 tx=0x7f1e4c032460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c041020 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e541a14a0 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e4c02c9f0 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c039040 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.025+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e541a19e0 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.026+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e54107d50 con 0x7f1e54109020 2024-03-08T23:04:59.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.028+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1e4c044020 con 0x7f1e54109020 2024-03-08T23:04:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.030+0000 7f1e4aa00640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 0x7f1e340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.030+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1e4c0f0680 con 0x7f1e54109020 2024-03-08T23:04:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.030+0000 7f1e51e00640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 0x7f1e340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.031+0000 7f1e51e00640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 0x7f1e340a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e44005fd0 tx=0x7f1e44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.034+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1e4c04b020 con 0x7f1e54109020 2024-03-08T23:04:59.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.240+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e54061ef0 con 0x7f1e54109020 2024-03-08T23:04:59.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.242+0000 7f1e4aa00640 1 -- 172.21.15.46:0/1041714162 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f1e4c0bd480 con 0x7f1e54109020 2024-03-08T23:04:59.256 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:04:59.257 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:04:59.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 msgr2=0x7f1e340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 0x7f1e340a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e44005fd0 tx=0x7f1e44009260 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 msgr2=0x7f1e541a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c02c180 tx=0x7f1e4c032460 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 shutdown_connections 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1e340a2800 0x7f1e340a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1e54109020 0x7f1e541a0ba0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1e54075330 0x7f1e54199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 --2- 172.21.15.46:0/1041714162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e54073d50 0x7f1e541995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 >> 172.21.15.46:0/1041714162 conn(0x7f1e540fc6d0 msgr2=0x7f1e54104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 shutdown_connections 2024-03-08T23:04:59.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.245+0000 7f1e53200640 1 -- 172.21.15.46:0/1041714162 wait complete. 2024-03-08T23:04:59.277 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:04:59.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 -- 172.21.15.46:0/2060068864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c81050c0 msgr2=0x7fe0c810b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 --2- 172.21.15.46:0/2060068864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c81050c0 0x7fe0c810b950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe0c0009ca0 tx=0x7fe0c002c0f0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 -- 172.21.15.46:0/2060068864 shutdown_connections 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 --2- 172.21.15.46:0/2060068864 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c81050c0 0x7fe0c810b950 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 --2- 172.21.15.46:0/2060068864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c8104700 0x7fe0c8104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 --2- 172.21.15.46:0/2060068864 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8103510 0x7fe0c8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 -- 172.21.15.46:0/2060068864 >> 172.21.15.46:0/2060068864 conn(0x7fe0c80feaa0 msgr2=0x7fe0c8100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 -- 172.21.15.46:0/2060068864 shutdown_connections 2024-03-08T23:04:59.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.510+0000 7fe0cde00640 1 -- 172.21.15.46:0/2060068864 wait complete. 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.511+0000 7fe0cde00640 1 Processor -- start 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 -- start start 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8104700 0x7fe0c8199ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 0x7fe0c81a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c810f1a0 con 0x7fe0c8103510 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c810f020 con 0x7fe0c8104700 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0cde00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c810f320 con 0x7fe0c81050c0 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0c6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0c6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49506/0 (socket says 172.21.15.46:49506) 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0c6a00640 1 -- 172.21.15.46:0/3086876386 learned_addr learned my addr 172.21.15.46:0/3086876386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0bfe00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8104700 0x7fe0c8199ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.512+0000 7fe0c7400640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 0x7fe0c81a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 -- 172.21.15.46:0/3086876386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 msgr2=0x7fe0c81a0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 0x7fe0c81a0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 -- 172.21.15.46:0/3086876386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8104700 msgr2=0x7fe0c8199ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8104700 0x7fe0c8199ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 -- 172.21.15.46:0/3086876386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c81a13c0 con 0x7fe0c8103510 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c7400640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 0x7fe0c81a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0c6a00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800e9c0 tx=0x7fe0b800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b800cd80 con 0x7fe0c8103510 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0b80035f0 con 0x7fe0c8103510 2024-03-08T23:04:59.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c81a16b0 con 0x7fe0c8103510 2024-03-08T23:04:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8010690 con 0x7fe0c8103510 2024-03-08T23:04:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.513+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c81a1c00 con 0x7fe0c8103510 2024-03-08T23:04:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.515+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c810a7d0 con 0x7fe0c8103510 2024-03-08T23:04:59.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.516+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe0b80037a0 con 0x7fe0c8103510 2024-03-08T23:04:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.519+0000 7fe0cca00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 0x7fe0ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.519+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe0b80cf0b0 con 0x7fe0c8103510 2024-03-08T23:04:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.519+0000 7fe0bfe00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 0x7fe0ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.520+0000 7fe0bfe00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 0x7fe0ac0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0c819acb0 tx=0x7fe0b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.524+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe0b809beb0 con 0x7fe0c8103510 2024-03-08T23:04:59.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:59 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2195575263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:04:59.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:04:59 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1041714162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:59.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.726+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0c8061f20 con 0x7fe0c8103510 2024-03-08T23:04:59.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.727+0000 7fe0cca00640 1 -- 172.21.15.46:0/3086876386 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0b809b7b0 con 0x7fe0c8103510 2024-03-08T23:04:59.731 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 msgr2=0x7fe0ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 0x7fe0ac0a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe0c819acb0 tx=0x7fe0b0005f90 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 msgr2=0x7fe0c81997a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe0b800e9c0 tx=0x7fe0b800ee80 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 shutdown_connections 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe0ac0a2800 0x7fe0ac0a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe0c81050c0 0x7fe0c81a0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe0c8104700 0x7fe0c8199ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 --2- 172.21.15.46:0/3086876386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe0c8103510 0x7fe0c81997a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 >> 172.21.15.46:0/3086876386 conn(0x7fe0c80feaa0 msgr2=0x7fe0c81079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:59.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 shutdown_connections 2024-03-08T23:04:59.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.730+0000 7fe0cde00640 1 -- 172.21.15.46:0/3086876386 wait complete. 2024-03-08T23:04:59.751 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:04:59.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:59 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2195575263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:04:59.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:04:59 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1041714162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:59.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:59 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2195575263' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:04:59.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:04:59 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1041714162' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:04:59.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 -- 172.21.15.46:0/3285003432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 msgr2=0x7f7680104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 --2- 172.21.15.46:0/3285003432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680104a80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f766c009900 tx=0x7f766c02bc40 comp rx=0 tx=0).stop 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 -- 172.21.15.46:0/3285003432 shutdown_connections 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 --2- 172.21.15.46:0/3285003432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680104fc0 0x7f768010b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 --2- 172.21.15.46:0/3285003432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680104a80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 --2- 172.21.15.46:0/3285003432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680103410 0x7f7680103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 -- 172.21.15.46:0/3285003432 >> 172.21.15.46:0/3285003432 conn(0x7f76800fe9a0 msgr2=0x7f7680100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 -- 172.21.15.46:0/3285003432 shutdown_connections 2024-03-08T23:04:59.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.982+0000 7f7687000640 1 -- 172.21.15.46:0/3285003432 wait complete. 2024-03-08T23:04:59.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.983+0000 7f7687000640 1 Processor -- start 2024-03-08T23:04:59.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.983+0000 7f7687000640 1 -- start start 2024-03-08T23:04:59.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.983+0000 7f7687000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 0x7f76801996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.983+0000 7f7687000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.983+0000 7f7687000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680104fc0 0x7f76801a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7687000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768010f0a0 con 0x7f7680104600 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7687000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768010ef20 con 0x7f7680104fc0 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7687000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768010f220 con 0x7f7680103410 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 0x7f76801996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49528/0 (socket says 172.21.15.46:49528) 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685200640 1 -- 172.21.15.46:0/1295592664 learned_addr learned my addr 172.21.15.46:0/1295592664 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7686600640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680104fc0 0x7f76801a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 -- 172.21.15.46:0/1295592664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680104fc0 msgr2=0x7f76801a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680104fc0 0x7f76801a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 -- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 msgr2=0x7f7680199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:04:59.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:04:59.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685c00640 1 -- 172.21.15.46:0/1295592664 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76801a12e0 con 0x7f7680103410 2024-03-08T23:04:59.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.984+0000 7f7685200640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f7685c00640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 0x7f76801996c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9c0 tx=0x7f767400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767400cd80 con 0x7f7680103410 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7674010910 con 0x7f7680103410 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76801a15d0 con 0x7f7680103410 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7674018980 con 0x7f7680103410 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.985+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76801a1af0 con 0x7f7680103410 2024-03-08T23:04:59.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.986+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f768010a7b0 con 0x7f7680103410 2024-03-08T23:04:59.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.988+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f767401f020 con 0x7f7680103410 2024-03-08T23:04:59.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.989+0000 7f767e000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 0x7f76680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:04:59.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.990+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f76740cecf0 con 0x7f7680103410 2024-03-08T23:04:59.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.990+0000 7f7685200640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 0x7f76680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:04:59.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.990+0000 7f7685200640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 0x7f76680a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c02c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:04:59.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:04:59.994+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f767409baf0 con 0x7f7680103410 2024-03-08T23:05:00.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.202+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7680061ef0 con 0x7f7680103410 2024-03-08T23:05:00.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.203+0000 7f767e000640 1 -- 172.21.15.46:0/1295592664 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f767409b3f0 con 0x7f7680103410 2024-03-08T23:05:00.206 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 msgr2=0x7f76680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 0x7f76680a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f766c000c00 tx=0x7f766c02c890 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 msgr2=0x7f76801996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 0x7f76801996c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9c0 tx=0x7f767400ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 shutdown_connections 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76680a2800 0x7f76680a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7680104fc0 0x7f76801a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7680104600 0x7f7680199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 --2- 172.21.15.46:0/1295592664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7680103410 0x7f76801996c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.205+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 >> 172.21.15.46:0/1295592664 conn(0x7f76800fe9a0 msgr2=0x7f7680107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.206+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 shutdown_connections 2024-03-08T23:05:00.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.206+0000 7f7687000640 1 -- 172.21.15.46:0/1295592664 wait complete. 2024-03-08T23:05:00.230 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:05:00.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 -- 172.21.15.46:0/2181781210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc103410 msgr2=0x7f1acc103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/2181781210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc103410 0x7f1acc103830 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009860 tx=0x7f1abc02bb70 comp rx=0 tx=0).stop 2024-03-08T23:05:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 -- 172.21.15.46:0/2181781210 shutdown_connections 2024-03-08T23:05:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/2181781210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc104fc0 0x7f1acc10b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/2181781210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 0x7f1acc104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/2181781210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc103410 0x7f1acc103830 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 -- 172.21.15.46:0/2181781210 >> 172.21.15.46:0/2181781210 conn(0x7f1acc0fe9a0 msgr2=0x7f1acc100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 -- 172.21.15.46:0/2181781210 shutdown_connections 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.459+0000 7f1ad0a00640 1 -- 172.21.15.46:0/2181781210 wait complete. 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.460+0000 7f1ad0a00640 1 Processor -- start 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.460+0000 7f1ad0a00640 1 -- start start 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 0x7f1acc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc104fc0 0x7f1acc1a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc10f0a0 con 0x7f1acc103410 2024-03-08T23:05:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc10ef20 con 0x7f1acc104600 2024-03-08T23:05:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ad0a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc10f220 con 0x7f1acc104fc0 2024-03-08T23:05:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49556/0 (socket says 172.21.15.46:49556) 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 -- 172.21.15.46:0/1733104384 learned_addr learned my addr 172.21.15.46:0/1733104384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acbe00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc104fc0 0x7f1acc1a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1ac3e00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 0x7f1acc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 -- 172.21.15.46:0/1733104384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc104fc0 msgr2=0x7f1acc1a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc104fc0 0x7f1acc1a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 -- 172.21.15.46:0/1733104384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 msgr2=0x7f1acc199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 0x7f1acc199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.461+0000 7f1acb400640 1 -- 172.21.15.46:0/1733104384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc1a1350 con 0x7f1acc103410 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1acb400640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1abc02c0b0 tx=0x7f1abc032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc041020 con 0x7f1acc103410 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1abc02c9f0 con 0x7f1acc103410 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1acc1a15e0 con 0x7f1acc103410 2024-03-08T23:05:00.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1abc039040 con 0x7f1acc103410 2024-03-08T23:05:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.462+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1acc1a1ad0 con 0x7f1acc103410 2024-03-08T23:05:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.463+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1acc10a750 con 0x7f1acc103410 2024-03-08T23:05:00.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.465+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1abc044020 con 0x7f1acc103410 2024-03-08T23:05:00.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.467+0000 7f1ac8c00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 0x7f1ab00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.467+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1abc0f05e0 con 0x7f1acc103410 2024-03-08T23:05:00.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.467+0000 7f1ac3e00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 0x7f1ab00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.468+0000 7f1ac3e00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 0x7f1ab00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40095f0 tx=0x7f1ab4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:00.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.471+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1abc04b020 con 0x7f1acc103410 2024-03-08T23:05:00.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.678+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1acc061ef0 con 0x7f1acc103410 2024-03-08T23:05:00.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.679+0000 7f1ac8c00640 1 -- 172.21.15.46:0/1733104384 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1abc0bd3e0 con 0x7f1acc103410 2024-03-08T23:05:00.682 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:05:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 msgr2=0x7f1ab00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 0x7f1ab00a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40095f0 tx=0x7f1ab4005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 msgr2=0x7f1acc199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1abc02c0b0 tx=0x7f1abc032430 comp rx=0 tx=0).stop 2024-03-08T23:05:00.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 shutdown_connections 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1ab00a2800 0x7f1ab00a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1acc104fc0 0x7f1acc1a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1acc104600 0x7f1acc199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 --2- 172.21.15.46:0/1733104384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1acc103410 0x7f1acc199730 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 >> 172.21.15.46:0/1733104384 conn(0x7f1acc0fe9a0 msgr2=0x7f1acc107920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.682+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 shutdown_connections 2024-03-08T23:05:00.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.683+0000 7f1ad0a00640 1 -- 172.21.15.46:0/1733104384 wait complete. 2024-03-08T23:05:00.706 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:05:00.784 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:00 smithi046 ceph-mon[31859]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:05:00.784 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:00 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3086876386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.785 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:00 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1295592664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:00 smithi192 ceph-mon[36700]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:05:00.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:00 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3086876386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:00 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1295592664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:00 smithi062 ceph-mon[35761]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-08T23:05:00.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:00 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3086876386' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:00 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1295592664' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:05:00.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 -- 172.21.15.46:0/2839684434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a101023e0 msgr2=0x7f9a10102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 --2- 172.21.15.46:0/2839684434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a101023e0 0x7f9a10102860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009900 tx=0x7f99f802bc30 comp rx=0 tx=0).stop 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 -- 172.21.15.46:0/2839684434 shutdown_connections 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 --2- 172.21.15.46:0/2839684434 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a10102da0 0x7f9a10109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 --2- 172.21.15.46:0/2839684434 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a101023e0 0x7f9a10102860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 --2- 172.21.15.46:0/2839684434 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 -- 172.21.15.46:0/2839684434 >> 172.21.15.46:0/2839684434 conn(0x7f9a100fc760 msgr2=0x7f9a100feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 -- 172.21.15.46:0/2839684434 shutdown_connections 2024-03-08T23:05:00.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.923+0000 7f9a15c00640 1 -- 172.21.15.46:0/2839684434 wait complete. 2024-03-08T23:05:00.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.924+0000 7f9a15c00640 1 Processor -- start 2024-03-08T23:05:00.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.924+0000 7f9a15c00640 1 -- start start 2024-03-08T23:05:00.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a101023e0 0x7f9a10197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a10102da0 0x7f9a1019ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1010ce80 con 0x7f9a10102da0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1010cd00 con 0x7f9a101023e0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1010d000 con 0x7f9a101011f0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56332/0 (socket says 172.21.15.46:56332) 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 -- 172.21.15.46:0/897593622 learned_addr learned my addr 172.21.15.46:0/897593622 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a15200640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a10102da0 0x7f9a1019ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0f400640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a101023e0 0x7f9a10197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 -- 172.21.15.46:0/897593622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a101023e0 msgr2=0x7f9a10197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a101023e0 0x7f9a10197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 -- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a10102da0 msgr2=0x7f9a1019ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a10102da0 0x7f9a1019ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.925+0000 7f9a0fe00640 1 -- 172.21.15.46:0/897593622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1019f0d0 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a0fe00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800ea80 tx=0x7f9a0800ef40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0800ce40 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a08011950 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0801ab80 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1019f3c0 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.926+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a1019f890 con 0x7f9a101011f0 2024-03-08T23:05:00.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.927+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a101084b0 con 0x7f9a101011f0 2024-03-08T23:05:00.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.929+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9a08011b00 con 0x7f9a101011f0 2024-03-08T23:05:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.930+0000 7f9a0cc00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 0x7f99f40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.931+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9a080cf2d0 con 0x7f9a101011f0 2024-03-08T23:05:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.931+0000 7f9a0f400640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 0x7f99f40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:00.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.932+0000 7f9a0f400640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 0x7f99f40a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009230 tx=0x7f99f8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:00.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:00.935+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9a08016030 con 0x7f9a101011f0 2024-03-08T23:05:01.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.137+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9a10061ef0 con 0x7f9a101011f0 2024-03-08T23:05:01.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.138+0000 7f9a0cc00640 1 -- 172.21.15.46:0/897593622 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9a0809c0d0 con 0x7f9a101011f0 2024-03-08T23:05:01.142 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:05:01.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 msgr2=0x7f99f40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 0x7f99f40a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009230 tx=0x7f99f8005fd0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 msgr2=0x7f9a10197550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9a0800ea80 tx=0x7f9a0800ef40 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 shutdown_connections 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99f40a2800 0x7f99f40a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9a10102da0 0x7f9a1019ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9a101023e0 0x7f9a10197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 --2- 172.21.15.46:0/897593622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9a101011f0 0x7f9a10197550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 >> 172.21.15.46:0/897593622 conn(0x7f9a100fc760 msgr2=0x7f9a10105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 shutdown_connections 2024-03-08T23:05:01.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.142+0000 7f9a15c00640 1 -- 172.21.15.46:0/897593622 wait complete. 2024-03-08T23:05:01.165 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T23:05:01.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.380+0000 7f6c2b000640 1 -- 172.21.15.46:0/2974910499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c24073d50 msgr2=0x7f6c240741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.380+0000 7f6c2b000640 1 --2- 172.21.15.46:0/2974910499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c24073d50 0x7f6c240741d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c009be0 tx=0x7f6c0c02bef0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 -- 172.21.15.46:0/2974910499 shutdown_connections 2024-03-08T23:05:01.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 --2- 172.21.15.46:0/2974910499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c241091c0 0x7f6c2410b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 --2- 172.21.15.46:0/2974910499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c24073d50 0x7f6c240741d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 --2- 172.21.15.46:0/2974910499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 0x7f6c24073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 -- 172.21.15.46:0/2974910499 >> 172.21.15.46:0/2974910499 conn(0x7f6c240fc890 msgr2=0x7f6c240fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:01.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 -- 172.21.15.46:0/2974910499 shutdown_connections 2024-03-08T23:05:01.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.381+0000 7f6c2b000640 1 -- 172.21.15.46:0/2974910499 wait complete. 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 Processor -- start 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 -- start start 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 0x7f6c24104b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c241091c0 0x7f6c24101190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2410f180 con 0x7f6c24073d50 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2410f000 con 0x7f6c24075330 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.382+0000 7f6c2b000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2410f300 con 0x7f6c241091c0 2024-03-08T23:05:01.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49588/0 (socket says 172.21.15.46:49588) 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 -- 172.21.15.46:0/1164427602 learned_addr learned my addr 172.21.15.46:0/1164427602 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c29000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c241091c0 0x7f6c24101190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23400640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 0x7f6c24104b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 -- 172.21.15.46:0/1164427602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c241091c0 msgr2=0x7f6c24101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c241091c0 0x7f6c24101190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 -- 172.21.15.46:0/1164427602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 msgr2=0x7f6c24104b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 0x7f6c24104b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 -- 172.21.15.46:0/1164427602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24101a50 con 0x7f6c24073d50 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.383+0000 7f6c23e00640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400e9f0 tx=0x7f6c1400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.384+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400cd80 con 0x7f6c24073d50 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.384+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c14005500 con 0x7f6c24073d50 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.384+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c241a5cc0 con 0x7f6c24073d50 2024-03-08T23:05:01.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.384+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14010690 con 0x7f6c24073d50 2024-03-08T23:05:01.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.384+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c241a60d0 con 0x7f6c24073d50 2024-03-08T23:05:01.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.385+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24107ef0 con 0x7f6c24073d50 2024-03-08T23:05:01.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.387+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6c140046c0 con 0x7f6c24073d50 2024-03-08T23:05:01.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.388+0000 7f6c20c00640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 0x7f6c080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:01.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.388+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6c140cf000 con 0x7f6c24073d50 2024-03-08T23:05:01.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.389+0000 7f6c23400640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 0x7f6c080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:01.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.390+0000 7f6c23400640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 0x7f6c080a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c241025a0 tx=0x7f6c0c000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:01.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.394+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6c1409be60 con 0x7f6c24073d50 2024-03-08T23:05:01.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.608+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6c2402ce70 con 0x7f6c24073d50 2024-03-08T23:05:01.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.609+0000 7f6c20c00640 1 -- 172.21.15.46:0/1164427602 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+590 (secure 0 0 0) 0x7f6c1409b760 con 0x7f6c24073d50 2024-03-08T23:05:01.614 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3556325343 2024-03-09T22:56:39.034402+0000 2024-03-08T23:05:01.614 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1288083028 2024-03-09T22:56:39.034402+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1166845583 2024-03-09T22:56:05.453593+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2084846526 2024-03-09T22:56:05.453593+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/300257767 2024-03-09T22:56:05.453593+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:05:01.615 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/256680715 2024-03-09T22:56:39.034402+0000 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:listed 10 entries 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.611+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 msgr2=0x7f6c080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.611+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 0x7f6c080a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c241025a0 tx=0x7f6c0c000f40 comp rx=0 tx=0).stop 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.611+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 msgr2=0x7f6c24104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.611+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6c1400e9f0 tx=0x7f6c1400edf0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 shutdown_connections 2024-03-08T23:05:01.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6c080a2880 0x7f6c080a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c241091c0 0x7f6c24101190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c24075330 0x7f6c24104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 --2- 172.21.15.46:0/1164427602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c24073d50 0x7f6c24104620 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 >> 172.21.15.46:0/1164427602 conn(0x7f6c240fc890 msgr2=0x7f6c241050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 shutdown_connections 2024-03-08T23:05:01.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:01.612+0000 7f6c2b000640 1 -- 172.21.15.46:0/1164427602 wait complete. 2024-03-08T23:05:01.636 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi046.front.sepia.ceph.com... 2024-03-08T23:05:01.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:05:01.636 DEBUG:teuthology.orchestra.run.smithi046:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-08T23:05:01.722 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi046.stderr:ceph-fuse[72841]: fuse finished with error 0 and tester_r 0 2024-03-08T23:05:01.729 INFO:teuthology.orchestra.run:waiting for 300 2024-03-08T23:05:01.792 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:01 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1733104384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:05:01.792 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:01 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/897593622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:05:01.800 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:05:01.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:05:01.800 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-08T23:05:01.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:01 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1733104384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:05:01.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:01 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/897593622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:05:01.866 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:05:01.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:01 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1733104384' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:05:01.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:01 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/897593622' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:05:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.094+0000 7fd55ac00640 1 -- 172.21.15.46:0/2383169441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554101310 msgr2=0x7fd554101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.094+0000 7fd55ac00640 1 --2- 172.21.15.46:0/2383169441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554101310 0x7fd554101730 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd544009900 tx=0x7fd54402bc40 comp rx=0 tx=0).stop 2024-03-08T23:05:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 -- 172.21.15.46:0/2383169441 shutdown_connections 2024-03-08T23:05:02.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 --2- 172.21.15.46:0/2383169441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554102ec0 0x7fd554109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 --2- 172.21.15.46:0/2383169441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102500 0x7fd554102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 --2- 172.21.15.46:0/2383169441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554101310 0x7fd554101730 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 -- 172.21.15.46:0/2383169441 >> 172.21.15.46:0/2383169441 conn(0x7fd5540fc8a0 msgr2=0x7fd5540fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 -- 172.21.15.46:0/2383169441 shutdown_connections 2024-03-08T23:05:02.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.095+0000 7fd55ac00640 1 -- 172.21.15.46:0/2383169441 wait complete. 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.096+0000 7fd55ac00640 1 Processor -- start 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.096+0000 7fd55ac00640 1 -- start start 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 0x7fd5541998b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 0x7fd554199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 0x7fd5541a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55410cfc0 con 0x7fd554101310 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55410ce40 con 0x7fd554102500 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd55ac00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd55410d140 con 0x7fd554102ec0 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd553e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 0x7fd5541998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 0x7fd554199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd553e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 0x7fd5541998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49606/0 (socket says 172.21.15.46:49606) 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd553e00640 1 -- 172.21.15.46:0/3819364729 learned_addr learned my addr 172.21.15.46:0/3819364729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd558c00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 0x7fd5541a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 -- 172.21.15.46:0/3819364729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 msgr2=0x7fd5541a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 0x7fd5541a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 -- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 msgr2=0x7fd5541998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 0x7fd5541998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd54b400640 1 -- 172.21.15.46:0/3819364729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5541a14d0 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.097+0000 7fd558c00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 0x7fd5541a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd54b400640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 0x7fd554199df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00e9f0 tx=0x7fd53c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c00cd80 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd53c0035f0 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd53c010690 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5541a17c0 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.098+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5541a1d60 con 0x7fd554102500 2024-03-08T23:05:02.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.099+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5541a1fc0 con 0x7fd554102500 2024-03-08T23:05:02.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.101+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd53c003b30 con 0x7fd554102500 2024-03-08T23:05:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.103+0000 7fd551600640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 0x7fd5380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.103+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fd53c0ceff0 con 0x7fd554102500 2024-03-08T23:05:02.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.103+0000 7fd553e00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 0x7fd5380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.104+0000 7fd553e00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 0x7fd5380a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd544005ab0 tx=0x7fd54402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:02.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.107+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd53c09bdf0 con 0x7fd554102500 2024-03-08T23:05:02.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.312+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5541a2230 con 0x7fd554102500 2024-03-08T23:05:02.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.315+0000 7fd551600640 1 -- 172.21.15.46:0/3819364729 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7fd53c09b6f0 con 0x7fd554102500 2024-03-08T23:05:02.319 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:02.319 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:05:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.317+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 msgr2=0x7fd5380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.317+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 0x7fd5380a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd544005ab0 tx=0x7fd54402cf90 comp rx=0 tx=0).stop 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.317+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 msgr2=0x7fd554199df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.317+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 0x7fd554199df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd53c00e9f0 tx=0x7fd53c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 shutdown_connections 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5380a2880 0x7fd5380a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd554102ec0 0x7fd5541a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd554102500 0x7fd554199df0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 --2- 172.21.15.46:0/3819364729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd554101310 0x7fd5541998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 >> 172.21.15.46:0/3819364729 conn(0x7fd5540fc8a0 msgr2=0x7fd554073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:02.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 shutdown_connections 2024-03-08T23:05:02.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.318+0000 7fd55ac00640 1 -- 172.21.15.46:0/3819364729 wait complete. 2024-03-08T23:05:02.343 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:05:02.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 -- 172.21.15.46:0/588920944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c1094e0 msgr2=0x7f1d8c10fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 --2- 172.21.15.46:0/588920944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c10fd70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009ca0 tx=0x7f1d8402c100 comp rx=0 tx=0).stop 2024-03-08T23:05:02.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 -- 172.21.15.46:0/588920944 shutdown_connections 2024-03-08T23:05:02.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 --2- 172.21.15.46:0/588920944 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c10fd70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 --2- 172.21.15.46:0/588920944 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c108b20 0x7f1d8c108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 --2- 172.21.15.46:0/588920944 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 -- 172.21.15.46:0/588920944 >> 172.21.15.46:0/588920944 conn(0x7f1d8c076900 msgr2=0x7f1d8c078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 -- 172.21.15.46:0/588920944 shutdown_connections 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.558+0000 7f1d91000640 1 -- 172.21.15.46:0/588920944 wait complete. 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 Processor -- start 2024-03-08T23:05:02.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 -- start start 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c1a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c113590 con 0x7f1d8c108b20 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.559+0000 7f1d91000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c113410 con 0x7f1d8c107930 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d91000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c113710 con 0x7f1d8c1094e0 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49634/0 (socket says 172.21.15.46:49634) 2024-03-08T23:05:02.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89400640 1 -- 172.21.15.46:0/1896051540 learned_addr learned my addr 172.21.15.46:0/1896051540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d8a800640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c1a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 -- 172.21.15.46:0/1896051540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c1094e0 msgr2=0x7f1d8c1a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c1a5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 -- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 msgr2=0x7f1d8c19e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89e00640 1 -- 172.21.15.46:0/1896051540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c1a5790 con 0x7f1d8c107930 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.560+0000 7f1d89400640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:02.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d89e00640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c19db70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00e9c0 tx=0x7f1d7c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c00cd80 con 0x7f1d8c107930 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d7c010910 con 0x7f1d8c107930 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c1a5a80 con 0x7f1d8c107930 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d7c018980 con 0x7f1d8c107930 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.561+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c1a6020 con 0x7f1d8c107930 2024-03-08T23:05:02.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.562+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c10eb80 con 0x7f1d8c107930 2024-03-08T23:05:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.564+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1d7c01f020 con 0x7f1d8c107930 2024-03-08T23:05:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.565+0000 7f1d82a00640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 0x7f1d700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.565+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1d7c0cec40 con 0x7f1d8c107930 2024-03-08T23:05:02.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.566+0000 7f1d89400640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 0x7f1d700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:02.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.567+0000 7f1d89400640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 0x7f1d700a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d740095f0 tx=0x7f1d74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:02.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.570+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1d7c09ba40 con 0x7f1d8c107930 2024-03-08T23:05:02.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:02 smithi046 ceph-mon[31859]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:02.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:02 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1164427602' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:05:02.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:02 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3819364729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:02.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.774+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d8c061ef0 con 0x7f1d8c107930 2024-03-08T23:05:02.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.776+0000 7f1d82a00640 1 -- 172.21.15.46:0/1896051540 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f1d7c080e40 con 0x7f1d8c107930 2024-03-08T23:05:02.779 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:02.780 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.778+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 msgr2=0x7f1d700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.778+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 0x7f1d700a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1d740095f0 tx=0x7f1d74005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 msgr2=0x7f1d8c19db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c19db70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1d7c00e9c0 tx=0x7f1d7c00ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 shutdown_connections 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1d700a2880 0x7f1d700a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1d8c1094e0 0x7f1d8c1a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1d8c108b20 0x7f1d8c19e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 --2- 172.21.15.46:0/1896051540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1d8c107930 0x7f1d8c19db70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:02.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 >> 172.21.15.46:0/1896051540 conn(0x7f1d8c076900 msgr2=0x7f1d8c10bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:02.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 shutdown_connections 2024-03-08T23:05:02.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:02.779+0000 7f1d91000640 1 -- 172.21.15.46:0/1896051540 wait complete. 2024-03-08T23:05:02.792 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:02 smithi062 ceph-mon[35761]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:02.792 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:02 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1164427602' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:05:02.792 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:02 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3819364729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:02.801 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-08T23:05:02.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:02 smithi192 ceph-mon[36700]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:02.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:02 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1164427602' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:05:02.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:02 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3819364729' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 -- 172.21.15.46:0/3700114603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 msgr2=0x7f7a6c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 --2- 172.21.15.46:0/3700114603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c102970 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a58009900 tx=0x7f7a5802bc30 comp rx=0 tx=0).stop 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 -- 172.21.15.46:0/3700114603 shutdown_connections 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 --2- 172.21.15.46:0/3700114603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 --2- 172.21.15.46:0/3700114603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c102970 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 --2- 172.21.15.46:0/3700114603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c101300 0x7f7a6c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 -- 172.21.15.46:0/3700114603 >> 172.21.15.46:0/3700114603 conn(0x7f7a6c0fc890 msgr2=0x7f7a6c0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 -- 172.21.15.46:0/3700114603 shutdown_connections 2024-03-08T23:05:03.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.034+0000 7f7a73600640 1 -- 172.21.15.46:0/3700114603 wait complete. 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.035+0000 7f7a73600640 1 Processor -- start 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.035+0000 7f7a73600640 1 -- start start 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c10cfe0 con 0x7f7a6c1024f0 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c10ce60 con 0x7f7a6c101300 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a73600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c10d160 con 0x7f7a6c102eb0 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:54780/0 (socket says 172.21.15.46:54780) 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 -- 172.21.15.46:0/1435973576 learned_addr learned my addr 172.21.15.46:0/1435973576 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a6be00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a71600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 -- 172.21.15.46:0/1435973576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 msgr2=0x7f7a6c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 -- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 msgr2=0x7f7a6c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a70c00640 1 -- 172.21.15.46:0/1435973576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c19f200 con 0x7f7a6c101300 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a71600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:05:03.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.036+0000 7f7a6be00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a70c00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000e9c0 tx=0x7f7a6000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6000cde0 con 0x7f7a6c101300 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a60010910 con 0x7f7a6c101300 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a6c19f4f0 con 0x7f7a6c101300 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a60018a00 con 0x7f7a6c101300 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.037+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a6c19fa60 con 0x7f7a6c101300 2024-03-08T23:05:03.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.038+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6c1086a0 con 0x7f7a6c101300 2024-03-08T23:05:03.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.040+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7a6001f020 con 0x7f7a6c101300 2024-03-08T23:05:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.042+0000 7f7a69600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 0x7f7a540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.042+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7a600cef10 con 0x7f7a6c101300 2024-03-08T23:05:03.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.042+0000 7f7a6be00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 0x7f7a540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.043+0000 7f7a6be00640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 0x7f7a540a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a5802c240 tx=0x7f7a5802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:03.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.046+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7a6009bd10 con 0x7f7a6c101300 2024-03-08T23:05:03.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.313+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7a6c02ce70 con 0x7f7a6c101300 2024-03-08T23:05:03.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.512+0000 7f7a69600640 1 -- 172.21.15.46:0/1435973576 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7a6009b610 con 0x7f7a6c101300 2024-03-08T23:05:03.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 msgr2=0x7f7a540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 0x7f7a540a4d40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a5802c240 tx=0x7f7a5802df90 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 msgr2=0x7f7a6c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7a6000e9c0 tx=0x7f7a6000ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 shutdown_connections 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7a540a2880 0x7f7a540a4d40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a6c102eb0 0x7f7a6c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a6c1024f0 0x7f7a6c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 --2- 172.21.15.46:0/1435973576 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a6c101300 0x7f7a6c1975e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.515+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 >> 172.21.15.46:0/1435973576 conn(0x7f7a6c0fc890 msgr2=0x7f7a6c105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.516+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 shutdown_connections 2024-03-08T23:05:03.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.516+0000 7f7a73600640 1 -- 172.21.15.46:0/1435973576 wait complete. 2024-03-08T23:05:03.540 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated'}} 2024-03-08T23:05:03.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-03-08T23:05:03.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:03 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1896051540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:03.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:03 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1435973576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:03.624 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:03 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.767+0000 7f39e5400640 1 -- 172.21.15.46:0/2847043227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 msgr2=0x7f39e010b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.767+0000 7f39e5400640 1 --2- 172.21.15.46:0/2847043227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e010b830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009ca0 tx=0x7f39d802c100 comp rx=0 tx=0).stop 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 -- 172.21.15.46:0/2847043227 shutdown_connections 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 --2- 172.21.15.46:0/2847043227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e010b830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 --2- 172.21.15.46:0/2847043227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e0104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 --2- 172.21.15.46:0/2847043227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 -- 172.21.15.46:0/2847043227 >> 172.21.15.46:0/2847043227 conn(0x7f39e00fe9a0 msgr2=0x7f39e0100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:03.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 -- 172.21.15.46:0/2847043227 shutdown_connections 2024-03-08T23:05:03.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.768+0000 7f39e5400640 1 -- 172.21.15.46:0/2847043227 wait complete. 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 Processor -- start 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 -- start start 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e00795e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e0075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010f080 con 0x7f39e01045e0 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010ef00 con 0x7f39e0104fa0 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e5400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e010f200 con 0x7f39e01033f0 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39e4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e0075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.769+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33320/0 (socket says 172.21.15.46:33320) 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 -- 172.21.15.46:0/3379508246 learned_addr learned my addr 172.21.15.46:0/3379508246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39d7e00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e00795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 -- 172.21.15.46:0/3379508246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 msgr2=0x7f39e0075c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e0075c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 -- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 msgr2=0x7f39e00795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e00795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 -- 172.21.15.46:0/3379508246 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e00764d0 con 0x7f39e01033f0 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39d7e00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e00795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39e4a00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e0075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e9c0 tx=0x7f39d000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:03.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d000cd80 con 0x7f39e01033f0 2024-03-08T23:05:03.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.770+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d00035f0 con 0x7f39e01033f0 2024-03-08T23:05:03.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.771+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0010690 con 0x7f39e01033f0 2024-03-08T23:05:03.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.771+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e01a5ac0 con 0x7f39e01033f0 2024-03-08T23:05:03.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.771+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e01a5ed0 con 0x7f39e01033f0 2024-03-08T23:05:03.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.772+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e010a6b0 con 0x7f39e01033f0 2024-03-08T23:05:03.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.774+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f39d00037a0 con 0x7f39e01033f0 2024-03-08T23:05:03.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.775+0000 7f39dd600640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 0x7f39c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:03.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.775+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f39d00cf090 con 0x7f39e01033f0 2024-03-08T23:05:03.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.775+0000 7f39d7e00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 0x7f39c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:03.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.776+0000 7f39d7e00640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 0x7f39c40a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39e0077020 tx=0x7f39c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:03.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:03.779+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f39d009be90 con 0x7f39e01033f0 2024-03-08T23:05:03.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:03 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1896051540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:03.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:03 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1435973576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:03.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:03 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:03.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:03 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1896051540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:03.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:03 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1435973576' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:03.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:03 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-08T23:05:04.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.027+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f39e00766a0 con 0x7f39e01033f0 2024-03-08T23:05:04.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.062+0000 7f39dd600640 1 -- 172.21.15.46:0/3379508246 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f39d009b790 con 0x7f39e01033f0 2024-03-08T23:05:04.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 msgr2=0x7f39c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:04.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 0x7f39c40a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f39e0077020 tx=0x7f39c8005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:04.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 msgr2=0x7f39e00790a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:04.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f39d000e9c0 tx=0x7f39d000ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 shutdown_connections 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f39c40a2880 0x7f39c40a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0104fa0 0x7f39e0075c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e01045e0 0x7f39e00795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 --2- 172.21.15.46:0/3379508246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e01033f0 0x7f39e00790a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.065+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 >> 172.21.15.46:0/3379508246 conn(0x7f39e00fe9a0 msgr2=0x7f39e0107880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.066+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 shutdown_connections 2024-03-08T23:05:04.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.066+0000 7f39e5400640 1 -- 172.21.15.46:0/3379508246 wait complete. 2024-03-08T23:05:04.091 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 -- 172.21.15.46:0/1182073643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0103410 msgr2=0x7f0ad0103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 --2- 172.21.15.46:0/1182073643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0103410 0x7f0ad0103830 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0ac0009860 tx=0x7f0ac002bb70 comp rx=0 tx=0).stop 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 -- 172.21.15.46:0/1182073643 shutdown_connections 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 --2- 172.21.15.46:0/1182073643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0104fc0 0x7f0ad010b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 --2- 172.21.15.46:0/1182073643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104600 0x7f0ad0104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 --2- 172.21.15.46:0/1182073643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0103410 0x7f0ad0103830 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.272+0000 7f0acf200640 1 -- 172.21.15.46:0/1182073643 >> 172.21.15.46:0/1182073643 conn(0x7f0ad00fe9a0 msgr2=0x7f0ad0100dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.273+0000 7f0acf200640 1 -- 172.21.15.46:0/1182073643 shutdown_connections 2024-03-08T23:05:04.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.273+0000 7f0acf200640 1 -- 172.21.15.46:0/1182073643 wait complete. 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 Processor -- start 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 -- start start 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0104600 0x7f0ad0199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 0x7f0ad01a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:04.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad010f070 con 0x7f0ad0103410 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad010eef0 con 0x7f0ad0104600 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acf200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad010f1f0 con 0x7f0ad0104fc0 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.274+0000 7f0acde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60754/0 (socket says 172.21.15.46:60754) 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 -- 172.21.15.46:0/3458427324 learned_addr learned my addr 172.21.15.46:0/3458427324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acd400640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0104600 0x7f0ad0199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0ace800640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 0x7f0ad01a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 -- 172.21.15.46:0/3458427324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 msgr2=0x7f0ad01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 0x7f0ad01a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 -- 172.21.15.46:0/3458427324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0104600 msgr2=0x7f0ad0199c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:04.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0104600 0x7f0ad0199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 -- 172.21.15.46:0/3458427324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad01a1340 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0ace800640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 0x7f0ad01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.275+0000 7f0acde00640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0ac002c0b0 tx=0x7f0ac0032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.276+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0041020 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.276+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ad01a15d0 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.276+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac002c9f0 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.276+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac0039040 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.276+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ad01a1b10 con 0x7f0ad0103410 2024-03-08T23:05:04.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.277+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad010a7b0 con 0x7f0ad0103410 2024-03-08T23:05:04.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.279+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ac0044020 con 0x7f0ad0103410 2024-03-08T23:05:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.281+0000 7f0ac6a00640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 0x7f0ab40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.281+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f0ac00f0680 con 0x7f0ad0103410 2024-03-08T23:05:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.281+0000 7f0acd400640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 0x7f0ab40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:04.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.282+0000 7f0acd400640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 0x7f0ab40a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0ad019ac30 tx=0x7f0ab8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:04.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.285+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0ac004b020 con 0x7f0ad0103410 2024-03-08T23:05:04.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:04.526+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f0ad0061ef0 con 0x7f0ad0103410 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3379508246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-08T23:05:04.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:04 smithi192 ceph-mon[36700]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3379508246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-08T23:05:04.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:04 smithi062 ceph-mon[35761]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-08T23:05:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-08T23:05:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-08T23:05:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-08T23:05:04.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3379508246' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: daemon mds.a assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: fsmap cephfs:2 {0=b=up:active,1=a=up:creating} 10 up:standby 2024-03-08T23:05:04.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:04 smithi046 ceph-mon[31859]: daemon mds.a is now active in filesystem cephfs as rank 1 2024-03-08T23:05:05.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.062+0000 7f0ac6a00640 1 -- 172.21.15.46:0/3458427324 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f0ac00bd480 con 0x7f0ad0103410 2024-03-08T23:05:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 msgr2=0x7f0ab40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 0x7f0ab40a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0ad019ac30 tx=0x7f0ab8009260 comp rx=0 tx=0).stop 2024-03-08T23:05:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 msgr2=0x7f0ad0199720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:05.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0ac002c0b0 tx=0x7f0ac0032430 comp rx=0 tx=0).stop 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 shutdown_connections 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ab40a2800 0x7f0ab40a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ad0104fc0 0x7f0ad01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ad0104600 0x7f0ad0199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 --2- 172.21.15.46:0/3458427324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ad0103410 0x7f0ad0199720 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 >> 172.21.15.46:0/3458427324 conn(0x7f0ad00fe9a0 msgr2=0x7f0ad0107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 shutdown_connections 2024-03-08T23:05:05.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.066+0000 7f0acf200640 1 -- 172.21.15.46:0/3458427324 wait complete. 2024-03-08T23:05:05.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.297+0000 7fa4cb200640 1 -- 172.21.15.46:0/3483043143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 msgr2=0x7fa4cc104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.297+0000 7fa4cb200640 1 --2- 172.21.15.46:0/3483043143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc104040 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc009ca0 tx=0x7fa4bc02c100 comp rx=0 tx=0).stop 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 -- 172.21.15.46:0/3483043143 shutdown_connections 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 --2- 172.21.15.46:0/3483043143 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 --2- 172.21.15.46:0/3483043143 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 0x7fa4cc106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 --2- 172.21.15.46:0/3483043143 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc104040 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 -- 172.21.15.46:0/3483043143 >> 172.21.15.46:0/3483043143 conn(0x7fa4cc0fb660 msgr2=0x7fa4cc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 -- 172.21.15.46:0/3483043143 shutdown_connections 2024-03-08T23:05:05.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.298+0000 7fa4cb200640 1 -- 172.21.15.46:0/3483043143 wait complete. 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 Processor -- start 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 -- start start 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 0x7fa4cc199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc10ce60 con 0x7fa4cc101c50 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc10cce0 con 0x7fa4cc104580 2024-03-08T23:05:05.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.299+0000 7fa4cb200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc10cfe0 con 0x7fa4cc106eb0 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60772/0 (socket says 172.21.15.46:60772) 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 -- 172.21.15.46:0/1482412731 learned_addr learned my addr 172.21.15.46:0/1482412731 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4ca800640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9400640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 0x7fa4cc199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 -- 172.21.15.46:0/1482412731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 msgr2=0x7fa4cc1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc1a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 -- 172.21.15.46:0/1482412731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 msgr2=0x7fa4cc199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 0x7fa4cc199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 -- 172.21.15.46:0/1482412731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc1a1310 con 0x7fa4cc101c50 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4ca800640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.300+0000 7fa4c9e00640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0325f0 tx=0x7fa4bc035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.301+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc039040 con 0x7fa4cc101c50 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.301+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4bc044070 con 0x7fa4cc101c50 2024-03-08T23:05:05.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.301+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4cc1a15a0 con 0x7fa4cc101c50 2024-03-08T23:05:05.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.301+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4bc0044d0 con 0x7fa4cc101c50 2024-03-08T23:05:05.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.301+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4cc1a1a60 con 0x7fa4cc101c50 2024-03-08T23:05:05.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.302+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4cc108a00 con 0x7fa4cc101c50 2024-03-08T23:05:05.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.304+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa4bc004670 con 0x7fa4cc101c50 2024-03-08T23:05:05.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.305+0000 7fa4c2a00640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 0x7fa4b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:05.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.305+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa4bc0f1250 con 0x7fa4cc101c50 2024-03-08T23:05:05.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.305+0000 7fa4c9400640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 0x7fa4b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:05.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.306+0000 7fa4c9400640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 0x7fa4b00a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80095f0 tx=0x7fa4b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:05.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.310+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa4bc0be050 con 0x7fa4cc101c50 2024-03-08T23:05:05.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:05.555+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fa4cc02ce70 con 0x7fa4cc101c50 2024-03-08T23:05:05.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:active 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-08T23:05:05.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:05 smithi192 ceph-mon[36700]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-08T23:05:05.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:active 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-08T23:05:05.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:05 smithi062 ceph-mon[35761]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-08T23:05:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-08T23:05:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3458427324' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-08T23:05:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] up:active 2024-03-08T23:05:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-08T23:05:05.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-08T23:05:05.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:05:05.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: fsmap cephfs:2 {0=b=up:active,1=a=up:active} 10 up:standby 2024-03-08T23:05:05.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:creating} 9 up:standby 2024-03-08T23:05:05.977 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:05 smithi046 ceph-mon[31859]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-03-08T23:05:06.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.073+0000 7fa4c2a00640 1 -- 172.21.15.46:0/1482412731 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7fa4bc0bd950 con 0x7fa4cc101c50 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.077+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 msgr2=0x7fa4b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.077+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 0x7fa4b00a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa4b80095f0 tx=0x7fa4b8005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.077+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 msgr2=0x7fa4cc1996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.077+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc0325f0 tx=0x7fa4bc035d10 comp rx=0 tx=0).stop 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 shutdown_connections 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa4b00a2800 0x7fa4b00a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4cc106eb0 0x7fa4cc1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4cc104580 0x7fa4cc199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 --2- 172.21.15.46:0/1482412731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4cc101c50 0x7fa4cc1996f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 >> 172.21.15.46:0/1482412731 conn(0x7fa4cc0fb660 msgr2=0x7fa4cc101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:06.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 shutdown_connections 2024-03-08T23:05:06.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.078+0000 7fa4cb200640 1 -- 172.21.15.46:0/1482412731 wait complete. 2024-03-08T23:05:06.103 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated'} subvols for filesystem 'cephfs' 2024-03-08T23:05:06.103 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-08T23:05:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.324+0000 7fa2ad400640 1 -- 172.21.15.46:0/4141952115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 msgr2=0x7fa2a8078130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.324+0000 7fa2ad400640 1 --2- 172.21.15.46:0/4141952115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a8078130 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0009900 tx=0x7fa2a002bc30 comp rx=0 tx=0).stop 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.324+0000 7fa2ad400640 1 -- 172.21.15.46:0/4141952115 shutdown_connections 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.324+0000 7fa2ad400640 1 --2- 172.21.15.46:0/4141952115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a8078130 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.325+0000 7fa2ad400640 1 --2- 172.21.15.46:0/4141952115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8076ef0 0x7fa2a8077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.325+0000 7fa2ad400640 1 --2- 172.21.15.46:0/4141952115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8075cb0 0x7fa2a80760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.325+0000 7fa2ad400640 1 -- 172.21.15.46:0/4141952115 >> 172.21.15.46:0/4141952115 conn(0x7fa2a80feaa0 msgr2=0x7fa2a8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.325+0000 7fa2ad400640 1 -- 172.21.15.46:0/4141952115 shutdown_connections 2024-03-08T23:05:06.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.325+0000 7fa2ad400640 1 -- 172.21.15.46:0/4141952115 wait complete. 2024-03-08T23:05:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 Processor -- start 2024-03-08T23:05:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 -- start start 2024-03-08T23:05:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8075cb0 0x7fa2a819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8076ef0 0x7fa2a819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a81135c0 con 0x7fa2a8077890 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8113440 con 0x7fa2a8076ef0 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2ad400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8113740 con 0x7fa2a8075cb0 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2a7200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2a7200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60792/0 (socket says 172.21.15.46:60792) 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2a7200640 1 -- 172.21.15.46:0/652213242 learned_addr learned my addr 172.21.15.46:0/652213242 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2a5e00640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8076ef0 0x7fa2a819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.326+0000 7fa2a6800640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8075cb0 0x7fa2a819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 -- 172.21.15.46:0/652213242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8075cb0 msgr2=0x7fa2a819dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8075cb0 0x7fa2a819dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 -- 172.21.15.46:0/652213242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8076ef0 msgr2=0x7fa2a819e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8076ef0 0x7fa2a819e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 -- 172.21.15.46:0/652213242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a81a57e0 con 0x7fa2a8077890 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa2a7200640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005ab0 tx=0x7fa2a002c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0041070 con 0x7fa2a8077890 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.327+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2a0004450 con 0x7fa2a8077890 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.328+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a81a5a70 con 0x7fa2a8077890 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.328+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a0003e60 con 0x7fa2a8077890 2024-03-08T23:05:06.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.328+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a81a5f60 con 0x7fa2a8077890 2024-03-08T23:05:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.329+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a8068560 con 0x7fa2a8077890 2024-03-08T23:05:06.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.331+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa2a0044070 con 0x7fa2a8077890 2024-03-08T23:05:06.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.332+0000 7fa29f400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 0x7fa28c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.333+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa2a00f0910 con 0x7fa2a8077890 2024-03-08T23:05:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.333+0000 7fa2a6800640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 0x7fa28c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.334+0000 7fa2a6800640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 0x7fa28c0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa2980099a0 tx=0x7fa298009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:06.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.336+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa2a8068560 con 0x7fa2a8077890 2024-03-08T23:05:06.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.569+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fa2a8061730 con 0x7fa28c0a2800 2024-03-08T23:05:06.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.604+0000 7fa29f400640 1 -- 172.21.15.46:0/652213242 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa2a8061730 con 0x7fa28c0a2800 2024-03-08T23:05:06.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 msgr2=0x7fa28c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 0x7fa28c0a4cc0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa2980099a0 tx=0x7fa298009500 comp rx=0 tx=0).stop 2024-03-08T23:05:06.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 msgr2=0x7fa2a81a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa2a0005ab0 tx=0x7fa2a002c6c0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 shutdown_connections 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2800 0x7fa28c0a4cc0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8077890 0x7fa2a81a5170 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8076ef0 0x7fa2a819e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.607+0000 7fa2ad400640 1 --2- 172.21.15.46:0/652213242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8075cb0 0x7fa2a819dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.608+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 >> 172.21.15.46:0/652213242 conn(0x7fa2a80feaa0 msgr2=0x7fa2a81005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.608+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 shutdown_connections 2024-03-08T23:05:06.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.608+0000 7fa2ad400640 1 -- 172.21.15.46:0/652213242 wait complete. 2024-03-08T23:05:06.631 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-08T23:05:06.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:06.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-08T23:05:06.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-08T23:05:06.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:active 2024-03-08T23:05:06.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-08T23:05:06.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:06 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 -- 172.21.15.46:0/1621175198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0101c50 msgr2=0x7faee0104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 --2- 172.21.15.46:0/1621175198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0101c50 0x7faee0104040 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7faed4009860 tx=0x7faed402bb70 comp rx=0 tx=0).stop 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 -- 172.21.15.46:0/1621175198 shutdown_connections 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 --2- 172.21.15.46:0/1621175198 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0106eb0 0x7faee0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 --2- 172.21.15.46:0/1621175198 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0104580 0x7faee0106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.850+0000 7faee6c00640 1 --2- 172.21.15.46:0/1621175198 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0101c50 0x7faee0104040 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.851+0000 7faee6c00640 1 -- 172.21.15.46:0/1621175198 >> 172.21.15.46:0/1621175198 conn(0x7faee00fb6a0 msgr2=0x7faee00fdac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.851+0000 7faee6c00640 1 -- 172.21.15.46:0/1621175198 shutdown_connections 2024-03-08T23:05:06.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.851+0000 7faee6c00640 1 -- 172.21.15.46:0/1621175198 wait complete. 2024-03-08T23:05:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.852+0000 7faee6c00640 1 Processor -- start 2024-03-08T23:05:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.852+0000 7faee6c00640 1 -- start start 2024-03-08T23:05:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.852+0000 7faee6c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0104580 0x7faee0195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0106eb0 0x7faee019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee010ac90 con 0x7faee0101c50 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee010ab10 con 0x7faee0104580 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee010ae10 con 0x7faee0106eb0 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60820/0 (socket says 172.21.15.46:60820) 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 -- 172.21.15.46:0/88820310 learned_addr learned my addr 172.21.15.46:0/88820310 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee4e00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0104580 0x7faee0195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee6200640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0106eb0 0x7faee019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 -- 172.21.15.46:0/88820310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0106eb0 msgr2=0x7faee019c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0106eb0 0x7faee019c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 -- 172.21.15.46:0/88820310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0104580 msgr2=0x7faee0195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0104580 0x7faee0195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:06.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.853+0000 7faee5800640 1 -- 172.21.15.46:0/88820310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faee019cee0 con 0x7faee0101c50 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faee5800640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faed40349e0 tx=0x7faed4032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4041030 con 0x7faee0101c50 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee019d170 con 0x7faee0101c50 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faed402c510 con 0x7faee0101c50 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4039040 con 0x7faee0101c50 2024-03-08T23:05:06.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.854+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee019d660 con 0x7faee0101c50 2024-03-08T23:05:06.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.856+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee00fd190 con 0x7faee0101c50 2024-03-08T23:05:06.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.857+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7faed4044070 con 0x7faee0101c50 2024-03-08T23:05:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.860+0000 7faede000640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 0x7faec80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.860+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7faed40f0290 con 0x7faee0101c50 2024-03-08T23:05:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.860+0000 7faee4e00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 0x7faec80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.861+0000 7faee4e00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 0x7faec80a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faee01967d0 tx=0x7faed0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:06.864+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7faed40bd0f0 con 0x7faee0101c50 2024-03-08T23:05:06.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:06.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-08T23:05:06.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-08T23:05:06.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:active 2024-03-08T23:05:06.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-08T23:05:06.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:06 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1482412731' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] up:active 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 9 up:standby 2024-03-08T23:05:06.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:05:06.977 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:05:06.579+0000 7ff011e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:05:06.977 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:05:06.579+0000 7ff011e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:05:06.977 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:05:06.579+0000 7ff011e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:05:06.977 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:05:06.579+0000 7ff011e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:05:06.977 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:05:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:05:06.579+0000 7ff011e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:05:07.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.101+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7faee0061730 con 0x7faec80a2800 2024-03-08T23:05:07.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.135+0000 7faede000640 1 -- 172.21.15.46:0/88820310 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7faee0061730 con 0x7faec80a2800 2024-03-08T23:05:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 msgr2=0x7faec80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 0x7faec80a4cc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7faee01967d0 tx=0x7faed0009260 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 msgr2=0x7faee01952c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faed40349e0 tx=0x7faed4032450 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 shutdown_connections 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7faec80a2800 0x7faec80a4cc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faee0106eb0 0x7faee019c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faee0104580 0x7faee0195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 --2- 172.21.15.46:0/88820310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faee0101c50 0x7faee01952c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 >> 172.21.15.46:0/88820310 conn(0x7faee00fb6a0 msgr2=0x7faee0102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 shutdown_connections 2024-03-08T23:05:07.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.138+0000 7faee6c00640 1 -- 172.21.15.46:0/88820310 wait complete. 2024-03-08T23:05:07.162 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-08T23:05:07.391 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:05:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: 2024-03-08T23:05:07.028+0000 7ff761276700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.395+0000 7f8f9b200640 1 -- 172.21.15.46:0/4050213230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 msgr2=0x7f8f9c0ffee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.395+0000 7f8f9b200640 1 --2- 172.21.15.46:0/4050213230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c0ffee0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009ca0 tx=0x7f8f9402c190 comp rx=0 tx=0).stop 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 -- 172.21.15.46:0/4050213230 shutdown_connections 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 --2- 172.21.15.46:0/4050213230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100dc0 0x7f8f9c101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 --2- 172.21.15.46:0/4050213230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100420 0x7f8f9c100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 --2- 172.21.15.46:0/4050213230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c0ffee0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 -- 172.21.15.46:0/4050213230 >> 172.21.15.46:0/4050213230 conn(0x7f8f9c0fb660 msgr2=0x7f8f9c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 -- 172.21.15.46:0/4050213230 shutdown_connections 2024-03-08T23:05:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.396+0000 7f8f9b200640 1 -- 172.21.15.46:0/4050213230 wait complete. 2024-03-08T23:05:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.397+0000 7f8f9b200640 1 Processor -- start 2024-03-08T23:05:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.397+0000 7f8f9b200640 1 -- start start 2024-03-08T23:05:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100420 0x7f8f9c199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100dc0 0x7f8f9c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c10f060 con 0x7f8f9c100420 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c10eee0 con 0x7f8f9c100dc0 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9c10f1e0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f99e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f99400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100420 0x7f8f9c199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f99e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33450/0 (socket says 172.21.15.46:33450) 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f99e00640 1 -- 172.21.15.46:0/1171392962 learned_addr learned my addr 172.21.15.46:0/1171392962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.398+0000 7f8f9a800640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100dc0 0x7f8f9c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 -- 172.21.15.46:0/1171392962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100dc0 msgr2=0x7f8f9c1a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100dc0 0x7f8f9c1a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 -- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100420 msgr2=0x7f8f9c199c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100420 0x7f8f9c199c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 -- 172.21.15.46:0/1171392962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9c1a1360 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f99e00640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f940326d0 tx=0x7f8f94035f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f94039040 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f94044070 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9c1a15f0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.399+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f940043a0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.400+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9c1a1ab0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.401+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9c10a810 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.403+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f94071b40 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.404+0000 7f8f92a00640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 0x7f8f800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.404+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8f940f12b0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.405+0000 7f8f99400640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 0x7f8f800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.405+0000 7f8f99400640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 0x7f8f800a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c19ac50 tx=0x7f8f84005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:07.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.407+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8f940be0b0 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.542+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8f940ba020 con 0x7f8f9c0ffac0 2024-03-08T23:05:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.656+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f9c0617c0 con 0x7f8f800a2800 2024-03-08T23:05:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.664+0000 7f8f92a00640 1 -- 172.21.15.46:0/1171392962 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f8f9c0617c0 con 0x7f8f800a2800 2024-03-08T23:05:07.668 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_0/6db3f9db-8e0b-42e9-b46c-2fb66f196760 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 msgr2=0x7f8f800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 0x7f8f800a4cc0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c19ac50 tx=0x7f8f84005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 msgr2=0x7f8f9c199740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8f940326d0 tx=0x7f8f94035f60 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 shutdown_connections 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8f800a2800 0x7f8f800a4cc0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8f9c100dc0 0x7f8f9c1a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8f9c100420 0x7f8f9c199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 --2- 172.21.15.46:0/1171392962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f9c0ffac0 0x7f8f9c199740 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 >> 172.21.15.46:0/1171392962 conn(0x7f8f9c0fb660 msgr2=0x7f8f9c0fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 shutdown_connections 2024-03-08T23:05:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.667+0000 7f8f9b200640 1 -- 172.21.15.46:0/1171392962 wait complete. 2024-03-08T23:05:07.691 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-08T23:05:07.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:07 smithi192 ceph-mon[36700]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:07.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:07 smithi062 ceph-mon[35761]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 -- 172.21.15.46:0/3026298338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318076ef0 msgr2=0x7f7318077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 --2- 172.21.15.46:0/3026298338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318076ef0 0x7f7318077350 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7304009900 tx=0x7f730402bc30 comp rx=0 tx=0).stop 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 -- 172.21.15.46:0/3026298338 shutdown_connections 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 --2- 172.21.15.46:0/3026298338 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318077890 0x7f7318078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 --2- 172.21.15.46:0/3026298338 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318076ef0 0x7f7318077350 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 --2- 172.21.15.46:0/3026298338 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318075cb0 0x7f73180760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 -- 172.21.15.46:0/3026298338 >> 172.21.15.46:0/3026298338 conn(0x7f73180feaa0 msgr2=0x7f7318100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 -- 172.21.15.46:0/3026298338 shutdown_connections 2024-03-08T23:05:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.920+0000 7f7320400640 1 -- 172.21.15.46:0/3026298338 wait complete. 2024-03-08T23:05:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.921+0000 7f7320400640 1 Processor -- start 2024-03-08T23:05:07.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.921+0000 7f7320400640 1 -- start start 2024-03-08T23:05:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 0x7f731819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318077890 0x7f73181a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7318113600 con 0x7f7318077890 2024-03-08T23:05:07.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7318113480 con 0x7f7318075cb0 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f7320400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7318113780 con 0x7f7318076ef0 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f731d000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f731d000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33472/0 (socket says 172.21.15.46:33472) 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f731d000640 1 -- 172.21.15.46:0/2644782718 learned_addr learned my addr 172.21.15.46:0/2644782718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f731da00640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 0x7f731819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.922+0000 7f731e400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318077890 0x7f73181a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 -- 172.21.15.46:0/2644782718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 msgr2=0x7f731819dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 0x7f731819dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 -- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318077890 msgr2=0x7f73181a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318077890 0x7f73181a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 -- 172.21.15.46:0/2644782718 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73181a5810 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731da00640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 0x7f731819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.923+0000 7f731d000640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7304000c00 tx=0x7f730402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.924+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304043020 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.924+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73181a5aa0 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.924+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73040054e0 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.924+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304003c40 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.924+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73181a6040 con 0x7f7318076ef0 2024-03-08T23:05:07.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.925+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731810fea0 con 0x7f7318076ef0 2024-03-08T23:05:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.928+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7304046020 con 0x7f7318076ef0 2024-03-08T23:05:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.929+0000 7f7316000640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 0x7f73000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.930+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f73040f12c0 con 0x7f7318076ef0 2024-03-08T23:05:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.930+0000 7f731da00640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 0x7f73000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.930+0000 7f731da00640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 0x7f73000a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f730c009fc0 tx=0x7f730c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:07.932+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f730404d020 con 0x7f7318076ef0 2024-03-08T23:05:07.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:07 smithi046 ceph-mon[31859]: from='client.14712 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.166+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f73180617c0 con 0x7f73000a2890 2024-03-08T23:05:08.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.199+0000 7f7316000640 1 -- 172.21.15.46:0/2644782718 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f73180617c0 con 0x7f73000a2890 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.202+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 msgr2=0x7f73000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.202+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 0x7f73000a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f730c009fc0 tx=0x7f730c005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.202+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 msgr2=0x7f731819e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.202+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7304000c00 tx=0x7f730402c650 comp rx=0 tx=0).stop 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 shutdown_connections 2024-03-08T23:05:08.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f73000a2890 0x7f73000a4d50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7318077890 0x7f73181a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7318076ef0 0x7f731819e130 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 --2- 172.21.15.46:0/2644782718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7318075cb0 0x7f731819dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 >> 172.21.15.46:0/2644782718 conn(0x7f73180feaa0 msgr2=0x7f73181005a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 shutdown_connections 2024-03-08T23:05:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.203+0000 7f7320400640 1 -- 172.21.15.46:0/2644782718 wait complete. 2024-03-08T23:05:08.227 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 -- 172.21.15.46:0/4271648367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba180752d0 msgr2=0x7fba18073720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 --2- 172.21.15.46:0/4271648367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba180752d0 0x7fba18073720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fba08009be0 tx=0x7fba0802bf80 comp rx=0 tx=0).stop 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 -- 172.21.15.46:0/4271648367 shutdown_connections 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 --2- 172.21.15.46:0/4271648367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18109060 0x7fba1810b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 --2- 172.21.15.46:0/4271648367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba18073cf0 0x7fba18074150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 --2- 172.21.15.46:0/4271648367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba180752d0 0x7fba18073720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 -- 172.21.15.46:0/4271648367 >> 172.21.15.46:0/4271648367 conn(0x7fba180fc760 msgr2=0x7fba180feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 -- 172.21.15.46:0/4271648367 shutdown_connections 2024-03-08T23:05:08.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.456+0000 7fba1d000640 1 -- 172.21.15.46:0/4271648367 wait complete. 2024-03-08T23:05:08.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.457+0000 7fba1d000640 1 Processor -- start 2024-03-08T23:05:08.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 -- start start 2024-03-08T23:05:08.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180752d0 0x7fba18199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba18109060 0x7fba181a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1810f0b0 con 0x7fba180752d0 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1810ef30 con 0x7fba18073cf0 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba1d000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1810f230 con 0x7fba18109060 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba17400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba17400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:54962/0 (socket says 172.21.15.46:54962) 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba17400640 1 -- 172.21.15.46:0/2876047546 learned_addr learned my addr 172.21.15.46:0/2876047546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.458+0000 7fba16a00640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180752d0 0x7fba18199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17e00640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba18109060 0x7fba181a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 -- 172.21.15.46:0/2876047546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba18109060 msgr2=0x7fba181a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba18109060 0x7fba181a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 -- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180752d0 msgr2=0x7fba18199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180752d0 0x7fba18199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 -- 172.21.15.46:0/2876047546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba181a12e0 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.459+0000 7fba17400640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fba0802c590 tx=0x7fba08004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.460+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08043020 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.460+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba08004670 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.460+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba181a1570 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.460+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0803e850 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.460+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba181a1b10 con 0x7fba18073cf0 2024-03-08T23:05:08.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.461+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba18107e20 con 0x7fba18073cf0 2024-03-08T23:05:08.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.463+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fba08046020 con 0x7fba18073cf0 2024-03-08T23:05:08.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.464+0000 7fba0fe00640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 0x7fb9fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.464+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fba080f1ba0 con 0x7fba18073cf0 2024-03-08T23:05:08.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.465+0000 7fba16a00640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 0x7fb9fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.465+0000 7fba16a00640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 0x7fb9fc0a4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba1819abd0 tx=0x7fba04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:08.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.467+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fba080f1e20 con 0x7fba18073cf0 2024-03-08T23:05:08.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:08 smithi192 ceph-mon[36700]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:08 smithi192 ceph-mon[36700]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:08.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:08 smithi192 ceph-mon[36700]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-08T23:05:08.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:08 smithi192 ceph-mon[36700]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.703+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fba180617c0 con 0x7fb9fc0a2910 2024-03-08T23:05:08.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.713+0000 7fba0fe00640 1 -- 172.21.15.46:0/2876047546 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fba180617c0 con 0x7fb9fc0a2910 2024-03-08T23:05:08.717 INFO:teuthology.orchestra.run.smithi046.stdout:/volumes/qa/sv_1/fd24f677-ddfc-4a4a-a669-a06e16ce0e48 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 msgr2=0x7fb9fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 0x7fb9fc0a4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fba1819abd0 tx=0x7fba04009260 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 msgr2=0x7fba181996c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fba0802c590 tx=0x7fba08004270 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 shutdown_connections 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb9fc0a2910 0x7fb9fc0a4dd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fba18109060 0x7fba181a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fba180752d0 0x7fba18199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 --2- 172.21.15.46:0/2876047546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fba18073cf0 0x7fba181996c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.716+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 >> 172.21.15.46:0/2876047546 conn(0x7fba180fc760 msgr2=0x7fba18104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:08.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.717+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 shutdown_connections 2024-03-08T23:05:08.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.717+0000 7fba1d000640 1 -- 172.21.15.46:0/2876047546 wait complete. 2024-03-08T23:05:08.740 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:05:08.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:08 smithi046 ceph-mon[31859]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.818 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:08 smithi046 ceph-mon[31859]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:08.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:08 smithi046 ceph-mon[31859]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-08T23:05:08.819 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:08 smithi046 ceph-mon[31859]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:08 smithi062 ceph-mon[35761]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:08 smithi062 ceph-mon[35761]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 2 op/s 2024-03-08T23:05:08.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:08 smithi062 ceph-mon[35761]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-08T23:05:08.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:08 smithi062 ceph-mon[35761]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 -- 172.21.15.46:0/2302833096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 msgr2=0x7fcea40760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 --2- 172.21.15.46:0/2302833096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea40760d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fce94009900 tx=0x7fce9402bc50 comp rx=0 tx=0).stop 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 -- 172.21.15.46:0/2302833096 shutdown_connections 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 --2- 172.21.15.46:0/2302833096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea4078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 --2- 172.21.15.46:0/2302833096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 0x7fcea4077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 --2- 172.21.15.46:0/2302833096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea40760d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 -- 172.21.15.46:0/2302833096 >> 172.21.15.46:0/2302833096 conn(0x7fcea40fe8d0 msgr2=0x7fcea4100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 -- 172.21.15.46:0/2302833096 shutdown_connections 2024-03-08T23:05:08.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.955+0000 7fcea3e00640 1 -- 172.21.15.46:0/2302833096 wait complete. 2024-03-08T23:05:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.956+0000 7fcea3e00640 1 Processor -- start 2024-03-08T23:05:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.956+0000 7fcea3e00640 1 -- start start 2024-03-08T23:05:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.956+0000 7fcea3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.956+0000 7fcea3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 0x7fcea419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea41a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea41133d0 con 0x7fcea4075cb0 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4113250 con 0x7fcea4077890 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4113550 con 0x7fcea4076ef0 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60882/0 (socket says 172.21.15.46:60882) 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 -- 172.21.15.46:0/328151273 learned_addr learned my addr 172.21.15.46:0/328151273 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3400640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea41a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2000640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 0x7fcea419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 -- 172.21.15.46:0/328151273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 msgr2=0x7fcea419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 0x7fcea419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 -- 172.21.15.46:0/328151273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 msgr2=0x7fcea41a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea41a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea2a00640 1 -- 172.21.15.46:0/328151273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea41a5700 con 0x7fcea4075cb0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.957+0000 7fcea3400640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea41a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fcea2a00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce94009230 tx=0x7fce9402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94042020 con 0x7fcea4075cb0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce94004450 con 0x7fcea4075cb0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea41a5990 con 0x7fcea4075cb0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94003db0 con 0x7fcea4075cb0 2024-03-08T23:05:08.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.958+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea41a5ee0 con 0x7fcea4075cb0 2024-03-08T23:05:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.959+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4068450 con 0x7fcea4075cb0 2024-03-08T23:05:08.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.961+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fce94045020 con 0x7fcea4075cb0 2024-03-08T23:05:08.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.962+0000 7fce9b400640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 0x7fce880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.963+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fce940f0d50 con 0x7fcea4075cb0 2024-03-08T23:05:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.963+0000 7fcea2000640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 0x7fce880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:08.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.963+0000 7fcea2000640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 0x7fce880a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcea419eff0 tx=0x7fce90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:08.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:08.969+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fce9404c020 con 0x7fcea4075cb0 2024-03-08T23:05:09.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.213+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcea402ce70 con 0x7fcea4075cb0 2024-03-08T23:05:09.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.215+0000 7fce9b400640 1 -- 172.21.15.46:0/328151273 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fce940bdb00 con 0x7fcea4075cb0 2024-03-08T23:05:09.218 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:09.218 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:05:09.219 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:05:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.217+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 msgr2=0x7fce880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.217+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 0x7fce880a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcea419eff0 tx=0x7fce90005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.217+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 msgr2=0x7fcea419dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.217+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fce94009230 tx=0x7fce9402c670 comp rx=0 tx=0).stop 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 shutdown_connections 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fce880a2890 0x7fce880a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcea4077890 0x7fcea41a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcea4076ef0 0x7fcea419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 --2- 172.21.15.46:0/328151273 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcea4075cb0 0x7fcea419dae0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 >> 172.21.15.46:0/328151273 conn(0x7fcea40fe8d0 msgr2=0x7fcea4100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 shutdown_connections 2024-03-08T23:05:09.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.218+0000 7fcea3e00640 1 -- 172.21.15.46:0/328151273 wait complete. 2024-03-08T23:05:09.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 -- 172.21.15.46:0/3385982790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e0108b20 msgr2=0x7f52e0108fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 --2- 172.21.15.46:0/3385982790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e0108b20 0x7f52e0108fa0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009900 tx=0x7f52c802bc30 comp rx=0 tx=0).stop 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 -- 172.21.15.46:0/3385982790 shutdown_connections 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 --2- 172.21.15.46:0/3385982790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e01094e0 0x7f52e010fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 --2- 172.21.15.46:0/3385982790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e0108b20 0x7f52e0108fa0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 --2- 172.21.15.46:0/3385982790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0107930 0x7f52e0107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 -- 172.21.15.46:0/3385982790 >> 172.21.15.46:0/3385982790 conn(0x7f52e0076900 msgr2=0x7f52e0078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 -- 172.21.15.46:0/3385982790 shutdown_connections 2024-03-08T23:05:09.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.468+0000 7f52e5c00640 1 -- 172.21.15.46:0/3385982790 wait complete. 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 Processor -- start 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 -- start start 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 0x7f52e019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e01094e0 0x7f52e01a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e0113600 con 0x7f52e01094e0 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e0113480 con 0x7f52e0108b20 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52e5c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e0113780 con 0x7f52e0107930 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.470+0000 7f52dea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33526/0 (socket says 172.21.15.46:33526) 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 -- 172.21.15.46:0/1397384149 learned_addr learned my addr 172.21.15.46:0/1397384149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52df400640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e01094e0 0x7f52e01a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52d7e00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 0x7f52e019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 -- 172.21.15.46:0/1397384149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 msgr2=0x7f52e019e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 0x7f52e019e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 -- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e01094e0 msgr2=0x7f52e01a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e01094e0 0x7f52e01a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 -- 172.21.15.46:0/1397384149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e01a5810 con 0x7f52e0107930 2024-03-08T23:05:09.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52d7e00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 0x7f52e019e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52dea00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f52d000e9c0 tx=0x7f52d000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.471+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d000cde0 con 0x7f52e0107930 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.472+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52d0010910 con 0x7f52e0107930 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.472+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e01a5b00 con 0x7f52e0107930 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.472+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d0018a00 con 0x7f52e0107930 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.472+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e01a60a0 con 0x7f52e0107930 2024-03-08T23:05:09.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.473+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e010ebf0 con 0x7f52e0107930 2024-03-08T23:05:09.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.475+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f52d001f020 con 0x7f52e0107930 2024-03-08T23:05:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.476+0000 7f52d7400640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 0x7f52c40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.477+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f52d00cf160 con 0x7f52e0107930 2024-03-08T23:05:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.477+0000 7f52d7e00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 0x7f52c40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.478+0000 7f52d7e00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 0x7f52c40a4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f52c80096c0 tx=0x7f52c802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:09.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.480+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f52d001e2c0 con 0x7f52e0107930 2024-03-08T23:05:09.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.685+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52e002d0d0 con 0x7f52e0107930 2024-03-08T23:05:09.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.687+0000 7f52d7400640 1 -- 172.21.15.46:0/1397384149 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f52d009be00 con 0x7f52e0107930 2024-03-08T23:05:09.691 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:09.691 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.689+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 msgr2=0x7f52c40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.689+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 0x7f52c40a4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f52c80096c0 tx=0x7f52c802df90 comp rx=0 tx=0).stop 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.689+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 msgr2=0x7f52e019dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.689+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f52d000e9c0 tx=0x7f52d000ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 shutdown_connections 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f52c40a2910 0x7f52c40a4dd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f52e01094e0 0x7f52e01a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f52e0108b20 0x7f52e019e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 --2- 172.21.15.46:0/1397384149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f52e0107930 0x7f52e019dbf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 >> 172.21.15.46:0/1397384149 conn(0x7f52e0076900 msgr2=0x7f52e010bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 shutdown_connections 2024-03-08T23:05:09.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.690+0000 7f52e5c00640 1 -- 172.21.15.46:0/1397384149 wait complete. 2024-03-08T23:05:09.712 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-08T23:05:09.712 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T23:05:09.799 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:09 smithi046 ceph-mon[31859]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.800 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:09 smithi046 ceph-mon[31859]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.800 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:09 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/328151273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:05:09.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:09 smithi192 ceph-mon[36700]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:09 smithi192 ceph-mon[36700]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:09 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/328151273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:05:09.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:09 smithi062 ceph-mon[35761]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:09 smithi062 ceph-mon[35761]: from='client.24848 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:05:09.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:09 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/328151273' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 -- 172.21.15.46:0/3011740264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 msgr2=0x7f320410b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 --2- 172.21.15.46:0/3011740264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f320410b790 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f31fc009ca0 tx=0x7f31fc02c100 comp rx=0 tx=0).stop 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 -- 172.21.15.46:0/3011740264 shutdown_connections 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 --2- 172.21.15.46:0/3011740264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f320410b790 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 --2- 172.21.15.46:0/3011740264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204104540 0x7f32041049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 --2- 172.21.15.46:0/3011740264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204103350 0x7f3204103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.939+0000 7f3208e00640 1 -- 172.21.15.46:0/3011740264 >> 172.21.15.46:0/3011740264 conn(0x7f32040fe8c0 msgr2=0x7f3204100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:09.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.940+0000 7f3208e00640 1 -- 172.21.15.46:0/3011740264 shutdown_connections 2024-03-08T23:05:09.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.940+0000 7f3208e00640 1 -- 172.21.15.46:0/3011740264 wait complete. 2024-03-08T23:05:09.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 Processor -- start 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 -- start start 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204104540 0x7f3204199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f32041a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410efe0 con 0x7f3204104f00 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410ee60 con 0x7f3204104540 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3208e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f320410f160 con 0x7f3204103350 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3203400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3203e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f32041a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3203400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33542/0 (socket says 172.21.15.46:33542) 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f3203400640 1 -- 172.21.15.46:0/458261423 learned_addr learned my addr 172.21.15.46:0/458261423 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.941+0000 7f31faa00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204104540 0x7f3204199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 -- 172.21.15.46:0/458261423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204104540 msgr2=0x7f3204199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204104540 0x7f3204199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 -- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 msgr2=0x7f32041a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f32041a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 -- 172.21.15.46:0/458261423 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32041a1200 con 0x7f3204103350 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f32041a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3203400640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e9c0 tx=0x7f31f400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f400cd80 con 0x7f3204103350 2024-03-08T23:05:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31f40035f0 con 0x7f3204103350 2024-03-08T23:05:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32041a14f0 con 0x7f3204103350 2024-03-08T23:05:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.942+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4010690 con 0x7f3204103350 2024-03-08T23:05:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.943+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32041a1a40 con 0x7f3204103350 2024-03-08T23:05:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.944+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f320410a670 con 0x7f3204103350 2024-03-08T23:05:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.946+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f31f40037a0 con 0x7f3204103350 2024-03-08T23:05:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.948+0000 7f3200c00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 0x7f31e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.948+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f31f40cf240 con 0x7f3204103350 2024-03-08T23:05:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.948+0000 7f31faa00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 0x7f31e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.949+0000 7f31faa00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 0x7f31e80a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f320419aaf0 tx=0x7f31ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:09.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:09.951+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f31f409bee0 con 0x7f3204103350 2024-03-08T23:05:10.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.153+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3204061730 con 0x7f31e80a2890 2024-03-08T23:05:10.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.190+0000 7f3200c00640 1 -- 172.21.15.46:0/458261423 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340624 (secure 0 0 0) 0x7f3204061730 con 0x7f31e80a2890 2024-03-08T23:05:10.197 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:10.201 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.198+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 msgr2=0x7f31e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.198+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 0x7f31e80a4d50 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f320419aaf0 tx=0x7f31ec005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.198+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 msgr2=0x7f32041995e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.198+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f31f400e9c0 tx=0x7f31f400ee80 comp rx=0 tx=0).stop 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 shutdown_connections 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f31e80a2890 0x7f31e80a4d50 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3204104f00 0x7f32041a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3204104540 0x7f3204199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 --2- 172.21.15.46:0/458261423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3204103350 0x7f32041995e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 >> 172.21.15.46:0/458261423 conn(0x7f32040fe8c0 msgr2=0x7f3204107840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 shutdown_connections 2024-03-08T23:05:10.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.199+0000 7f3208e00640 1 -- 172.21.15.46:0/458261423 wait complete. 2024-03-08T23:05:10.228 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-08T23:05:09.185243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597124,"num_objects":55,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":152,"num_read_kb":132,"num_write":302,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":242,"ondisk_log_size":242,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360788,"kb_used_data":11636,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581036,"statfs":{"total":1151940427776,"available":1151570980864,"internally_reserved":0,"allocated":11915264,"data_stored":7542445,"data_compressed":284217,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3897,"num_objects":31,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":75,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005860"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135551+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:04:45.135551+0000","last_peered":"2024-03-08T23:04:45.135551+0000","last_clean":"2024-03-08T23:04:45.135551+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:04:45.135551+0000","last_undegraded":"2024-03-08T23:04:45.135551+0000","last_fullsized":"2024-03-08T23:04:45.135551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514549+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:04:45.514549+0000","last_peered":"2024-03-08T23:04:45.514549+0000","last_clean":"2024-03-08T23:04:45.514549+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:04:45.514549+0000","last_undegraded":"2024-03-08T23:04:45.514549+0000","last_fullsized":"2024-03-08T23:04:45.514549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514232+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:04:45.514232+0000","last_peered":"2024-03-08T23:04:45.514232+0000","last_clean":"2024-03-08T23:04:45.514232+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:04:45.514232+0000","last_undegraded":"2024-03-08T23:04:45.514232+0000","last_fullsized":"2024-03-08T23:04:45.514232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512894+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:04:45.512894+0000","last_peered":"2024-03-08T23:04:45.512894+0000","last_clean":"2024-03-08T23:04:45.512894+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:04:45.512894+0000","last_undegraded":"2024-03-08T23:04:45.512894+0000","last_fullsized":"2024-03-08T23:04:45.512894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621289+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:04:45.621289+0000","last_peered":"2024-03-08T23:04:45.621289+0000","last_clean":"2024-03-08T23:04:45.621289+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:04:45.621289+0000","last_undegraded":"2024-03-08T23:04:45.621289+0000","last_fullsized":"2024-03-08T23:04:45.621289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514000+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:04:45.514000+0000","last_peered":"2024-03-08T23:04:45.514000+0000","last_clean":"2024-03-08T23:04:45.514000+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:04:45.514000+0000","last_undegraded":"2024-03-08T23:04:45.514000+0000","last_fullsized":"2024-03-08T23:04:45.514000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514554+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:04:45.514554+0000","last_peered":"2024-03-08T23:04:45.514554+0000","last_clean":"2024-03-08T23:04:45.514554+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:04:45.514554+0000","last_undegraded":"2024-03-08T23:04:45.514554+0000","last_fullsized":"2024-03-08T23:04:45.514554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414748+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:04:44.414748+0000","last_peered":"2024-03-08T23:04:44.414748+0000","last_clean":"2024-03-08T23:04:44.414748+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:04:44.414748+0000","last_undegraded":"2024-03-08T23:04:44.414748+0000","last_fullsized":"2024-03-08T23:04:44.414748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513800+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:04:45.513800+0000","last_peered":"2024-03-08T23:04:45.513800+0000","last_clean":"2024-03-08T23:04:45.513800+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:04:45.513800+0000","last_undegraded":"2024-03-08T23:04:45.513800+0000","last_fullsized":"2024-03-08T23:04:45.513800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514546+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:04:45.514546+0000","last_peered":"2024-03-08T23:04:45.514546+0000","last_clean":"2024-03-08T23:04:45.514546+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:04:45.514546+0000","last_undegraded":"2024-03-08T23:04:45.514546+0000","last_fullsized":"2024-03-08T23:04:45.514546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514580+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:04:45.514580+0000","last_peered":"2024-03-08T23:04:45.514580+0000","last_clean":"2024-03-08T23:04:45.514580+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:04:45.514580+0000","last_undegraded":"2024-03-08T23:04:45.514580+0000","last_fullsized":"2024-03-08T23:04:45.514580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512821+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:04:45.512821+0000","last_peered":"2024-03-08T23:04:45.512821+0000","last_clean":"2024-03-08T23:04:45.512821+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:04:45.512821+0000","last_undegraded":"2024-03-08T23:04:45.512821+0000","last_fullsized":"2024-03-08T23:04:45.512821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512787+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:04:45.512787+0000","last_peered":"2024-03-08T23:04:45.512787+0000","last_clean":"2024-03-08T23:04:45.512787+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:04:45.512787+0000","last_undegraded":"2024-03-08T23:04:45.512787+0000","last_fullsized":"2024-03-08T23:04:45.512787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514826+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:04:45.514826+0000","last_peered":"2024-03-08T23:04:45.514826+0000","last_clean":"2024-03-08T23:04:45.514826+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:04:45.514826+0000","last_undegraded":"2024-03-08T23:04:45.514826+0000","last_fullsized":"2024-03-08T23:04:45.514826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513821+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:04:45.513821+0000","last_peered":"2024-03-08T23:04:45.513821+0000","last_clean":"2024-03-08T23:04:45.513821+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:04:45.513821+0000","last_undegraded":"2024-03-08T23:04:45.513821+0000","last_fullsized":"2024-03-08T23:04:45.513821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512929+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:04:45.512929+0000","last_peered":"2024-03-08T23:04:45.512929+0000","last_clean":"2024-03-08T23:04:45.512929+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:04:45.512929+0000","last_undegraded":"2024-03-08T23:04:45.512929+0000","last_fullsized":"2024-03-08T23:04:45.512929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135376+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:04:45.135376+0000","last_peered":"2024-03-08T23:04:45.135376+0000","last_clean":"2024-03-08T23:04:45.135376+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:04:45.135376+0000","last_undegraded":"2024-03-08T23:04:45.135376+0000","last_fullsized":"2024-03-08T23:04:45.135376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415777+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:04:44.415777+0000","last_peered":"2024-03-08T23:04:44.415777+0000","last_clean":"2024-03-08T23:04:44.415777+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:04:44.415777+0000","last_undegraded":"2024-03-08T23:04:44.415777+0000","last_fullsized":"2024-03-08T23:04:44.415777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412124+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:04:44.412124+0000","last_peered":"2024-03-08T23:04:44.412124+0000","last_clean":"2024-03-08T23:04:44.412124+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:04:44.412124+0000","last_undegraded":"2024-03-08T23:04:44.412124+0000","last_fullsized":"2024-03-08T23:04:44.412124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411532+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:04:44.411532+0000","last_peered":"2024-03-08T23:04:44.411532+0000","last_clean":"2024-03-08T23:04:44.411532+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:04:44.411532+0000","last_undegraded":"2024-03-08T23:04:44.411532+0000","last_fullsized":"2024-03-08T23:04:44.411532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.087917+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:05:05.087917+0000","last_peered":"2024-03-08T23:05:05.087917+0000","last_clean":"2024-03-08T23:05:05.087917+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:05:05.087917+0000","last_undegraded":"2024-03-08T23:05:05.087917+0000","last_fullsized":"2024-03-08T23:05:05.087917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419183+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:04:44.419183+0000","last_peered":"2024-03-08T23:04:44.419183+0000","last_clean":"2024-03-08T23:04:44.419183+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:04:44.419183+0000","last_undegraded":"2024-03-08T23:04:44.419183+0000","last_fullsized":"2024-03-08T23:04:44.419183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512497+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:04:45.512497+0000","last_peered":"2024-03-08T23:04:45.512497+0000","last_clean":"2024-03-08T23:04:45.512497+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:04:45.512497+0000","last_undegraded":"2024-03-08T23:04:45.512497+0000","last_fullsized":"2024-03-08T23:04:45.512497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419465+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:04:44.419465+0000","last_peered":"2024-03-08T23:04:44.419465+0000","last_clean":"2024-03-08T23:04:44.419465+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:04:44.419465+0000","last_undegraded":"2024-03-08T23:04:44.419465+0000","last_fullsized":"2024-03-08T23:04:44.419465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514871+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:04:45.514871+0000","last_peered":"2024-03-08T23:04:45.514871+0000","last_clean":"2024-03-08T23:04:45.514871+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:04:45.514871+0000","last_undegraded":"2024-03-08T23:04:45.514871+0000","last_fullsized":"2024-03-08T23:04:45.514871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513117+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:04:45.513117+0000","last_peered":"2024-03-08T23:04:45.513117+0000","last_clean":"2024-03-08T23:04:45.513117+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:04:45.513117+0000","last_undegraded":"2024-03-08T23:04:45.513117+0000","last_fullsized":"2024-03-08T23:04:45.513117+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.079260+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:05:05.079260+0000","last_peered":"2024-03-08T23:05:05.079260+0000","last_clean":"2024-03-08T23:05:05.079260+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:05:05.079260+0000","last_undegraded":"2024-03-08T23:05:05.079260+0000","last_fullsized":"2024-03-08T23:05:05.079260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513120+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:04:45.513120+0000","last_peered":"2024-03-08T23:04:45.513120+0000","last_clean":"2024-03-08T23:04:45.513120+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:04:45.513120+0000","last_undegraded":"2024-03-08T23:04:45.513120+0000","last_fullsized":"2024-03-08T23:04:45.513120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514691+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:04:45.514691+0000","last_peered":"2024-03-08T23:04:45.514691+0000","last_clean":"2024-03-08T23:04:45.514691+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:04:45.514691+0000","last_undegraded":"2024-03-08T23:04:45.514691+0000","last_fullsized":"2024-03-08T23:04:45.514691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621628+0000","last_change":"2024-03-08T23:04:38.326247+0000","last_active":"2024-03-08T23:04:45.621628+0000","last_peered":"2024-03-08T23:04:45.621628+0000","last_clean":"2024-03-08T23:04:45.621628+0000","last_became_active":"2024-03-08T23:04:38.325843+0000","last_became_peered":"2024-03-08T23:04:38.325843+0000","last_unstale":"2024-03-08T23:04:45.621628+0000","last_undegraded":"2024-03-08T23:04:45.621628+0000","last_fullsized":"2024-03-08T23:04:45.621628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:38:07.047189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460244+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:04:44.460244+0000","last_peered":"2024-03-08T23:04:44.460244+0000","last_clean":"2024-03-08T23:04:44.460244+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:04:44.460244+0000","last_undegraded":"2024-03-08T23:04:44.460244+0000","last_fullsized":"2024-03-08T23:04:44.460244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412035+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:04:44.412035+0000","last_peered":"2024-03-08T23:04:44.412035+0000","last_clean":"2024-03-08T23:04:44.412035+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:04:44.412035+0000","last_undegraded":"2024-03-08T23:04:44.412035+0000","last_fullsized":"2024-03-08T23:04:44.412035+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:07.093822+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:05:07.093822+0000","last_peered":"2024-03-08T23:05:07.093822+0000","last_clean":"2024-03-08T23:05:07.093822+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:05:07.093822+0000","last_undegraded":"2024-03-08T23:05:07.093822+0000","last_fullsized":"2024-03-08T23:05:07.093822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621353+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:04:45.621353+0000","last_peered":"2024-03-08T23:04:45.621353+0000","last_clean":"2024-03-08T23:04:45.621353+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:04:45.621353+0000","last_undegraded":"2024-03-08T23:04:45.621353+0000","last_fullsized":"2024-03-08T23:04:45.621353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411790+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:04:44.411790+0000","last_peered":"2024-03-08T23:04:44.411790+0000","last_clean":"2024-03-08T23:04:44.411790+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:04:44.411790+0000","last_undegraded":"2024-03-08T23:04:44.411790+0000","last_fullsized":"2024-03-08T23:04:44.411790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513972+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:04:45.513972+0000","last_peered":"2024-03-08T23:04:45.513972+0000","last_clean":"2024-03-08T23:04:45.513972+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:04:45.513972+0000","last_undegraded":"2024-03-08T23:04:45.513972+0000","last_fullsized":"2024-03-08T23:04:45.513972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460110+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:04:44.460110+0000","last_peered":"2024-03-08T23:04:44.460110+0000","last_clean":"2024-03-08T23:04:44.460110+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:04:44.460110+0000","last_undegraded":"2024-03-08T23:04:44.460110+0000","last_fullsized":"2024-03-08T23:04:44.460110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135480+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:04:45.135480+0000","last_peered":"2024-03-08T23:04:45.135480+0000","last_clean":"2024-03-08T23:04:45.135480+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:04:45.135480+0000","last_undegraded":"2024-03-08T23:04:45.135480+0000","last_fullsized":"2024-03-08T23:04:45.135480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.088371+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:05:05.088371+0000","last_peered":"2024-03-08T23:05:05.088371+0000","last_clean":"2024-03-08T23:05:05.088371+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:05:05.088371+0000","last_undegraded":"2024-03-08T23:05:05.088371+0000","last_fullsized":"2024-03-08T23:05:05.088371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415138+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:04:44.415138+0000","last_peered":"2024-03-08T23:04:44.415138+0000","last_clean":"2024-03-08T23:04:44.415138+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:04:44.415138+0000","last_undegraded":"2024-03-08T23:04:44.415138+0000","last_fullsized":"2024-03-08T23:04:44.415138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514476+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:04:45.514476+0000","last_peered":"2024-03-08T23:04:45.514476+0000","last_clean":"2024-03-08T23:04:45.514476+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:04:45.514476+0000","last_undegraded":"2024-03-08T23:04:45.514476+0000","last_fullsized":"2024-03-08T23:04:45.514476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459799+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:04:44.459799+0000","last_peered":"2024-03-08T23:04:44.459799+0000","last_clean":"2024-03-08T23:04:44.459799+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:04:44.459799+0000","last_undegraded":"2024-03-08T23:04:44.459799+0000","last_fullsized":"2024-03-08T23:04:44.459799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414688+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:04:44.414688+0000","last_peered":"2024-03-08T23:04:44.414688+0000","last_clean":"2024-03-08T23:04:44.414688+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:04:44.414688+0000","last_undegraded":"2024-03-08T23:04:44.414688+0000","last_fullsized":"2024-03-08T23:04:44.414688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415862+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:04:44.415862+0000","last_peered":"2024-03-08T23:04:44.415862+0000","last_clean":"2024-03-08T23:04:44.415862+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:04:44.415862+0000","last_undegraded":"2024-03-08T23:04:44.415862+0000","last_fullsized":"2024-03-08T23:04:44.415862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074292+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:05:04.074292+0000","last_peered":"2024-03-08T23:05:04.074292+0000","last_clean":"2024-03-08T23:05:04.074292+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:05:04.074292+0000","last_undegraded":"2024-03-08T23:05:04.074292+0000","last_fullsized":"2024-03-08T23:05:04.074292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513926+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:04:45.513926+0000","last_peered":"2024-03-08T23:04:45.513926+0000","last_clean":"2024-03-08T23:04:45.513926+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:04:45.513926+0000","last_undegraded":"2024-03-08T23:04:45.513926+0000","last_fullsized":"2024-03-08T23:04:45.513926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414966+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:04:44.414966+0000","last_peered":"2024-03-08T23:04:44.414966+0000","last_clean":"2024-03-08T23:04:44.414966+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:04:44.414966+0000","last_undegraded":"2024-03-08T23:04:44.414966+0000","last_fullsized":"2024-03-08T23:04:44.414966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074386+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:05:04.074386+0000","last_peered":"2024-03-08T23:05:04.074386+0000","last_clean":"2024-03-08T23:05:04.074386+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:05:04.074386+0000","last_undegraded":"2024-03-08T23:05:04.074386+0000","last_fullsized":"2024-03-08T23:05:04.074386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514928+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:04:45.514928+0000","last_peered":"2024-03-08T23:04:45.514928+0000","last_clean":"2024-03-08T23:04:45.514928+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:04:45.514928+0000","last_undegraded":"2024-03-08T23:04:45.514928+0000","last_fullsized":"2024-03-08T23:04:45.514928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513486+0000","last_change":"2024-03-08T23:04:43.446232+0000","last_active":"2024-03-08T23:04:45.513486+0000","last_peered":"2024-03-08T23:04:45.513486+0000","last_clean":"2024-03-08T23:04:45.513486+0000","last_became_active":"2024-03-08T23:04:43.445805+0000","last_became_peered":"2024-03-08T23:04:43.445805+0000","last_unstale":"2024-03-08T23:04:45.513486+0000","last_undegraded":"2024-03-08T23:04:45.513486+0000","last_fullsized":"2024-03-08T23:04:45.513486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:24.425824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.076844+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:05:04.076844+0000","last_peered":"2024-03-08T23:05:04.076844+0000","last_clean":"2024-03-08T23:05:04.076844+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:05:04.076844+0000","last_undegraded":"2024-03-08T23:05:04.076844+0000","last_fullsized":"2024-03-08T23:05:04.076844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513325+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:04:45.513325+0000","last_peered":"2024-03-08T23:04:45.513325+0000","last_clean":"2024-03-08T23:04:45.513325+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:04:45.513325+0000","last_undegraded":"2024-03-08T23:04:45.513325+0000","last_fullsized":"2024-03-08T23:04:45.513325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.416310+0000","last_change":"2024-03-08T23:04:43.443827+0000","last_active":"2024-03-08T23:04:44.416310+0000","last_peered":"2024-03-08T23:04:44.416310+0000","last_clean":"2024-03-08T23:04:44.416310+0000","last_became_active":"2024-03-08T23:04:43.442112+0000","last_became_peered":"2024-03-08T23:04:43.442112+0000","last_unstale":"2024-03-08T23:04:44.416310+0000","last_undegraded":"2024-03-08T23:04:44.416310+0000","last_fullsized":"2024-03-08T23:04:44.416310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:21:31.929876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.073437+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:05:04.073437+0000","last_peered":"2024-03-08T23:05:04.073437+0000","last_clean":"2024-03-08T23:05:04.073437+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:05:04.073437+0000","last_undegraded":"2024-03-08T23:05:04.073437+0000","last_fullsized":"2024-03-08T23:05:04.073437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621549+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:04:45.621549+0000","last_peered":"2024-03-08T23:04:45.621549+0000","last_clean":"2024-03-08T23:04:45.621549+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:04:45.621549+0000","last_undegraded":"2024-03-08T23:04:45.621549+0000","last_fullsized":"2024-03-08T23:04:45.621549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460531+0000","last_change":"2024-03-08T23:04:43.439283+0000","last_active":"2024-03-08T23:04:44.460531+0000","last_peered":"2024-03-08T23:04:44.460531+0000","last_clean":"2024-03-08T23:04:44.460531+0000","last_became_active":"2024-03-08T23:04:43.438890+0000","last_became_peered":"2024-03-08T23:04:43.438890+0000","last_unstale":"2024-03-08T23:04:44.460531+0000","last_undegraded":"2024-03-08T23:04:44.460531+0000","last_fullsized":"2024-03-08T23:04:44.460531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:45:05.798040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.078406+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:05:04.078406+0000","last_peered":"2024-03-08T23:05:04.078406+0000","last_clean":"2024-03-08T23:05:04.078406+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:05:04.078406+0000","last_undegraded":"2024-03-08T23:05:04.078406+0000","last_fullsized":"2024-03-08T23:05:04.078406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512382+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:04:45.512382+0000","last_peered":"2024-03-08T23:04:45.512382+0000","last_clean":"2024-03-08T23:04:45.512382+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:04:45.512382+0000","last_undegraded":"2024-03-08T23:04:45.512382+0000","last_fullsized":"2024-03-08T23:04:45.512382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135298+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:04:45.135298+0000","last_peered":"2024-03-08T23:04:45.135298+0000","last_clean":"2024-03-08T23:04:45.135298+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:04:45.135298+0000","last_undegraded":"2024-03-08T23:04:45.135298+0000","last_fullsized":"2024-03-08T23:04:45.135298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.077219+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:05:04.077219+0000","last_peered":"2024-03-08T23:05:04.077219+0000","last_clean":"2024-03-08T23:05:04.077219+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:05:04.077219+0000","last_undegraded":"2024-03-08T23:05:04.077219+0000","last_fullsized":"2024-03-08T23:05:04.077219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514247+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:04:45.514247+0000","last_peered":"2024-03-08T23:04:45.514247+0000","last_clean":"2024-03-08T23:04:45.514247+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:04:45.514247+0000","last_undegraded":"2024-03-08T23:04:45.514247+0000","last_fullsized":"2024-03-08T23:04:45.514247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.417458+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:04:44.417458+0000","last_peered":"2024-03-08T23:04:44.417458+0000","last_clean":"2024-03-08T23:04:44.417458+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:04:44.417458+0000","last_undegraded":"2024-03-08T23:04:44.417458+0000","last_fullsized":"2024-03-08T23:04:44.417458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513022+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:04:45.513022+0000","last_peered":"2024-03-08T23:04:45.513022+0000","last_clean":"2024-03-08T23:04:45.513022+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:04:45.513022+0000","last_undegraded":"2024-03-08T23:04:45.513022+0000","last_fullsized":"2024-03-08T23:04:45.513022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513812+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:04:45.513812+0000","last_peered":"2024-03-08T23:04:45.513812+0000","last_clean":"2024-03-08T23:04:45.513812+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:04:45.513812+0000","last_undegraded":"2024-03-08T23:04:45.513812+0000","last_fullsized":"2024-03-08T23:04:45.513812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514146+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:04:45.514146+0000","last_peered":"2024-03-08T23:04:45.514146+0000","last_clean":"2024-03-08T23:04:45.514146+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:04:45.514146+0000","last_undegraded":"2024-03-08T23:04:45.514146+0000","last_fullsized":"2024-03-08T23:04:45.514146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.088392+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:05:05.088392+0000","last_peered":"2024-03-08T23:05:05.088392+0000","last_clean":"2024-03-08T23:05:05.088392+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:05:05.088392+0000","last_undegraded":"2024-03-08T23:05:05.088392+0000","last_fullsized":"2024-03-08T23:05:05.088392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459745+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:04:44.459745+0000","last_peered":"2024-03-08T23:04:44.459745+0000","last_clean":"2024-03-08T23:04:44.459745+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:04:44.459745+0000","last_undegraded":"2024-03-08T23:04:44.459745+0000","last_fullsized":"2024-03-08T23:04:44.459745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459802+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:04:44.459802+0000","last_peered":"2024-03-08T23:04:44.459802+0000","last_clean":"2024-03-08T23:04:44.459802+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:04:44.459802+0000","last_undegraded":"2024-03-08T23:04:44.459802+0000","last_fullsized":"2024-03-08T23:04:44.459802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:06.076740+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:05:06.076740+0000","last_peered":"2024-03-08T23:05:06.076740+0000","last_clean":"2024-03-08T23:05:06.076740+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:05:06.076740+0000","last_undegraded":"2024-03-08T23:05:06.076740+0000","last_fullsized":"2024-03-08T23:05:06.076740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135220+0000","last_change":"2024-03-08T23:04:39.075656+0000","last_active":"2024-03-08T23:04:45.135220+0000","last_peered":"2024-03-08T23:04:45.135220+0000","last_clean":"2024-03-08T23:04:45.135220+0000","last_became_active":"2024-03-08T23:04:39.075211+0000","last_became_peered":"2024-03-08T23:04:39.075211+0000","last_unstale":"2024-03-08T23:04:45.135220+0000","last_undegraded":"2024-03-08T23:04:45.135220+0000","last_fullsized":"2024-03-08T23:04:45.135220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:04:14.996902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621378+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:04:45.621378+0000","last_peered":"2024-03-08T23:04:45.621378+0000","last_clean":"2024-03-08T23:04:45.621378+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:04:45.621378+0000","last_undegraded":"2024-03-08T23:04:45.621378+0000","last_fullsized":"2024-03-08T23:04:45.621378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415211+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:04:44.415211+0000","last_peered":"2024-03-08T23:04:44.415211+0000","last_clean":"2024-03-08T23:04:44.415211+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:04:44.415211+0000","last_undegraded":"2024-03-08T23:04:44.415211+0000","last_fullsized":"2024-03-08T23:04:44.415211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514015+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:04:45.514015+0000","last_peered":"2024-03-08T23:04:45.514015+0000","last_clean":"2024-03-08T23:04:45.514015+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:04:45.514015+0000","last_undegraded":"2024-03-08T23:04:45.514015+0000","last_fullsized":"2024-03-08T23:04:45.514015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513630+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:04:45.513630+0000","last_peered":"2024-03-08T23:04:45.513630+0000","last_clean":"2024-03-08T23:04:45.513630+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:04:45.513630+0000","last_undegraded":"2024-03-08T23:04:45.513630+0000","last_fullsized":"2024-03-08T23:04:45.513630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514324+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:04:45.514324+0000","last_peered":"2024-03-08T23:04:45.514324+0000","last_clean":"2024-03-08T23:04:45.514324+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:04:45.514324+0000","last_undegraded":"2024-03-08T23:04:45.514324+0000","last_fullsized":"2024-03-08T23:04:45.514324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513165+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:04:45.513165+0000","last_peered":"2024-03-08T23:04:45.513165+0000","last_clean":"2024-03-08T23:04:45.513165+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:04:45.513165+0000","last_undegraded":"2024-03-08T23:04:45.513165+0000","last_fullsized":"2024-03-08T23:04:45.513165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513147+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:04:45.513147+0000","last_peered":"2024-03-08T23:04:45.513147+0000","last_clean":"2024-03-08T23:04:45.513147+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:04:45.513147+0000","last_undegraded":"2024-03-08T23:04:45.513147+0000","last_fullsized":"2024-03-08T23:04:45.513147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459935+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:04:44.459935+0000","last_peered":"2024-03-08T23:04:44.459935+0000","last_clean":"2024-03-08T23:04:44.459935+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:04:44.459935+0000","last_undegraded":"2024-03-08T23:04:44.459935+0000","last_fullsized":"2024-03-08T23:04:44.459935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:06.090444+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:05:06.090444+0000","last_peered":"2024-03-08T23:05:06.090444+0000","last_clean":"2024-03-08T23:05:06.090444+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:05:06.090444+0000","last_undegraded":"2024-03-08T23:05:06.090444+0000","last_fullsized":"2024-03-08T23:05:06.090444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513863+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:04:45.513863+0000","last_peered":"2024-03-08T23:04:45.513863+0000","last_clean":"2024-03-08T23:04:45.513863+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:04:45.513863+0000","last_undegraded":"2024-03-08T23:04:45.513863+0000","last_fullsized":"2024-03-08T23:04:45.513863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.417776+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:04:44.417776+0000","last_peered":"2024-03-08T23:04:44.417776+0000","last_clean":"2024-03-08T23:04:44.417776+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:04:44.417776+0000","last_undegraded":"2024-03-08T23:04:44.417776+0000","last_fullsized":"2024-03-08T23:04:44.417776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.075117+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:05:04.075117+0000","last_peered":"2024-03-08T23:05:04.075117+0000","last_clean":"2024-03-08T23:05:04.075117+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:05:04.075117+0000","last_undegraded":"2024-03-08T23:05:04.075117+0000","last_fullsized":"2024-03-08T23:05:04.075117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513560+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:04:45.513560+0000","last_peered":"2024-03-08T23:04:45.513560+0000","last_clean":"2024-03-08T23:04:45.513560+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:04:45.513560+0000","last_undegraded":"2024-03-08T23:04:45.513560+0000","last_fullsized":"2024-03-08T23:04:45.513560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460352+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:04:44.460352+0000","last_peered":"2024-03-08T23:04:44.460352+0000","last_clean":"2024-03-08T23:04:44.460352+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:04:44.460352+0000","last_undegraded":"2024-03-08T23:04:44.460352+0000","last_fullsized":"2024-03-08T23:04:44.460352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512135+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:04:45.512135+0000","last_peered":"2024-03-08T23:04:45.512135+0000","last_clean":"2024-03-08T23:04:45.512135+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:04:45.512135+0000","last_undegraded":"2024-03-08T23:04:45.512135+0000","last_fullsized":"2024-03-08T23:04:45.512135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.079554+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:05:05.079554+0000","last_peered":"2024-03-08T23:05:05.079554+0000","last_clean":"2024-03-08T23:05:05.079554+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:05:05.079554+0000","last_undegraded":"2024-03-08T23:05:05.079554+0000","last_fullsized":"2024-03-08T23:05:05.079554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135240+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:04:45.135240+0000","last_peered":"2024-03-08T23:04:45.135240+0000","last_clean":"2024-03-08T23:04:45.135240+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:04:45.135240+0000","last_undegraded":"2024-03-08T23:04:45.135240+0000","last_fullsized":"2024-03-08T23:04:45.135240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.075265+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:05:04.075265+0000","last_peered":"2024-03-08T23:05:04.075265+0000","last_clean":"2024-03-08T23:05:04.075265+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:05:04.075265+0000","last_undegraded":"2024-03-08T23:05:04.075265+0000","last_fullsized":"2024-03-08T23:05:04.075265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513717+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:04:45.513717+0000","last_peered":"2024-03-08T23:04:45.513717+0000","last_clean":"2024-03-08T23:04:45.513717+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:04:45.513717+0000","last_undegraded":"2024-03-08T23:04:45.513717+0000","last_fullsized":"2024-03-08T23:04:45.513717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415074+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:04:44.415074+0000","last_peered":"2024-03-08T23:04:44.415074+0000","last_clean":"2024-03-08T23:04:44.415074+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:04:44.415074+0000","last_undegraded":"2024-03-08T23:04:44.415074+0000","last_fullsized":"2024-03-08T23:04:44.415074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:07.365999+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:05:07.365999+0000","last_peered":"2024-03-08T23:05:07.365999+0000","last_clean":"2024-03-08T23:05:07.365999+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:05:07.365999+0000","last_undegraded":"2024-03-08T23:05:07.365999+0000","last_fullsized":"2024-03-08T23:05:07.365999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419475+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:04:44.419475+0000","last_peered":"2024-03-08T23:04:44.419475+0000","last_clean":"2024-03-08T23:04:44.419475+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:04:44.419475+0000","last_undegraded":"2024-03-08T23:04:44.419475+0000","last_fullsized":"2024-03-08T23:04:44.419475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411633+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:04:44.411633+0000","last_peered":"2024-03-08T23:04:44.411633+0000","last_clean":"2024-03-08T23:04:44.411633+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:04:44.411633+0000","last_undegraded":"2024-03-08T23:04:44.411633+0000","last_fullsized":"2024-03-08T23:04:44.411633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.072998+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:05:04.072998+0000","last_peered":"2024-03-08T23:05:04.072998+0000","last_clean":"2024-03-08T23:05:04.072998+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:05:04.072998+0000","last_undegraded":"2024-03-08T23:05:04.072998+0000","last_fullsized":"2024-03-08T23:05:04.072998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514650+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:04:45.514650+0000","last_peered":"2024-03-08T23:04:45.514650+0000","last_clean":"2024-03-08T23:04:45.514650+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:04:45.514650+0000","last_undegraded":"2024-03-08T23:04:45.514650+0000","last_fullsized":"2024-03-08T23:04:45.514650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512618+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:04:45.512618+0000","last_peered":"2024-03-08T23:04:45.512618+0000","last_clean":"2024-03-08T23:04:45.512618+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:04:45.512618+0000","last_undegraded":"2024-03-08T23:04:45.512618+0000","last_fullsized":"2024-03-08T23:04:45.512618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.078099+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:05:04.078099+0000","last_peered":"2024-03-08T23:05:04.078099+0000","last_clean":"2024-03-08T23:05:04.078099+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:05:04.078099+0000","last_undegraded":"2024-03-08T23:05:04.078099+0000","last_fullsized":"2024-03-08T23:05:04.078099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621187+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:04:45.621187+0000","last_peered":"2024-03-08T23:04:45.621187+0000","last_clean":"2024-03-08T23:04:45.621187+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:04:45.621187+0000","last_undegraded":"2024-03-08T23:04:45.621187+0000","last_fullsized":"2024-03-08T23:04:45.621187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.418567+0000","last_change":"2024-03-08T23:04:43.441758+0000","last_active":"2024-03-08T23:04:44.418567+0000","last_peered":"2024-03-08T23:04:44.418567+0000","last_clean":"2024-03-08T23:04:44.418567+0000","last_became_active":"2024-03-08T23:04:43.441297+0000","last_became_peered":"2024-03-08T23:04:43.441297+0000","last_unstale":"2024-03-08T23:04:44.418567+0000","last_undegraded":"2024-03-08T23:04:44.418567+0000","last_fullsized":"2024-03-08T23:04:44.418567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:43:56.396622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.080321+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:05:04.080321+0000","last_peered":"2024-03-08T23:05:04.080321+0000","last_clean":"2024-03-08T23:05:04.080321+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:05:04.080321+0000","last_undegraded":"2024-03-08T23:05:04.080321+0000","last_fullsized":"2024-03-08T23:05:04.080321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.134896+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:04:45.134896+0000","last_peered":"2024-03-08T23:04:45.134896+0000","last_clean":"2024-03-08T23:04:45.134896+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:04:45.134896+0000","last_undegraded":"2024-03-08T23:04:45.134896+0000","last_fullsized":"2024-03-08T23:04:45.134896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512734+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:04:45.512734+0000","last_peered":"2024-03-08T23:04:45.512734+0000","last_clean":"2024-03-08T23:04:45.512734+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:04:45.512734+0000","last_undegraded":"2024-03-08T23:04:45.512734+0000","last_fullsized":"2024-03-08T23:04:45.512734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411468+0000","last_change":"2024-03-08T23:04:38.331866+0000","last_active":"2024-03-08T23:04:44.411468+0000","last_peered":"2024-03-08T23:04:44.411468+0000","last_clean":"2024-03-08T23:04:44.411468+0000","last_became_active":"2024-03-08T23:04:38.331506+0000","last_became_peered":"2024-03-08T23:04:38.331506+0000","last_unstale":"2024-03-08T23:04:44.411468+0000","last_undegraded":"2024-03-08T23:04:44.411468+0000","last_fullsized":"2024-03-08T23:04:44.411468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:02:10.577391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.418089+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:04:44.418089+0000","last_peered":"2024-03-08T23:04:44.418089+0000","last_clean":"2024-03-08T23:04:44.418089+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:04:44.418089+0000","last_undegraded":"2024-03-08T23:04:44.418089+0000","last_fullsized":"2024-03-08T23:04:44.418089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513524+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:04:45.513524+0000","last_peered":"2024-03-08T23:04:45.513524+0000","last_clean":"2024-03-08T23:04:45.513524+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:04:45.513524+0000","last_undegraded":"2024-03-08T23:04:45.513524+0000","last_fullsized":"2024-03-08T23:04:45.513524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074648+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:05:04.074648+0000","last_peered":"2024-03-08T23:05:04.074648+0000","last_clean":"2024-03-08T23:05:04.074648+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:05:04.074648+0000","last_undegraded":"2024-03-08T23:05:04.074648+0000","last_fullsized":"2024-03-08T23:05:04.074648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514465+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:04:45.514465+0000","last_peered":"2024-03-08T23:04:45.514465+0000","last_clean":"2024-03-08T23:04:45.514465+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:04:45.514465+0000","last_undegraded":"2024-03-08T23:04:45.514465+0000","last_fullsized":"2024-03-08T23:04:45.514465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513204+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:04:45.513204+0000","last_peered":"2024-03-08T23:04:45.513204+0000","last_clean":"2024-03-08T23:04:45.513204+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:04:45.513204+0000","last_undegraded":"2024-03-08T23:04:45.513204+0000","last_fullsized":"2024-03-08T23:04:45.513204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411938+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:04:44.411938+0000","last_peered":"2024-03-08T23:04:44.411938+0000","last_clean":"2024-03-08T23:04:44.411938+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:04:44.411938+0000","last_undegraded":"2024-03-08T23:04:44.411938+0000","last_fullsized":"2024-03-08T23:04:44.411938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514516+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:04:45.514516+0000","last_peered":"2024-03-08T23:04:45.514516+0000","last_clean":"2024-03-08T23:04:45.514516+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:04:45.514516+0000","last_undegraded":"2024-03-08T23:04:45.514516+0000","last_fullsized":"2024-03-08T23:04:45.514516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514811+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:04:45.514811+0000","last_peered":"2024-03-08T23:04:45.514811+0000","last_clean":"2024-03-08T23:04:45.514811+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:04:45.514811+0000","last_undegraded":"2024-03-08T23:04:45.514811+0000","last_fullsized":"2024-03-08T23:04:45.514811+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460313+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:04:44.460313+0000","last_peered":"2024-03-08T23:04:44.460313+0000","last_clean":"2024-03-08T23:04:44.460313+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:04:44.460313+0000","last_undegraded":"2024-03-08T23:04:44.460313+0000","last_fullsized":"2024-03-08T23:04:44.460313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514026+0000","last_change":"2024-03-08T23:04:39.068576+0000","last_active":"2024-03-08T23:04:45.514026+0000","last_peered":"2024-03-08T23:04:45.514026+0000","last_clean":"2024-03-08T23:04:45.514026+0000","last_became_active":"2024-03-08T23:04:39.068139+0000","last_became_peered":"2024-03-08T23:04:39.068139+0000","last_unstale":"2024-03-08T23:04:45.514026+0000","last_undegraded":"2024-03-08T23:04:45.514026+0000","last_fullsized":"2024-03-08T23:04:45.514026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:24.688345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459573+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:04:44.459573+0000","last_peered":"2024-03-08T23:04:44.459573+0000","last_clean":"2024-03-08T23:04:44.459573+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:04:44.459573+0000","last_undegraded":"2024-03-08T23:04:44.459573+0000","last_fullsized":"2024-03-08T23:04:44.459573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513158+0000","last_change":"2024-03-08T23:04:38.341284+0000","last_active":"2024-03-08T23:04:45.513158+0000","last_peered":"2024-03-08T23:04:45.513158+0000","last_clean":"2024-03-08T23:04:45.513158+0000","last_became_active":"2024-03-08T23:04:38.340964+0000","last_became_peered":"2024-03-08T23:04:38.340964+0000","last_unstale":"2024-03-08T23:04:45.513158+0000","last_undegraded":"2024-03-08T23:04:45.513158+0000","last_fullsized":"2024-03-08T23:04:45.513158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:55:28.970771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514421+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:04:45.514421+0000","last_peered":"2024-03-08T23:04:45.514421+0000","last_clean":"2024-03-08T23:04:45.514421+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:04:45.514421+0000","last_undegraded":"2024-03-08T23:04:45.514421+0000","last_fullsized":"2024-03-08T23:04:45.514421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.134966+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:04:45.134966+0000","last_peered":"2024-03-08T23:04:45.134966+0000","last_clean":"2024-03-08T23:04:45.134966+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:04:45.134966+0000","last_undegraded":"2024-03-08T23:04:45.134966+0000","last_fullsized":"2024-03-08T23:04:45.134966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411902+0000","last_change":"2024-03-08T23:04:39.073037+0000","last_active":"2024-03-08T23:04:44.411902+0000","last_peered":"2024-03-08T23:04:44.411902+0000","last_clean":"2024-03-08T23:04:44.411902+0000","last_became_active":"2024-03-08T23:04:39.072677+0000","last_became_peered":"2024-03-08T23:04:39.072677+0000","last_unstale":"2024-03-08T23:04:44.411902+0000","last_undegraded":"2024-03-08T23:04:44.411902+0000","last_fullsized":"2024-03-08T23:04:44.411902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:36:53.863759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411494+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:04:44.411494+0000","last_peered":"2024-03-08T23:04:44.411494+0000","last_clean":"2024-03-08T23:04:44.411494+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:04:44.411494+0000","last_undegraded":"2024-03-08T23:04:44.411494+0000","last_fullsized":"2024-03-08T23:04:44.411494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.515212+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:04:45.515212+0000","last_peered":"2024-03-08T23:04:45.515212+0000","last_clean":"2024-03-08T23:04:45.515212+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:04:45.515212+0000","last_undegraded":"2024-03-08T23:04:45.515212+0000","last_fullsized":"2024-03-08T23:04:45.515212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513811+0000","last_change":"2024-03-08T23:04:39.102942+0000","last_active":"2024-03-08T23:04:45.513811+0000","last_peered":"2024-03-08T23:04:45.513811+0000","last_clean":"2024-03-08T23:04:45.513811+0000","last_became_active":"2024-03-08T23:04:39.100378+0000","last_became_peered":"2024-03-08T23:04:39.100378+0000","last_unstale":"2024-03-08T23:04:45.513811+0000","last_undegraded":"2024-03-08T23:04:45.513811+0000","last_fullsized":"2024-03-08T23:04:45.513811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:28.691558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135233+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:04:45.135233+0000","last_peered":"2024-03-08T23:04:45.135233+0000","last_clean":"2024-03-08T23:04:45.135233+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:04:45.135233+0000","last_undegraded":"2024-03-08T23:04:45.135233+0000","last_fullsized":"2024-03-08T23:04:45.135233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412146+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:04:44.412146+0000","last_peered":"2024-03-08T23:04:44.412146+0000","last_clean":"2024-03-08T23:04:44.412146+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:04:44.412146+0000","last_undegraded":"2024-03-08T23:04:44.412146+0000","last_fullsized":"2024-03-08T23:04:44.412146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419674+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:04:44.419674+0000","last_peered":"2024-03-08T23:04:44.419674+0000","last_clean":"2024-03-08T23:04:44.419674+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:04:44.419674+0000","last_undegraded":"2024-03-08T23:04:44.419674+0000","last_fullsized":"2024-03-08T23:04:44.419674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459738+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:04:44.459738+0000","last_peered":"2024-03-08T23:04:44.459738+0000","last_clean":"2024-03-08T23:04:44.459738+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:04:44.459738+0000","last_undegraded":"2024-03-08T23:04:44.459738+0000","last_fullsized":"2024-03-08T23:04:44.459738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.416216+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:04:44.416216+0000","last_peered":"2024-03-08T23:04:44.416216+0000","last_clean":"2024-03-08T23:04:44.416216+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:04:44.416216+0000","last_undegraded":"2024-03-08T23:04:44.416216+0000","last_fullsized":"2024-03-08T23:04:44.416216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513273+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:04:45.513273+0000","last_peered":"2024-03-08T23:04:45.513273+0000","last_clean":"2024-03-08T23:04:45.513273+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:04:45.513273+0000","last_undegraded":"2024-03-08T23:04:45.513273+0000","last_fullsized":"2024-03-08T23:04:45.513273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512590+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:04:45.512590+0000","last_peered":"2024-03-08T23:04:45.512590+0000","last_clean":"2024-03-08T23:04:45.512590+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:04:45.512590+0000","last_undegraded":"2024-03-08T23:04:45.512590+0000","last_fullsized":"2024-03-08T23:04:45.512590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135269+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:04:45.135269+0000","last_peered":"2024-03-08T23:04:45.135269+0000","last_clean":"2024-03-08T23:04:45.135269+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:04:45.135269+0000","last_undegraded":"2024-03-08T23:04:45.135269+0000","last_fullsized":"2024-03-08T23:04:45.135269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6639,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":120,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":21591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":486225,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":482049,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1078349,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1073546,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:05:10.235 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T23:05:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 -- 172.21.15.46:0/4096810484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c104ef0 msgr2=0x7fe80c10b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 --2- 172.21.15.46:0/4096810484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c104ef0 0x7fe80c10b780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe804009ca0 tx=0x7fe80402c0f0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 -- 172.21.15.46:0/4096810484 shutdown_connections 2024-03-08T23:05:10.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 --2- 172.21.15.46:0/4096810484 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c104ef0 0x7fe80c10b780 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 --2- 172.21.15.46:0/4096810484 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104530 0x7fe80c1049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 --2- 172.21.15.46:0/4096810484 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c103340 0x7fe80c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 -- 172.21.15.46:0/4096810484 >> 172.21.15.46:0/4096810484 conn(0x7fe80c0fe8d0 msgr2=0x7fe80c100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 -- 172.21.15.46:0/4096810484 shutdown_connections 2024-03-08T23:05:10.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.498+0000 7fe80be00640 1 -- 172.21.15.46:0/4096810484 wait complete. 2024-03-08T23:05:10.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.499+0000 7fe80be00640 1 Processor -- start 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.499+0000 7fe80be00640 1 -- start start 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.499+0000 7fe80be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c103340 0x7fe80c1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 0x7fe80c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c10efd0 con 0x7fe80c103340 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c10ee50 con 0x7fe80c104ef0 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80be00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80c10f150 con 0x7fe80c104530 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c103340 0x7fe80c1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33554/0 (socket says 172.21.15.46:33554) 2024-03-08T23:05:10.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 -- 172.21.15.46:0/3220697181 learned_addr learned my addr 172.21.15.46:0/3220697181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80b400640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 0x7fe80c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 -- 172.21.15.46:0/3220697181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 msgr2=0x7fe80c1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 0x7fe80c1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 -- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c103340 msgr2=0x7fe80c1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c103340 0x7fe80c1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe803e00640 1 -- 172.21.15.46:0/3220697181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c1a11d0 con 0x7fe80c104530 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.500+0000 7fe80b400640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 0x7fe80c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:10.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe803e00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400e9f0 tx=0x7fe7f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f400cd80 con 0x7fe80c104530 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80c1a14c0 con 0x7fe80c104530 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7f4010910 con 0x7fe80c104530 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f40188f0 con 0x7fe80c104530 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.501+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80c1a1a60 con 0x7fe80c104530 2024-03-08T23:05:10.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.502+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80c10a660 con 0x7fe80c104530 2024-03-08T23:05:10.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.504+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fe7f4010ac0 con 0x7fe80c104530 2024-03-08T23:05:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.506+0000 7fe803400640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 0x7fe7f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.506+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fe7f40cf600 con 0x7fe80c104530 2024-03-08T23:05:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.506+0000 7fe80aa00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 0x7fe7f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:10.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.507+0000 7fe80aa00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 0x7fe7f00a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe80c104390 tx=0x7fe7fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:10.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.508+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe7f409bc50 con 0x7fe80c104530 2024-03-08T23:05:10.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.709+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe80c061730 con 0x7fe7f00a2890 2024-03-08T23:05:10.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.735+0000 7fe803400640 1 -- 172.21.15.46:0/3220697181 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340624 (secure 0 0 0) 0x7fe80c061730 con 0x7fe7f00a2890 2024-03-08T23:05:10.741 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:10.746 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 msgr2=0x7fe7f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 0x7fe7f00a4d50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe80c104390 tx=0x7fe7fc005f90 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 msgr2=0x7fe80c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe7f400e9f0 tx=0x7fe7f400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 shutdown_connections 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe7f00a2890 0x7fe7f00a4d50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe80c104ef0 0x7fe80c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe80c104530 0x7fe80c199af0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 --2- 172.21.15.46:0/3220697181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe80c103340 0x7fe80c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.744+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 >> 172.21.15.46:0/3220697181 conn(0x7fe80c0fe8d0 msgr2=0x7fe80c107830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:10.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.745+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 shutdown_connections 2024-03-08T23:05:10.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:10.745+0000 7fe80be00640 1 -- 172.21.15.46:0/3220697181 wait complete. 2024-03-08T23:05:10.769 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-03-08T23:05:09.185243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597124,"num_objects":55,"num_object_clones":0,"num_object_copies":166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":152,"num_read_kb":132,"num_write":302,"num_write_kb":2982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":242,"ondisk_log_size":242,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360788,"kb_used_data":11636,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581036,"statfs":{"total":1151940427776,"available":1151570980864,"internally_reserved":0,"allocated":11915264,"data_stored":7542445,"data_compressed":284217,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3897,"num_objects":31,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":75,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":21,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005860"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135551+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:04:45.135551+0000","last_peered":"2024-03-08T23:04:45.135551+0000","last_clean":"2024-03-08T23:04:45.135551+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:04:45.135551+0000","last_undegraded":"2024-03-08T23:04:45.135551+0000","last_fullsized":"2024-03-08T23:04:45.135551+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514549+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:04:45.514549+0000","last_peered":"2024-03-08T23:04:45.514549+0000","last_clean":"2024-03-08T23:04:45.514549+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:04:45.514549+0000","last_undegraded":"2024-03-08T23:04:45.514549+0000","last_fullsized":"2024-03-08T23:04:45.514549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514232+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:04:45.514232+0000","last_peered":"2024-03-08T23:04:45.514232+0000","last_clean":"2024-03-08T23:04:45.514232+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:04:45.514232+0000","last_undegraded":"2024-03-08T23:04:45.514232+0000","last_fullsized":"2024-03-08T23:04:45.514232+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512894+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:04:45.512894+0000","last_peered":"2024-03-08T23:04:45.512894+0000","last_clean":"2024-03-08T23:04:45.512894+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:04:45.512894+0000","last_undegraded":"2024-03-08T23:04:45.512894+0000","last_fullsized":"2024-03-08T23:04:45.512894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621289+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:04:45.621289+0000","last_peered":"2024-03-08T23:04:45.621289+0000","last_clean":"2024-03-08T23:04:45.621289+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:04:45.621289+0000","last_undegraded":"2024-03-08T23:04:45.621289+0000","last_fullsized":"2024-03-08T23:04:45.621289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514000+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:04:45.514000+0000","last_peered":"2024-03-08T23:04:45.514000+0000","last_clean":"2024-03-08T23:04:45.514000+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:04:45.514000+0000","last_undegraded":"2024-03-08T23:04:45.514000+0000","last_fullsized":"2024-03-08T23:04:45.514000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514554+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:04:45.514554+0000","last_peered":"2024-03-08T23:04:45.514554+0000","last_clean":"2024-03-08T23:04:45.514554+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:04:45.514554+0000","last_undegraded":"2024-03-08T23:04:45.514554+0000","last_fullsized":"2024-03-08T23:04:45.514554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414748+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:04:44.414748+0000","last_peered":"2024-03-08T23:04:44.414748+0000","last_clean":"2024-03-08T23:04:44.414748+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:04:44.414748+0000","last_undegraded":"2024-03-08T23:04:44.414748+0000","last_fullsized":"2024-03-08T23:04:44.414748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513800+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:04:45.513800+0000","last_peered":"2024-03-08T23:04:45.513800+0000","last_clean":"2024-03-08T23:04:45.513800+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:04:45.513800+0000","last_undegraded":"2024-03-08T23:04:45.513800+0000","last_fullsized":"2024-03-08T23:04:45.513800+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514546+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:04:45.514546+0000","last_peered":"2024-03-08T23:04:45.514546+0000","last_clean":"2024-03-08T23:04:45.514546+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:04:45.514546+0000","last_undegraded":"2024-03-08T23:04:45.514546+0000","last_fullsized":"2024-03-08T23:04:45.514546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514580+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:04:45.514580+0000","last_peered":"2024-03-08T23:04:45.514580+0000","last_clean":"2024-03-08T23:04:45.514580+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:04:45.514580+0000","last_undegraded":"2024-03-08T23:04:45.514580+0000","last_fullsized":"2024-03-08T23:04:45.514580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512821+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:04:45.512821+0000","last_peered":"2024-03-08T23:04:45.512821+0000","last_clean":"2024-03-08T23:04:45.512821+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:04:45.512821+0000","last_undegraded":"2024-03-08T23:04:45.512821+0000","last_fullsized":"2024-03-08T23:04:45.512821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512787+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:04:45.512787+0000","last_peered":"2024-03-08T23:04:45.512787+0000","last_clean":"2024-03-08T23:04:45.512787+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:04:45.512787+0000","last_undegraded":"2024-03-08T23:04:45.512787+0000","last_fullsized":"2024-03-08T23:04:45.512787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514826+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:04:45.514826+0000","last_peered":"2024-03-08T23:04:45.514826+0000","last_clean":"2024-03-08T23:04:45.514826+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:04:45.514826+0000","last_undegraded":"2024-03-08T23:04:45.514826+0000","last_fullsized":"2024-03-08T23:04:45.514826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513821+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:04:45.513821+0000","last_peered":"2024-03-08T23:04:45.513821+0000","last_clean":"2024-03-08T23:04:45.513821+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:04:45.513821+0000","last_undegraded":"2024-03-08T23:04:45.513821+0000","last_fullsized":"2024-03-08T23:04:45.513821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512929+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:04:45.512929+0000","last_peered":"2024-03-08T23:04:45.512929+0000","last_clean":"2024-03-08T23:04:45.512929+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:04:45.512929+0000","last_undegraded":"2024-03-08T23:04:45.512929+0000","last_fullsized":"2024-03-08T23:04:45.512929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135376+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:04:45.135376+0000","last_peered":"2024-03-08T23:04:45.135376+0000","last_clean":"2024-03-08T23:04:45.135376+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:04:45.135376+0000","last_undegraded":"2024-03-08T23:04:45.135376+0000","last_fullsized":"2024-03-08T23:04:45.135376+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415777+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:04:44.415777+0000","last_peered":"2024-03-08T23:04:44.415777+0000","last_clean":"2024-03-08T23:04:44.415777+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:04:44.415777+0000","last_undegraded":"2024-03-08T23:04:44.415777+0000","last_fullsized":"2024-03-08T23:04:44.415777+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412124+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:04:44.412124+0000","last_peered":"2024-03-08T23:04:44.412124+0000","last_clean":"2024-03-08T23:04:44.412124+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:04:44.412124+0000","last_undegraded":"2024-03-08T23:04:44.412124+0000","last_fullsized":"2024-03-08T23:04:44.412124+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411532+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:04:44.411532+0000","last_peered":"2024-03-08T23:04:44.411532+0000","last_clean":"2024-03-08T23:04:44.411532+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:04:44.411532+0000","last_undegraded":"2024-03-08T23:04:44.411532+0000","last_fullsized":"2024-03-08T23:04:44.411532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.087917+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:05:05.087917+0000","last_peered":"2024-03-08T23:05:05.087917+0000","last_clean":"2024-03-08T23:05:05.087917+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:05:05.087917+0000","last_undegraded":"2024-03-08T23:05:05.087917+0000","last_fullsized":"2024-03-08T23:05:05.087917+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419183+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:04:44.419183+0000","last_peered":"2024-03-08T23:04:44.419183+0000","last_clean":"2024-03-08T23:04:44.419183+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:04:44.419183+0000","last_undegraded":"2024-03-08T23:04:44.419183+0000","last_fullsized":"2024-03-08T23:04:44.419183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512497+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:04:45.512497+0000","last_peered":"2024-03-08T23:04:45.512497+0000","last_clean":"2024-03-08T23:04:45.512497+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:04:45.512497+0000","last_undegraded":"2024-03-08T23:04:45.512497+0000","last_fullsized":"2024-03-08T23:04:45.512497+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419465+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:04:44.419465+0000","last_peered":"2024-03-08T23:04:44.419465+0000","last_clean":"2024-03-08T23:04:44.419465+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:04:44.419465+0000","last_undegraded":"2024-03-08T23:04:44.419465+0000","last_fullsized":"2024-03-08T23:04:44.419465+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514871+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:04:45.514871+0000","last_peered":"2024-03-08T23:04:45.514871+0000","last_clean":"2024-03-08T23:04:45.514871+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:04:45.514871+0000","last_undegraded":"2024-03-08T23:04:45.514871+0000","last_fullsized":"2024-03-08T23:04:45.514871+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513117+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:04:45.513117+0000","last_peered":"2024-03-08T23:04:45.513117+0000","last_clean":"2024-03-08T23:04:45.513117+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:04:45.513117+0000","last_undegraded":"2024-03-08T23:04:45.513117+0000","last_fullsized":"2024-03-08T23:04:45.513117+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.079260+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:05:05.079260+0000","last_peered":"2024-03-08T23:05:05.079260+0000","last_clean":"2024-03-08T23:05:05.079260+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:05:05.079260+0000","last_undegraded":"2024-03-08T23:05:05.079260+0000","last_fullsized":"2024-03-08T23:05:05.079260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513120+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:04:45.513120+0000","last_peered":"2024-03-08T23:04:45.513120+0000","last_clean":"2024-03-08T23:04:45.513120+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:04:45.513120+0000","last_undegraded":"2024-03-08T23:04:45.513120+0000","last_fullsized":"2024-03-08T23:04:45.513120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514691+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:04:45.514691+0000","last_peered":"2024-03-08T23:04:45.514691+0000","last_clean":"2024-03-08T23:04:45.514691+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:04:45.514691+0000","last_undegraded":"2024-03-08T23:04:45.514691+0000","last_fullsized":"2024-03-08T23:04:45.514691+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621628+0000","last_change":"2024-03-08T23:04:38.326247+0000","last_active":"2024-03-08T23:04:45.621628+0000","last_peered":"2024-03-08T23:04:45.621628+0000","last_clean":"2024-03-08T23:04:45.621628+0000","last_became_active":"2024-03-08T23:04:38.325843+0000","last_became_peered":"2024-03-08T23:04:38.325843+0000","last_unstale":"2024-03-08T23:04:45.621628+0000","last_undegraded":"2024-03-08T23:04:45.621628+0000","last_fullsized":"2024-03-08T23:04:45.621628+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:38:07.047189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460244+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:04:44.460244+0000","last_peered":"2024-03-08T23:04:44.460244+0000","last_clean":"2024-03-08T23:04:44.460244+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:04:44.460244+0000","last_undegraded":"2024-03-08T23:04:44.460244+0000","last_fullsized":"2024-03-08T23:04:44.460244+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412035+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:04:44.412035+0000","last_peered":"2024-03-08T23:04:44.412035+0000","last_clean":"2024-03-08T23:04:44.412035+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:04:44.412035+0000","last_undegraded":"2024-03-08T23:04:44.412035+0000","last_fullsized":"2024-03-08T23:04:44.412035+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:07.093822+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:05:07.093822+0000","last_peered":"2024-03-08T23:05:07.093822+0000","last_clean":"2024-03-08T23:05:07.093822+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:05:07.093822+0000","last_undegraded":"2024-03-08T23:05:07.093822+0000","last_fullsized":"2024-03-08T23:05:07.093822+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621353+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:04:45.621353+0000","last_peered":"2024-03-08T23:04:45.621353+0000","last_clean":"2024-03-08T23:04:45.621353+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:04:45.621353+0000","last_undegraded":"2024-03-08T23:04:45.621353+0000","last_fullsized":"2024-03-08T23:04:45.621353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411790+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:04:44.411790+0000","last_peered":"2024-03-08T23:04:44.411790+0000","last_clean":"2024-03-08T23:04:44.411790+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:04:44.411790+0000","last_undegraded":"2024-03-08T23:04:44.411790+0000","last_fullsized":"2024-03-08T23:04:44.411790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513972+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:04:45.513972+0000","last_peered":"2024-03-08T23:04:45.513972+0000","last_clean":"2024-03-08T23:04:45.513972+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:04:45.513972+0000","last_undegraded":"2024-03-08T23:04:45.513972+0000","last_fullsized":"2024-03-08T23:04:45.513972+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460110+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:04:44.460110+0000","last_peered":"2024-03-08T23:04:44.460110+0000","last_clean":"2024-03-08T23:04:44.460110+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:04:44.460110+0000","last_undegraded":"2024-03-08T23:04:44.460110+0000","last_fullsized":"2024-03-08T23:04:44.460110+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135480+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:04:45.135480+0000","last_peered":"2024-03-08T23:04:45.135480+0000","last_clean":"2024-03-08T23:04:45.135480+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:04:45.135480+0000","last_undegraded":"2024-03-08T23:04:45.135480+0000","last_fullsized":"2024-03-08T23:04:45.135480+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.088371+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:05:05.088371+0000","last_peered":"2024-03-08T23:05:05.088371+0000","last_clean":"2024-03-08T23:05:05.088371+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:05:05.088371+0000","last_undegraded":"2024-03-08T23:05:05.088371+0000","last_fullsized":"2024-03-08T23:05:05.088371+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415138+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:04:44.415138+0000","last_peered":"2024-03-08T23:04:44.415138+0000","last_clean":"2024-03-08T23:04:44.415138+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:04:44.415138+0000","last_undegraded":"2024-03-08T23:04:44.415138+0000","last_fullsized":"2024-03-08T23:04:44.415138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514476+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:04:45.514476+0000","last_peered":"2024-03-08T23:04:45.514476+0000","last_clean":"2024-03-08T23:04:45.514476+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:04:45.514476+0000","last_undegraded":"2024-03-08T23:04:45.514476+0000","last_fullsized":"2024-03-08T23:04:45.514476+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459799+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:04:44.459799+0000","last_peered":"2024-03-08T23:04:44.459799+0000","last_clean":"2024-03-08T23:04:44.459799+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:04:44.459799+0000","last_undegraded":"2024-03-08T23:04:44.459799+0000","last_fullsized":"2024-03-08T23:04:44.459799+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414688+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:04:44.414688+0000","last_peered":"2024-03-08T23:04:44.414688+0000","last_clean":"2024-03-08T23:04:44.414688+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:04:44.414688+0000","last_undegraded":"2024-03-08T23:04:44.414688+0000","last_fullsized":"2024-03-08T23:04:44.414688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415862+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:04:44.415862+0000","last_peered":"2024-03-08T23:04:44.415862+0000","last_clean":"2024-03-08T23:04:44.415862+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:04:44.415862+0000","last_undegraded":"2024-03-08T23:04:44.415862+0000","last_fullsized":"2024-03-08T23:04:44.415862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074292+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:05:04.074292+0000","last_peered":"2024-03-08T23:05:04.074292+0000","last_clean":"2024-03-08T23:05:04.074292+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:05:04.074292+0000","last_undegraded":"2024-03-08T23:05:04.074292+0000","last_fullsized":"2024-03-08T23:05:04.074292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513926+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:04:45.513926+0000","last_peered":"2024-03-08T23:04:45.513926+0000","last_clean":"2024-03-08T23:04:45.513926+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:04:45.513926+0000","last_undegraded":"2024-03-08T23:04:45.513926+0000","last_fullsized":"2024-03-08T23:04:45.513926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.414966+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:04:44.414966+0000","last_peered":"2024-03-08T23:04:44.414966+0000","last_clean":"2024-03-08T23:04:44.414966+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:04:44.414966+0000","last_undegraded":"2024-03-08T23:04:44.414966+0000","last_fullsized":"2024-03-08T23:04:44.414966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074386+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:05:04.074386+0000","last_peered":"2024-03-08T23:05:04.074386+0000","last_clean":"2024-03-08T23:05:04.074386+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:05:04.074386+0000","last_undegraded":"2024-03-08T23:05:04.074386+0000","last_fullsized":"2024-03-08T23:05:04.074386+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514928+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:04:45.514928+0000","last_peered":"2024-03-08T23:04:45.514928+0000","last_clean":"2024-03-08T23:04:45.514928+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:04:45.514928+0000","last_undegraded":"2024-03-08T23:04:45.514928+0000","last_fullsized":"2024-03-08T23:04:45.514928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513486+0000","last_change":"2024-03-08T23:04:43.446232+0000","last_active":"2024-03-08T23:04:45.513486+0000","last_peered":"2024-03-08T23:04:45.513486+0000","last_clean":"2024-03-08T23:04:45.513486+0000","last_became_active":"2024-03-08T23:04:43.445805+0000","last_became_peered":"2024-03-08T23:04:43.445805+0000","last_unstale":"2024-03-08T23:04:45.513486+0000","last_undegraded":"2024-03-08T23:04:45.513486+0000","last_fullsized":"2024-03-08T23:04:45.513486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:24.425824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.076844+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:05:04.076844+0000","last_peered":"2024-03-08T23:05:04.076844+0000","last_clean":"2024-03-08T23:05:04.076844+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:05:04.076844+0000","last_undegraded":"2024-03-08T23:05:04.076844+0000","last_fullsized":"2024-03-08T23:05:04.076844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513325+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:04:45.513325+0000","last_peered":"2024-03-08T23:04:45.513325+0000","last_clean":"2024-03-08T23:04:45.513325+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:04:45.513325+0000","last_undegraded":"2024-03-08T23:04:45.513325+0000","last_fullsized":"2024-03-08T23:04:45.513325+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.416310+0000","last_change":"2024-03-08T23:04:43.443827+0000","last_active":"2024-03-08T23:04:44.416310+0000","last_peered":"2024-03-08T23:04:44.416310+0000","last_clean":"2024-03-08T23:04:44.416310+0000","last_became_active":"2024-03-08T23:04:43.442112+0000","last_became_peered":"2024-03-08T23:04:43.442112+0000","last_unstale":"2024-03-08T23:04:44.416310+0000","last_undegraded":"2024-03-08T23:04:44.416310+0000","last_fullsized":"2024-03-08T23:04:44.416310+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:21:31.929876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.073437+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:05:04.073437+0000","last_peered":"2024-03-08T23:05:04.073437+0000","last_clean":"2024-03-08T23:05:04.073437+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:05:04.073437+0000","last_undegraded":"2024-03-08T23:05:04.073437+0000","last_fullsized":"2024-03-08T23:05:04.073437+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621549+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:04:45.621549+0000","last_peered":"2024-03-08T23:04:45.621549+0000","last_clean":"2024-03-08T23:04:45.621549+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:04:45.621549+0000","last_undegraded":"2024-03-08T23:04:45.621549+0000","last_fullsized":"2024-03-08T23:04:45.621549+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460531+0000","last_change":"2024-03-08T23:04:43.439283+0000","last_active":"2024-03-08T23:04:44.460531+0000","last_peered":"2024-03-08T23:04:44.460531+0000","last_clean":"2024-03-08T23:04:44.460531+0000","last_became_active":"2024-03-08T23:04:43.438890+0000","last_became_peered":"2024-03-08T23:04:43.438890+0000","last_unstale":"2024-03-08T23:04:44.460531+0000","last_undegraded":"2024-03-08T23:04:44.460531+0000","last_fullsized":"2024-03-08T23:04:44.460531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:45:05.798040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.078406+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:05:04.078406+0000","last_peered":"2024-03-08T23:05:04.078406+0000","last_clean":"2024-03-08T23:05:04.078406+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:05:04.078406+0000","last_undegraded":"2024-03-08T23:05:04.078406+0000","last_fullsized":"2024-03-08T23:05:04.078406+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512382+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:04:45.512382+0000","last_peered":"2024-03-08T23:04:45.512382+0000","last_clean":"2024-03-08T23:04:45.512382+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:04:45.512382+0000","last_undegraded":"2024-03-08T23:04:45.512382+0000","last_fullsized":"2024-03-08T23:04:45.512382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135298+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:04:45.135298+0000","last_peered":"2024-03-08T23:04:45.135298+0000","last_clean":"2024-03-08T23:04:45.135298+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:04:45.135298+0000","last_undegraded":"2024-03-08T23:04:45.135298+0000","last_fullsized":"2024-03-08T23:04:45.135298+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.077219+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:05:04.077219+0000","last_peered":"2024-03-08T23:05:04.077219+0000","last_clean":"2024-03-08T23:05:04.077219+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:05:04.077219+0000","last_undegraded":"2024-03-08T23:05:04.077219+0000","last_fullsized":"2024-03-08T23:05:04.077219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514247+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:04:45.514247+0000","last_peered":"2024-03-08T23:04:45.514247+0000","last_clean":"2024-03-08T23:04:45.514247+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:04:45.514247+0000","last_undegraded":"2024-03-08T23:04:45.514247+0000","last_fullsized":"2024-03-08T23:04:45.514247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.417458+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:04:44.417458+0000","last_peered":"2024-03-08T23:04:44.417458+0000","last_clean":"2024-03-08T23:04:44.417458+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:04:44.417458+0000","last_undegraded":"2024-03-08T23:04:44.417458+0000","last_fullsized":"2024-03-08T23:04:44.417458+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513022+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:04:45.513022+0000","last_peered":"2024-03-08T23:04:45.513022+0000","last_clean":"2024-03-08T23:04:45.513022+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:04:45.513022+0000","last_undegraded":"2024-03-08T23:04:45.513022+0000","last_fullsized":"2024-03-08T23:04:45.513022+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513812+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:04:45.513812+0000","last_peered":"2024-03-08T23:04:45.513812+0000","last_clean":"2024-03-08T23:04:45.513812+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:04:45.513812+0000","last_undegraded":"2024-03-08T23:04:45.513812+0000","last_fullsized":"2024-03-08T23:04:45.513812+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514146+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:04:45.514146+0000","last_peered":"2024-03-08T23:04:45.514146+0000","last_clean":"2024-03-08T23:04:45.514146+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:04:45.514146+0000","last_undegraded":"2024-03-08T23:04:45.514146+0000","last_fullsized":"2024-03-08T23:04:45.514146+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":39,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.088392+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:05:05.088392+0000","last_peered":"2024-03-08T23:05:05.088392+0000","last_clean":"2024-03-08T23:05:05.088392+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:05:05.088392+0000","last_undegraded":"2024-03-08T23:05:05.088392+0000","last_fullsized":"2024-03-08T23:05:05.088392+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459745+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:04:44.459745+0000","last_peered":"2024-03-08T23:04:44.459745+0000","last_clean":"2024-03-08T23:04:44.459745+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:04:44.459745+0000","last_undegraded":"2024-03-08T23:04:44.459745+0000","last_fullsized":"2024-03-08T23:04:44.459745+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459802+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:04:44.459802+0000","last_peered":"2024-03-08T23:04:44.459802+0000","last_clean":"2024-03-08T23:04:44.459802+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:04:44.459802+0000","last_undegraded":"2024-03-08T23:04:44.459802+0000","last_fullsized":"2024-03-08T23:04:44.459802+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:06.076740+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:05:06.076740+0000","last_peered":"2024-03-08T23:05:06.076740+0000","last_clean":"2024-03-08T23:05:06.076740+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:05:06.076740+0000","last_undegraded":"2024-03-08T23:05:06.076740+0000","last_fullsized":"2024-03-08T23:05:06.076740+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135220+0000","last_change":"2024-03-08T23:04:39.075656+0000","last_active":"2024-03-08T23:04:45.135220+0000","last_peered":"2024-03-08T23:04:45.135220+0000","last_clean":"2024-03-08T23:04:45.135220+0000","last_became_active":"2024-03-08T23:04:39.075211+0000","last_became_peered":"2024-03-08T23:04:39.075211+0000","last_unstale":"2024-03-08T23:04:45.135220+0000","last_undegraded":"2024-03-08T23:04:45.135220+0000","last_fullsized":"2024-03-08T23:04:45.135220+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:04:14.996902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621378+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:04:45.621378+0000","last_peered":"2024-03-08T23:04:45.621378+0000","last_clean":"2024-03-08T23:04:45.621378+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:04:45.621378+0000","last_undegraded":"2024-03-08T23:04:45.621378+0000","last_fullsized":"2024-03-08T23:04:45.621378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415211+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:04:44.415211+0000","last_peered":"2024-03-08T23:04:44.415211+0000","last_clean":"2024-03-08T23:04:44.415211+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:04:44.415211+0000","last_undegraded":"2024-03-08T23:04:44.415211+0000","last_fullsized":"2024-03-08T23:04:44.415211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514015+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:04:45.514015+0000","last_peered":"2024-03-08T23:04:45.514015+0000","last_clean":"2024-03-08T23:04:45.514015+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:04:45.514015+0000","last_undegraded":"2024-03-08T23:04:45.514015+0000","last_fullsized":"2024-03-08T23:04:45.514015+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513630+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:04:45.513630+0000","last_peered":"2024-03-08T23:04:45.513630+0000","last_clean":"2024-03-08T23:04:45.513630+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:04:45.513630+0000","last_undegraded":"2024-03-08T23:04:45.513630+0000","last_fullsized":"2024-03-08T23:04:45.513630+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514324+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:04:45.514324+0000","last_peered":"2024-03-08T23:04:45.514324+0000","last_clean":"2024-03-08T23:04:45.514324+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:04:45.514324+0000","last_undegraded":"2024-03-08T23:04:45.514324+0000","last_fullsized":"2024-03-08T23:04:45.514324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513165+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:04:45.513165+0000","last_peered":"2024-03-08T23:04:45.513165+0000","last_clean":"2024-03-08T23:04:45.513165+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:04:45.513165+0000","last_undegraded":"2024-03-08T23:04:45.513165+0000","last_fullsized":"2024-03-08T23:04:45.513165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513147+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:04:45.513147+0000","last_peered":"2024-03-08T23:04:45.513147+0000","last_clean":"2024-03-08T23:04:45.513147+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:04:45.513147+0000","last_undegraded":"2024-03-08T23:04:45.513147+0000","last_fullsized":"2024-03-08T23:04:45.513147+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459935+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:04:44.459935+0000","last_peered":"2024-03-08T23:04:44.459935+0000","last_clean":"2024-03-08T23:04:44.459935+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:04:44.459935+0000","last_undegraded":"2024-03-08T23:04:44.459935+0000","last_fullsized":"2024-03-08T23:04:44.459935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:06.090444+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:05:06.090444+0000","last_peered":"2024-03-08T23:05:06.090444+0000","last_clean":"2024-03-08T23:05:06.090444+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:05:06.090444+0000","last_undegraded":"2024-03-08T23:05:06.090444+0000","last_fullsized":"2024-03-08T23:05:06.090444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513863+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:04:45.513863+0000","last_peered":"2024-03-08T23:04:45.513863+0000","last_clean":"2024-03-08T23:04:45.513863+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:04:45.513863+0000","last_undegraded":"2024-03-08T23:04:45.513863+0000","last_fullsized":"2024-03-08T23:04:45.513863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.417776+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:04:44.417776+0000","last_peered":"2024-03-08T23:04:44.417776+0000","last_clean":"2024-03-08T23:04:44.417776+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:04:44.417776+0000","last_undegraded":"2024-03-08T23:04:44.417776+0000","last_fullsized":"2024-03-08T23:04:44.417776+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.075117+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:05:04.075117+0000","last_peered":"2024-03-08T23:05:04.075117+0000","last_clean":"2024-03-08T23:05:04.075117+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:05:04.075117+0000","last_undegraded":"2024-03-08T23:05:04.075117+0000","last_fullsized":"2024-03-08T23:05:04.075117+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":358,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513560+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:04:45.513560+0000","last_peered":"2024-03-08T23:04:45.513560+0000","last_clean":"2024-03-08T23:04:45.513560+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:04:45.513560+0000","last_undegraded":"2024-03-08T23:04:45.513560+0000","last_fullsized":"2024-03-08T23:04:45.513560+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460352+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:04:44.460352+0000","last_peered":"2024-03-08T23:04:44.460352+0000","last_clean":"2024-03-08T23:04:44.460352+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:04:44.460352+0000","last_undegraded":"2024-03-08T23:04:44.460352+0000","last_fullsized":"2024-03-08T23:04:44.460352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512135+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:04:45.512135+0000","last_peered":"2024-03-08T23:04:45.512135+0000","last_clean":"2024-03-08T23:04:45.512135+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:04:45.512135+0000","last_undegraded":"2024-03-08T23:04:45.512135+0000","last_fullsized":"2024-03-08T23:04:45.512135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:05.079554+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:05:05.079554+0000","last_peered":"2024-03-08T23:05:05.079554+0000","last_clean":"2024-03-08T23:05:05.079554+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:05:05.079554+0000","last_undegraded":"2024-03-08T23:05:05.079554+0000","last_fullsized":"2024-03-08T23:05:05.079554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135240+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:04:45.135240+0000","last_peered":"2024-03-08T23:04:45.135240+0000","last_clean":"2024-03-08T23:04:45.135240+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:04:45.135240+0000","last_undegraded":"2024-03-08T23:04:45.135240+0000","last_fullsized":"2024-03-08T23:04:45.135240+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.075265+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:05:04.075265+0000","last_peered":"2024-03-08T23:05:04.075265+0000","last_clean":"2024-03-08T23:05:04.075265+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:05:04.075265+0000","last_undegraded":"2024-03-08T23:05:04.075265+0000","last_fullsized":"2024-03-08T23:05:04.075265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513717+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:04:45.513717+0000","last_peered":"2024-03-08T23:04:45.513717+0000","last_clean":"2024-03-08T23:04:45.513717+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:04:45.513717+0000","last_undegraded":"2024-03-08T23:04:45.513717+0000","last_fullsized":"2024-03-08T23:04:45.513717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.415074+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:04:44.415074+0000","last_peered":"2024-03-08T23:04:44.415074+0000","last_clean":"2024-03-08T23:04:44.415074+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:04:44.415074+0000","last_undegraded":"2024-03-08T23:04:44.415074+0000","last_fullsized":"2024-03-08T23:04:44.415074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:07.365999+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:05:07.365999+0000","last_peered":"2024-03-08T23:05:07.365999+0000","last_clean":"2024-03-08T23:05:07.365999+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:05:07.365999+0000","last_undegraded":"2024-03-08T23:05:07.365999+0000","last_fullsized":"2024-03-08T23:05:07.365999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419475+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:04:44.419475+0000","last_peered":"2024-03-08T23:04:44.419475+0000","last_clean":"2024-03-08T23:04:44.419475+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:04:44.419475+0000","last_undegraded":"2024-03-08T23:04:44.419475+0000","last_fullsized":"2024-03-08T23:04:44.419475+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411633+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:04:44.411633+0000","last_peered":"2024-03-08T23:04:44.411633+0000","last_clean":"2024-03-08T23:04:44.411633+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:04:44.411633+0000","last_undegraded":"2024-03-08T23:04:44.411633+0000","last_fullsized":"2024-03-08T23:04:44.411633+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.072998+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:05:04.072998+0000","last_peered":"2024-03-08T23:05:04.072998+0000","last_clean":"2024-03-08T23:05:04.072998+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:05:04.072998+0000","last_undegraded":"2024-03-08T23:05:04.072998+0000","last_fullsized":"2024-03-08T23:05:04.072998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514650+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:04:45.514650+0000","last_peered":"2024-03-08T23:04:45.514650+0000","last_clean":"2024-03-08T23:04:45.514650+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:04:45.514650+0000","last_undegraded":"2024-03-08T23:04:45.514650+0000","last_fullsized":"2024-03-08T23:04:45.514650+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512618+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:04:45.512618+0000","last_peered":"2024-03-08T23:04:45.512618+0000","last_clean":"2024-03-08T23:04:45.512618+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:04:45.512618+0000","last_undegraded":"2024-03-08T23:04:45.512618+0000","last_fullsized":"2024-03-08T23:04:45.512618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.078099+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:05:04.078099+0000","last_peered":"2024-03-08T23:05:04.078099+0000","last_clean":"2024-03-08T23:05:04.078099+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:05:04.078099+0000","last_undegraded":"2024-03-08T23:05:04.078099+0000","last_fullsized":"2024-03-08T23:05:04.078099+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.621187+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:04:45.621187+0000","last_peered":"2024-03-08T23:04:45.621187+0000","last_clean":"2024-03-08T23:04:45.621187+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:04:45.621187+0000","last_undegraded":"2024-03-08T23:04:45.621187+0000","last_fullsized":"2024-03-08T23:04:45.621187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.418567+0000","last_change":"2024-03-08T23:04:43.441758+0000","last_active":"2024-03-08T23:04:44.418567+0000","last_peered":"2024-03-08T23:04:44.418567+0000","last_clean":"2024-03-08T23:04:44.418567+0000","last_became_active":"2024-03-08T23:04:43.441297+0000","last_became_peered":"2024-03-08T23:04:43.441297+0000","last_unstale":"2024-03-08T23:04:44.418567+0000","last_undegraded":"2024-03-08T23:04:44.418567+0000","last_fullsized":"2024-03-08T23:04:44.418567+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:43:56.396622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.080321+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:05:04.080321+0000","last_peered":"2024-03-08T23:05:04.080321+0000","last_clean":"2024-03-08T23:05:04.080321+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:05:04.080321+0000","last_undegraded":"2024-03-08T23:05:04.080321+0000","last_fullsized":"2024-03-08T23:05:04.080321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.134896+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:04:45.134896+0000","last_peered":"2024-03-08T23:04:45.134896+0000","last_clean":"2024-03-08T23:04:45.134896+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:04:45.134896+0000","last_undegraded":"2024-03-08T23:04:45.134896+0000","last_fullsized":"2024-03-08T23:04:45.134896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512734+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:04:45.512734+0000","last_peered":"2024-03-08T23:04:45.512734+0000","last_clean":"2024-03-08T23:04:45.512734+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:04:45.512734+0000","last_undegraded":"2024-03-08T23:04:45.512734+0000","last_fullsized":"2024-03-08T23:04:45.512734+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411468+0000","last_change":"2024-03-08T23:04:38.331866+0000","last_active":"2024-03-08T23:04:44.411468+0000","last_peered":"2024-03-08T23:04:44.411468+0000","last_clean":"2024-03-08T23:04:44.411468+0000","last_became_active":"2024-03-08T23:04:38.331506+0000","last_became_peered":"2024-03-08T23:04:38.331506+0000","last_unstale":"2024-03-08T23:04:44.411468+0000","last_undegraded":"2024-03-08T23:04:44.411468+0000","last_fullsized":"2024-03-08T23:04:44.411468+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:02:10.577391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.418089+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:04:44.418089+0000","last_peered":"2024-03-08T23:04:44.418089+0000","last_clean":"2024-03-08T23:04:44.418089+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:04:44.418089+0000","last_undegraded":"2024-03-08T23:04:44.418089+0000","last_fullsized":"2024-03-08T23:04:44.418089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513524+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:04:45.513524+0000","last_peered":"2024-03-08T23:04:45.513524+0000","last_clean":"2024-03-08T23:04:45.513524+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:04:45.513524+0000","last_undegraded":"2024-03-08T23:04:45.513524+0000","last_fullsized":"2024-03-08T23:04:45.513524+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:05:04.074648+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:05:04.074648+0000","last_peered":"2024-03-08T23:05:04.074648+0000","last_clean":"2024-03-08T23:05:04.074648+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:05:04.074648+0000","last_undegraded":"2024-03-08T23:05:04.074648+0000","last_fullsized":"2024-03-08T23:05:04.074648+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514465+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:04:45.514465+0000","last_peered":"2024-03-08T23:04:45.514465+0000","last_clean":"2024-03-08T23:04:45.514465+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:04:45.514465+0000","last_undegraded":"2024-03-08T23:04:45.514465+0000","last_fullsized":"2024-03-08T23:04:45.514465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513204+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:04:45.513204+0000","last_peered":"2024-03-08T23:04:45.513204+0000","last_clean":"2024-03-08T23:04:45.513204+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:04:45.513204+0000","last_undegraded":"2024-03-08T23:04:45.513204+0000","last_fullsized":"2024-03-08T23:04:45.513204+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411938+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:04:44.411938+0000","last_peered":"2024-03-08T23:04:44.411938+0000","last_clean":"2024-03-08T23:04:44.411938+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:04:44.411938+0000","last_undegraded":"2024-03-08T23:04:44.411938+0000","last_fullsized":"2024-03-08T23:04:44.411938+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514516+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:04:45.514516+0000","last_peered":"2024-03-08T23:04:45.514516+0000","last_clean":"2024-03-08T23:04:45.514516+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:04:45.514516+0000","last_undegraded":"2024-03-08T23:04:45.514516+0000","last_fullsized":"2024-03-08T23:04:45.514516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514811+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:04:45.514811+0000","last_peered":"2024-03-08T23:04:45.514811+0000","last_clean":"2024-03-08T23:04:45.514811+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:04:45.514811+0000","last_undegraded":"2024-03-08T23:04:45.514811+0000","last_fullsized":"2024-03-08T23:04:45.514811+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.460313+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:04:44.460313+0000","last_peered":"2024-03-08T23:04:44.460313+0000","last_clean":"2024-03-08T23:04:44.460313+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:04:44.460313+0000","last_undegraded":"2024-03-08T23:04:44.460313+0000","last_fullsized":"2024-03-08T23:04:44.460313+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514026+0000","last_change":"2024-03-08T23:04:39.068576+0000","last_active":"2024-03-08T23:04:45.514026+0000","last_peered":"2024-03-08T23:04:45.514026+0000","last_clean":"2024-03-08T23:04:45.514026+0000","last_became_active":"2024-03-08T23:04:39.068139+0000","last_became_peered":"2024-03-08T23:04:39.068139+0000","last_unstale":"2024-03-08T23:04:45.514026+0000","last_undegraded":"2024-03-08T23:04:45.514026+0000","last_fullsized":"2024-03-08T23:04:45.514026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:24.688345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459573+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:04:44.459573+0000","last_peered":"2024-03-08T23:04:44.459573+0000","last_clean":"2024-03-08T23:04:44.459573+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:04:44.459573+0000","last_undegraded":"2024-03-08T23:04:44.459573+0000","last_fullsized":"2024-03-08T23:04:44.459573+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513158+0000","last_change":"2024-03-08T23:04:38.341284+0000","last_active":"2024-03-08T23:04:45.513158+0000","last_peered":"2024-03-08T23:04:45.513158+0000","last_clean":"2024-03-08T23:04:45.513158+0000","last_became_active":"2024-03-08T23:04:38.340964+0000","last_became_peered":"2024-03-08T23:04:38.340964+0000","last_unstale":"2024-03-08T23:04:45.513158+0000","last_undegraded":"2024-03-08T23:04:45.513158+0000","last_fullsized":"2024-03-08T23:04:45.513158+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:55:28.970771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.514421+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:04:45.514421+0000","last_peered":"2024-03-08T23:04:45.514421+0000","last_clean":"2024-03-08T23:04:45.514421+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:04:45.514421+0000","last_undegraded":"2024-03-08T23:04:45.514421+0000","last_fullsized":"2024-03-08T23:04:45.514421+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.134966+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:04:45.134966+0000","last_peered":"2024-03-08T23:04:45.134966+0000","last_clean":"2024-03-08T23:04:45.134966+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:04:45.134966+0000","last_undegraded":"2024-03-08T23:04:45.134966+0000","last_fullsized":"2024-03-08T23:04:45.134966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411902+0000","last_change":"2024-03-08T23:04:39.073037+0000","last_active":"2024-03-08T23:04:44.411902+0000","last_peered":"2024-03-08T23:04:44.411902+0000","last_clean":"2024-03-08T23:04:44.411902+0000","last_became_active":"2024-03-08T23:04:39.072677+0000","last_became_peered":"2024-03-08T23:04:39.072677+0000","last_unstale":"2024-03-08T23:04:44.411902+0000","last_undegraded":"2024-03-08T23:04:44.411902+0000","last_fullsized":"2024-03-08T23:04:44.411902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:36:53.863759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.411494+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:04:44.411494+0000","last_peered":"2024-03-08T23:04:44.411494+0000","last_clean":"2024-03-08T23:04:44.411494+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:04:44.411494+0000","last_undegraded":"2024-03-08T23:04:44.411494+0000","last_fullsized":"2024-03-08T23:04:44.411494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.515212+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:04:45.515212+0000","last_peered":"2024-03-08T23:04:45.515212+0000","last_clean":"2024-03-08T23:04:45.515212+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:04:45.515212+0000","last_undegraded":"2024-03-08T23:04:45.515212+0000","last_fullsized":"2024-03-08T23:04:45.515212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513811+0000","last_change":"2024-03-08T23:04:39.102942+0000","last_active":"2024-03-08T23:04:45.513811+0000","last_peered":"2024-03-08T23:04:45.513811+0000","last_clean":"2024-03-08T23:04:45.513811+0000","last_became_active":"2024-03-08T23:04:39.100378+0000","last_became_peered":"2024-03-08T23:04:39.100378+0000","last_unstale":"2024-03-08T23:04:45.513811+0000","last_undegraded":"2024-03-08T23:04:45.513811+0000","last_fullsized":"2024-03-08T23:04:45.513811+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:28.691558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135233+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:04:45.135233+0000","last_peered":"2024-03-08T23:04:45.135233+0000","last_clean":"2024-03-08T23:04:45.135233+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:04:45.135233+0000","last_undegraded":"2024-03-08T23:04:45.135233+0000","last_fullsized":"2024-03-08T23:04:45.135233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.412146+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:04:44.412146+0000","last_peered":"2024-03-08T23:04:44.412146+0000","last_clean":"2024-03-08T23:04:44.412146+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:04:44.412146+0000","last_undegraded":"2024-03-08T23:04:44.412146+0000","last_fullsized":"2024-03-08T23:04:44.412146+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.419674+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:04:44.419674+0000","last_peered":"2024-03-08T23:04:44.419674+0000","last_clean":"2024-03-08T23:04:44.419674+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:04:44.419674+0000","last_undegraded":"2024-03-08T23:04:44.419674+0000","last_fullsized":"2024-03-08T23:04:44.419674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.459738+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:04:44.459738+0000","last_peered":"2024-03-08T23:04:44.459738+0000","last_clean":"2024-03-08T23:04:44.459738+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:04:44.459738+0000","last_undegraded":"2024-03-08T23:04:44.459738+0000","last_fullsized":"2024-03-08T23:04:44.459738+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:44.416216+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:04:44.416216+0000","last_peered":"2024-03-08T23:04:44.416216+0000","last_clean":"2024-03-08T23:04:44.416216+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:04:44.416216+0000","last_undegraded":"2024-03-08T23:04:44.416216+0000","last_fullsized":"2024-03-08T23:04:44.416216+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.513273+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:04:45.513273+0000","last_peered":"2024-03-08T23:04:45.513273+0000","last_clean":"2024-03-08T23:04:45.513273+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:04:45.513273+0000","last_undegraded":"2024-03-08T23:04:45.513273+0000","last_fullsized":"2024-03-08T23:04:45.513273+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.512590+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:04:45.512590+0000","last_peered":"2024-03-08T23:04:45.512590+0000","last_clean":"2024-03-08T23:04:45.512590+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:04:45.512590+0000","last_undegraded":"2024-03-08T23:04:45.512590+0000","last_fullsized":"2024-03-08T23:04:45.512590+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-08T23:04:45.135269+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:04:45.135269+0000","last_peered":"2024-03-08T23:04:45.135269+0000","last_clean":"2024-03-08T23:04:45.135269+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:04:45.135269+0000","last_undegraded":"2024-03-08T23:04:45.135269+0000","last_fullsized":"2024-03-08T23:04:45.135269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":24576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6639,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":16,"num_read_kb":13,"num_write":120,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":35,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":21591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645335,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1048576,"data_stored":1072381,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808862,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":486225,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070501,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":482049,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332137,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1078349,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495663,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":974848,"data_stored":477909,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":978944,"data_stored":481091,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182463,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":477499,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920827,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14633,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757301,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1073546,"data_compressed":50840,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:05:10.776 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-08T23:05:10.776 INFO:tasks.cephadm:Setting up client nodes... 2024-03-08T23:05:10.777 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-08T23:05:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:10 smithi192 ceph-mon[36700]: pgmap v334: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-08T23:05:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:10 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1397384149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:10.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:10 smithi062 ceph-mon[35761]: pgmap v334: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-08T23:05:10.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:10 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1397384149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:10.900 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:10 smithi046 ceph-mon[31859]: pgmap v334: 129 pgs: 129 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2024-03-08T23:05:10.900 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:10 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1397384149' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:11.292 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:11.712 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:11 smithi046 ceph-mon[31859]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:11.712 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:11 smithi046 ceph-mon[31859]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:11.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:11 smithi192 ceph-mon[36700]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:11.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:11 smithi192 ceph-mon[36700]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:11.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:11 smithi062 ceph-mon[35761]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:11.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:11 smithi062 ceph-mon[35761]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:05:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:12 smithi046 ceph-mon[31859]: pgmap v335: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-03-08T23:05:12.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:12 smithi192 ceph-mon[36700]: pgmap v335: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-03-08T23:05:12.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:12 smithi062 ceph-mon[35761]: pgmap v335: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 6.3 KiB/s wr, 12 op/s 2024-03-08T23:05:13.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 -- 172.21.15.46:0/2832477177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 msgr2=0x7fc560076bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 --2- 172.21.15.46:0/2832477177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc560076bb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc5540098f0 tx=0x7fc554009c00 comp rx=0 tx=0).stop 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 -- 172.21.15.46:0/2832477177 shutdown_connections 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 --2- 172.21.15.46:0/2832477177 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 0x7fc560078c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 --2- 172.21.15.46:0/2832477177 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc560076bb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 --2- 172.21.15.46:0/2832477177 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc560077e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.704+0000 7fc568000700 1 -- 172.21.15.46:0/2832477177 >> 172.21.15.46:0/2832477177 conn(0x7fc5600ff600 msgr2=0x7fc560101a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.705+0000 7fc568000700 1 -- 172.21.15.46:0/2832477177 shutdown_connections 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.705+0000 7fc568000700 1 -- 172.21.15.46:0/2832477177 wait complete. 2024-03-08T23:05:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 Processor -- start 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 -- start start 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc56019eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 0x7fc5601a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560114060 con 0x7fc5600783a0 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc560113ee0 con 0x7fc560076790 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc568000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5601141e0 con 0x7fc5600779e0 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.706+0000 7fc564c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc56019eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:13.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc566000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 0x7fc5601a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc564c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc56019eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:40202/0 (socket says 172.21.15.46:40202) 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36812/0 (socket says 172.21.15.46:36812) 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc564c00700 1 -- 172.21.15.46:0/1919719810 learned_addr learned my addr 172.21.15.46:0/1919719810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 -- 172.21.15.46:0/1919719810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 msgr2=0x7fc56019eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc56019eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 -- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 msgr2=0x7fc5601a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 0x7fc5601a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 -- 172.21.15.46:0/1919719810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5601a6210 con 0x7fc560076790 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.707+0000 7fc565600700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc554009230 tx=0x7fc554005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.708+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400f950 con 0x7fc560076790 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.708+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc55400ba40 con 0x7fc560076790 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.708+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5601a64a0 con 0x7fc560076790 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.708+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc554003ba0 con 0x7fc560076790 2024-03-08T23:05:13.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.708+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5601a6950 con 0x7fc560076790 2024-03-08T23:05:13.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.709+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5600502f0 con 0x7fc560076790 2024-03-08T23:05:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.711+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc55400bbf0 con 0x7fc560076790 2024-03-08T23:05:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.712+0000 7fc55e000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 0x7fc5480a50b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.712+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc5540cfd60 con 0x7fc560076790 2024-03-08T23:05:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.712+0000 7fc564c00700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 0x7fc5480a50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.713+0000 7fc564c00700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 0x7fc5480a50b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009d20 tx=0x7fc54c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:13.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:13.715+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc554013030 con 0x7fc560076790 2024-03-08T23:05:14.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.037+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc56002d830 con 0x7fc560076790 2024-03-08T23:05:14.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.044+0000 7fc55e000700 1 -- 172.21.15.46:0/1919719810 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc55409ca90 con 0x7fc560076790 2024-03-08T23:05:14.047 INFO:teuthology.orchestra.run.smithi046.stdout:[client.0] 2024-03-08T23:05:14.047 INFO:teuthology.orchestra.run.smithi046.stdout: key = AQCqmetl+elSAhAAHPWZvLCY96qCsgl9SwmHsg== 2024-03-08T23:05:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 msgr2=0x7fc5480a50b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 0x7fc5480a50b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc54c009d20 tx=0x7fc54c009260 comp rx=0 tx=0).stop 2024-03-08T23:05:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 msgr2=0x7fc56019e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:14.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc554009230 tx=0x7fc554005250 comp rx=0 tx=0).stop 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 shutdown_connections 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc5480a2bf0 0x7fc5480a50b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc5600783a0 0x7fc5601a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc560076790 0x7fc56019e600 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 --2- 172.21.15.46:0/1919719810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc5600779e0 0x7fc56019eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 >> 172.21.15.46:0/1919719810 conn(0x7fc5600ff600 msgr2=0x7fc560101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 shutdown_connections 2024-03-08T23:05:14.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:14.047+0000 7fc568000700 1 -- 172.21.15.46:0/1919719810 wait complete. 2024-03-08T23:05:14.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:14 smithi046 ceph-mon[31859]: pgmap v336: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:14.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1919719810' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:14.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:14 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:14.832 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:14 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-08T23:05:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:14 smithi062 ceph-mon[35761]: pgmap v336: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1919719810' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:14 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:14.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:14 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-08T23:05:14.901 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:05:14.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-08T23:05:14.902 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-08T23:05:15.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:14 smithi192 ceph-mon[36700]: pgmap v336: 129 pgs: 129 active+clean; 622 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:15.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1919719810' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:15.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:14 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-08T23:05:15.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:14 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-08T23:05:15.072 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-08T23:05:15.073 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-08T23:05:15.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-08T23:05:15.576 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:16.777 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:16 smithi046 ceph-mon[31859]: pgmap v337: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-08T23:05:16.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:16 smithi062 ceph-mon[35761]: pgmap v337: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-08T23:05:17.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:16 smithi192 ceph-mon[36700]: pgmap v337: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.8 KiB/s wr, 15 op/s 2024-03-08T23:05:17.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.704+0000 7f85b3e00700 1 -- 172.21.15.46:0/1470409602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 msgr2=0x7f85ac108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.704+0000 7f85b3e00700 1 --2- 172.21.15.46:0/1470409602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac108290 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009920 tx=0x7f85a0009c30 comp rx=0 tx=0).stop 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 -- 172.21.15.46:0/1470409602 shutdown_connections 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 --2- 172.21.15.46:0/1470409602 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac10ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 --2- 172.21.15.46:0/1470409602 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac108290 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 --2- 172.21.15.46:0/1470409602 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 -- 172.21.15.46:0/1470409602 >> 172.21.15.46:0/1470409602 conn(0x7f85ac078f30 msgr2=0x7f85ac079340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 -- 172.21.15.46:0/1470409602 shutdown_connections 2024-03-08T23:05:17.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.705+0000 7f85b3e00700 1 -- 172.21.15.46:0/1470409602 wait complete. 2024-03-08T23:05:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.707+0000 7f85b3e00700 1 Processor -- start 2024-03-08T23:05:17.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.707+0000 7f85b3e00700 1 -- start start 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac19ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac1120d0 con 0x7f85ac102820 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac111f50 con 0x7f85ac101e60 2024-03-08T23:05:17.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b3e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac112250 con 0x7f85ac107eb0 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b1400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36816/0 (socket says 172.21.15.46:36816) 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b1400700 1 -- 172.21.15.46:0/135561912 learned_addr learned my addr 172.21.15.46:0/135561912 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b0a00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.708+0000 7f85b1e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac19ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:17.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 -- 172.21.15.46:0/135561912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 msgr2=0x7f85ac19ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac19ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 -- 172.21.15.46:0/135561912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 msgr2=0x7f85ac1985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 -- 172.21.15.46:0/135561912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac19d5d0 con 0x7f85ac102820 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b1400700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b0a00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac198b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f859800e9f0 tx=0x7f859800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859800c9c0 con 0x7f85ac102820 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f859800cb60 con 0x7f85ac102820 2024-03-08T23:05:17.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac19d8c0 con 0x7f85ac102820 2024-03-08T23:05:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8598018680 con 0x7f85ac102820 2024-03-08T23:05:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.709+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac1a5140 con 0x7f85ac102820 2024-03-08T23:05:17.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.711+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac10a370 con 0x7f85ac102820 2024-03-08T23:05:17.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.713+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8598010430 con 0x7f85ac102820 2024-03-08T23:05:17.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.714+0000 7f85aa000700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 0x7f85940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:17.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.714+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f85980cf160 con 0x7f85ac102820 2024-03-08T23:05:17.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.714+0000 7f85b1400700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 0x7f85940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:17.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.715+0000 7f85b1400700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 0x7f85940a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009680 tx=0x7f85a0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:17.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:17.721+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f859809be00 con 0x7f85ac102820 2024-03-08T23:05:18.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.016+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f85ac109d50 con 0x7f85ac102820 2024-03-08T23:05:18.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.026+0000 7f85aa000700 1 -- 172.21.15.46:0/135561912 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262658 (secure 0 0 0) 0x7f859809b700 con 0x7f85ac102820 2024-03-08T23:05:18.035 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.033+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 msgr2=0x7f85940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.033+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 0x7f85940a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f85a0009680 tx=0x7f85a0019040 comp rx=0 tx=0).stop 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.033+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 msgr2=0x7f85ac198b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.033+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac198b00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f859800e9f0 tx=0x7f859800ed00 comp rx=0 tx=0).stop 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 shutdown_connections 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f85940a2920 0x7f85940a4de0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f85ac107eb0 0x7f85ac19ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f85ac101e60 0x7f85ac1985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:18.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 --2- 172.21.15.46:0/135561912 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f85ac102820 0x7f85ac198b00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 >> 172.21.15.46:0/135561912 conn(0x7f85ac078f30 msgr2=0x7f85ac100ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 shutdown_connections 2024-03-08T23:05:18.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:18.034+0000 7f85b3e00700 1 -- 172.21.15.46:0/135561912 wait complete. 2024-03-08T23:05:18.675 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:18 smithi046 ceph-mon[31859]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:18.675 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:18 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/135561912' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-08T23:05:18.752 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6800","nonce":757291496},{"type":"v1","addr":"172.21.15.46:6801","nonce":757291496}]},"active_addr":"172.21.15.46:6801/757291496","active_change":"2024-03-08T22:56:39.034651+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.46:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1737358861}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1465757142}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1832130111}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1962446760}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.46:0","nonce":1469842539}]}]} 2024-03-08T23:05:18.757 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-08T23:05:18.757 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-08T23:05:18.757 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-08T23:05:18.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:18 smithi062 ceph-mon[35761]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:18.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:18 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/135561912' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-08T23:05:19.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:18 smithi192 ceph-mon[36700]: pgmap v338: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 6.1 KiB/s wr, 12 op/s 2024-03-08T23:05:19.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:18 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/135561912' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-08T23:05:19.285 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:20.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:20 smithi062 ceph-mon[35761]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-08T23:05:20.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:20 smithi046 ceph-mon[31859]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-08T23:05:21.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:20 smithi192 ceph-mon[36700]: pgmap v339: 129 pgs: 129 active+clean; 628 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.0 KiB/s wr, 13 op/s 2024-03-08T23:05:21.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.464+0000 7f7ef2400700 1 -- 172.21.15.46:0/1805776509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec104be0 msgr2=0x7f7eec108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:21.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.464+0000 7f7ef2400700 1 --2- 172.21.15.46:0/1805776509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec104be0 0x7f7eec108c30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7ed80099c0 tx=0x7f7ed8009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 -- 172.21.15.46:0/1805776509 shutdown_connections 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 --2- 172.21.15.46:0/1805776509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec109310 0x7f7eec110ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 --2- 172.21.15.46:0/1805776509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 --2- 172.21.15.46:0/1805776509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec104be0 0x7f7eec108c30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 -- 172.21.15.46:0/1805776509 >> 172.21.15.46:0/1805776509 conn(0x7f7eec0ffac0 msgr2=0x7f7eec101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 -- 172.21.15.46:0/1805776509 shutdown_connections 2024-03-08T23:05:21.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.465+0000 7f7ef2400700 1 -- 172.21.15.46:0/1805776509 wait complete. 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.466+0000 7f7ef2400700 1 Processor -- start 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.466+0000 7f7ef2400700 1 -- start start 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.466+0000 7f7ef2400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.466+0000 7f7ef2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec104be0 0x7f7eec19ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7ef2400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec109310 0x7f7eec19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7ef2400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec114340 con 0x7f7eec104230 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7ef2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1141c0 con 0x7f7eec104be0 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7ef2400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1144c0 con 0x7f7eec109310 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeaa00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec104be0 0x7f7eec19ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:35958/0 (socket says 172.21.15.46:35958) 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 -- 172.21.15.46:0/2923874989 learned_addr learned my addr 172.21.15.46:0/2923874989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:21.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eebe00700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec109310 0x7f7eec19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 -- 172.21.15.46:0/2923874989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec109310 msgr2=0x7f7eec19f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec109310 0x7f7eec19f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 -- 172.21.15.46:0/2923874989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec104be0 msgr2=0x7f7eec19ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec104be0 0x7f7eec19ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.467+0000 7f7eeb400700 1 -- 172.21.15.46:0/2923874989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec19f760 con 0x7f7eec104230 2024-03-08T23:05:21.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7eeb400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e9a0 tx=0x7f7edc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00ca10 con 0x7f7eec104230 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7edc00cbb0 con 0x7f7eec104230 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec19fa50 con 0x7f7eec104230 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc018680 con 0x7f7eec104230 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.468+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec1a72d0 con 0x7f7eec104230 2024-03-08T23:05:21.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.469+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec068aa0 con 0x7f7eec104230 2024-03-08T23:05:21.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.471+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7edc010430 con 0x7f7eec104230 2024-03-08T23:05:21.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.473+0000 7f7ee3e00700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 0x7f7ed00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.474+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7edc0cf0d0 con 0x7f7eec104230 2024-03-08T23:05:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.474+0000 7f7eeaa00700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 0x7f7ed00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:21.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.475+0000 7f7eeaa00700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 0x7f7ed00a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8000c00 tx=0x7f7ed8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:21.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.478+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7edc09bd70 con 0x7f7eec104230 2024-03-08T23:05:21.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.724+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7eec1a7560 con 0x7f7eec104230 2024-03-08T23:05:21.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.726+0000 7f7ee3e00700 1 -- 172.21.15.46:0/2923874989 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f7edc09b670 con 0x7f7eec104230 2024-03-08T23:05:21.729 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:21.730 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 msgr2=0x7f7ed00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 0x7f7ed00a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8000c00 tx=0x7f7ed8005fb0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 msgr2=0x7f7eec19a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00e9a0 tx=0x7f7edc00ecb0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 shutdown_connections 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ed00a2920 0x7f7ed00a4de0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7eec109310 0x7f7eec19f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7eec104230 0x7f7eec19a750 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 --2- 172.21.15.46:0/2923874989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7eec104be0 0x7f7eec19ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:21.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 >> 172.21.15.46:0/2923874989 conn(0x7f7eec0ffac0 msgr2=0x7f7eec100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:21.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 shutdown_connections 2024-03-08T23:05:21.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:21.729+0000 7f7ef2400700 1 -- 172.21.15.46:0/2923874989 wait complete. 2024-03-08T23:05:22.541 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-08T23:05:22.541 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-08T23:05:22.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:22 smithi046 ceph-mon[31859]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-08T23:05:22.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:22 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2923874989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:22.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:22 smithi062 ceph-mon[35761]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-08T23:05:22.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:22 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2923874989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:23.025 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:23.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:22 smithi192 ceph-mon[36700]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 5.7 KiB/s wr, 11 op/s 2024-03-08T23:05:23.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:22 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2923874989' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:24.678 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:24 smithi046 ceph-mon[31859]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-08T23:05:24.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:24 smithi062 ceph-mon[35761]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-08T23:05:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:24 smithi192 ceph-mon[36700]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 -- 172.21.15.46:0/1177196532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a18069a90 msgr2=0x7f7a18069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1177196532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a18069a90 0x7f7a18069ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7a000099c0 tx=0x7f7a00009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 -- 172.21.15.46:0/1177196532 shutdown_connections 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1177196532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a1806a430 0x7f7a1810ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1177196532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 0x7f7a18108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1177196532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a18069a90 0x7f7a18069ef0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 -- 172.21.15.46:0/1177196532 >> 172.21.15.46:0/1177196532 conn(0x7f7a18077c10 msgr2=0x7f7a18078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 -- 172.21.15.46:0/1177196532 shutdown_connections 2024-03-08T23:05:25.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.229+0000 7f7a1e200700 1 -- 172.21.15.46:0/1177196532 wait complete. 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.230+0000 7f7a1e200700 1 Processor -- start 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 -- start start 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a18069a90 0x7f7a181a0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 0x7f7a1819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18112090 con 0x7f7a1806a430 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18111f10 con 0x7f7a18107e30 2024-03-08T23:05:25.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a1e200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a18112210 con 0x7f7a18069a90 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a0ea00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a0ea00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:41168/0 (socket says 172.21.15.46:41168) 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.231+0000 7f7a0ea00700 1 -- 172.21.15.46:0/1153364511 learned_addr learned my addr 172.21.15.46:0/1153364511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a17400700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a18069a90 0x7f7a181a0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a17e00700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 0x7f7a1819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 -- 172.21.15.46:0/1153364511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a18069a90 msgr2=0x7f7a181a0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a18069a90 0x7f7a181a0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 -- 172.21.15.46:0/1153364511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 msgr2=0x7f7a1819a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 0x7f7a1819a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 -- 172.21.15.46:0/1153364511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a1819ad50 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.232+0000 7f7a0ea00700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006010 tx=0x7f7a00003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.233+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a00003680 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.233+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a000055a0 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.233+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1819afe0 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.233+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0000fe50 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.233+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a181a7300 con 0x7f7a1806a430 2024-03-08T23:05:25.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.234+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a18068aa0 con 0x7f7a1806a430 2024-03-08T23:05:25.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.236+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7a0000f580 con 0x7f7a1806a430 2024-03-08T23:05:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.238+0000 7f7a14c00700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 0x7f79fc0a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.238+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7a000cfa50 con 0x7f7a1806a430 2024-03-08T23:05:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.238+0000 7f7a17400700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 0x7f79fc0a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:25.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.239+0000 7f7a17400700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 0x7f79fc0a51a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009be0 tx=0x7f7a08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:25.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.243+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7a00013070 con 0x7f7a1806a430 2024-03-08T23:05:25.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.470+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a181a74d0 con 0x7f7a1806a430 2024-03-08T23:05:25.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.473+0000 7f7a14c00700 1 -- 172.21.15.46:0/1153364511 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24104 (secure 0 0 0) 0x7f7a0009c780 con 0x7f7a1806a430 2024-03-08T23:05:25.477 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:05:25.477 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":89,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:04:43.888785+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:05:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 msgr2=0x7f79fc0a51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 0x7f79fc0a51a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009be0 tx=0x7f7a08009260 comp rx=0 tx=0).stop 2024-03-08T23:05:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 msgr2=0x7f7a181a0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:25.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7a00006010 tx=0x7f7a00003a90 comp rx=0 tx=0).stop 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 shutdown_connections 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f79fc0a2ce0 0x7f79fc0a51a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7a18107e30 0x7f7a1819a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7a18069a90 0x7f7a181a0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 --2- 172.21.15.46:0/1153364511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7a1806a430 0x7f7a181a0970 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 >> 172.21.15.46:0/1153364511 conn(0x7f7a18077c10 msgr2=0x7f7a1810cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 shutdown_connections 2024-03-08T23:05:25.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:25.476+0000 7f7a1e200700 1 -- 172.21.15.46:0/1153364511 wait complete. 2024-03-08T23:05:25.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:25 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1153364511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:25.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:25 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1153364511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:26.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:25 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1153364511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:05:26.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-08T23:05:26.273 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-08T23:05:26.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-08T23:05:26.274 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-08T23:05:26.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-08T23:05:26.275 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-08T23:05:26.276 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-08T23:05:26.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-08T23:05:26.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-08T23:05:26.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-08T23:05:26.278 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-08T23:05:26.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-08T23:05:26.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:26 smithi062 ceph-mon[35761]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-08T23:05:26.965 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:26 smithi192 ceph-mon[36700]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-08T23:05:26.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:26 smithi046 ceph-mon[31859]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2024-03-08T23:05:27.324 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.339 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.367 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.372 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.385 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:27.671 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:28.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:28 smithi046 ceph-mon[31859]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-08T23:05:28.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:28 smithi062 ceph-mon[35761]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-08T23:05:29.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:28 smithi192 ceph-mon[36700]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-08T23:05:30.628 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:30 smithi046 ceph-mon[31859]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-08T23:05:30.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:30 smithi062 ceph-mon[35761]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-08T23:05:31.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:30 smithi192 ceph-mon[36700]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2024-03-08T23:05:31.444 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:31.444 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:32.874 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:32 smithi046 ceph-mon[31859]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-08T23:05:32.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:32 smithi062 ceph-mon[35761]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-08T23:05:33.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:32 smithi192 ceph-mon[36700]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2024-03-08T23:05:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:34.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:34 smithi046 ceph-mon[31859]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:34.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:34 smithi062 ceph-mon[35761]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:35.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:34 smithi192 ceph-mon[36700]: pgmap v346: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:36.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:36 smithi046 ceph-mon[31859]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-08T23:05:36.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:36 smithi062 ceph-mon[35761]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-08T23:05:37.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:36 smithi192 ceph-mon[36700]: pgmap v347: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 341 B/s wr, 7 op/s 2024-03-08T23:05:38.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:38 smithi046 ceph-mon[31859]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:38.684 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:38 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:05:38.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:38 smithi062 ceph-mon[35761]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:38.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:38 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:05:39.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:38 smithi192 ceph-mon[36700]: pgmap v348: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-03-08T23:05:39.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:38 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.720 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-08T23:05:39.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:39 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-08T23:05:39.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-08T23:05:39.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:39 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-08T23:05:39.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-08T23:05:39.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:39 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]: dispatch 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-08T23:05:40.773 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: osdmap e90: 12 total, 12 up, 12 in 2024-03-08T23:05:40.774 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:40.774 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:40.774 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:40.774 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 -- 172.21.15.46:0/3895356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 msgr2=0x7f931010ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 --2- 172.21.15.46:0/3895356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 0x7f931010ecd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f930800b0a0 tx=0x7f930800b3b0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 -- 172.21.15.46:0/3895356226 shutdown_connections 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 --2- 172.21.15.46:0/3895356226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 0x7f931010ecd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 --2- 172.21.15.46:0/3895356226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f931006a6e0 0x7f931006aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 --2- 172.21.15.46:0/3895356226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006b090 0x7f9310106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 -- 172.21.15.46:0/3895356226 >> 172.21.15.46:0/3895356226 conn(0x7f9310077c50 msgr2=0x7f9310078060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 -- 172.21.15.46:0/3895356226 shutdown_connections 2024-03-08T23:05:40.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.500+0000 7f9318600700 1 -- 172.21.15.46:0/3895356226 wait complete. 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.501+0000 7f9318600700 1 Processor -- start 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 -- start start 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 0x7f9310100da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f931006b090 0x7f93101012e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 0x7f9310101820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9310112140 con 0x7f931006b090 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9310111fc0 con 0x7f9310107150 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9318600700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93101122c0 con 0x7f931006a6e0 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9315c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 0x7f9310100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9315c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 0x7f9310100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37340/0 (socket says 172.21.15.46:37340) 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9315c00700 1 -- 172.21.15.46:0/1931661819 learned_addr learned my addr 172.21.15.46:0/1931661819 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:40.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9316600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 0x7f9310101820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.502+0000 7f9315200700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f931006b090 0x7f93101012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 -- 172.21.15.46:0/1931661819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 msgr2=0x7f9310101820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9310107150 0x7f9310101820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 -- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f931006b090 msgr2=0x7f93101012e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f931006b090 0x7f93101012e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93101a6b60 con 0x7f931006a6e0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f9315c00700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 0x7f9310100da0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f930400e9a0 tx=0x7f930400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.503+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f930400ca10 con 0x7f931006a6e0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.504+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f930400cbb0 con 0x7f931006a6e0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.504+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93101a6e50 con 0x7f931006a6e0 2024-03-08T23:05:40.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.504+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304018680 con 0x7f931006a6e0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.504+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93101a7300 con 0x7f931006a6e0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.505+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f93100624f0 con 0x7f931006a6e0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.507+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9304047a90 con 0x7f931006a6e0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.508+0000 7f930e000700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92f80a2920 0x7f92f80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.508+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f93040cf2c0 con 0x7f931006a6e0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.508+0000 7f9315200700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92f80a2920 0x7f92f80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.508+0000 7f930e000700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] conn(0x7f92f80ada00 0x7f92f80afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:40.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.508+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f92f80b0570 con 0x7f92f80ada00 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.509+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93040cf670 con 0x7f931006a6e0 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.509+0000 7f9316600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] conn(0x7f92f80ada00 0x7f92f80afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.509+0000 7f9315200700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92f80a2920 0x7f92f80a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f93101008f0 tx=0x7f92fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.510+0000 7f9316600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] conn(0x7f92f80ada00 0x7f92f80afea0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.511+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== osd.0 v2:172.21.15.46:6802/3833351406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f92f80b0570 con 0x7f92f80ada00 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.530+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 --> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f93100628f0 con 0x7f92f80ada00 2024-03-08T23:05:40.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.530+0000 7f930e000700 1 -- 172.21.15.46:0/1931661819 <== osd.0 v2:172.21.15.46:6802/3833351406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f93100628f0 con 0x7f92f80ada00 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] conn(0x7f92f80ada00 msgr2=0x7f92f80afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] conn(0x7f92f80ada00 0x7f92f80afea0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92f80a2920 msgr2=0x7f92f80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f92f80a2920 0x7f92f80a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f93101008f0 tx=0x7f92fc009260 comp rx=0 tx=0).stop 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 msgr2=0x7f9310100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 --2- 172.21.15.46:0/1931661819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f931006a6e0 0x7f9310100da0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f930400e9a0 tx=0x7f930400ecb0 comp rx=0 tx=0).stop 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9316600700 1 -- 172.21.15.46:0/1931661819 reap_dead start 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 shutdown_connections 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 >> 172.21.15.46:0/1931661819 conn(0x7f9310077c50 msgr2=0x7f93100fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.531+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 shutdown_connections 2024-03-08T23:05:40.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:40.532+0000 7f9318600700 1 -- 172.21.15.46:0/1931661819 wait complete. 2024-03-08T23:05:41.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: osdmap e90: 12 total, 12 up, 12 in 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: pgmap v349: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 341 B/s wr, 6 op/s 2024-03-08T23:05:41.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.e", "id": [2, 4]}]': finished 2024-03-08T23:05:41.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-08T23:05:41.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 6]}]': finished 2024-03-08T23:05:41.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [4, 8, 5, 9]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [5, 10]}]': finished 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: osdmap e90: 12 total, 12 up, 12 in 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:05:41.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:41 smithi046 ceph-mon[31859]: osdmap e91: 12 total, 12 up, 12 in 2024-03-08T23:05:42.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:41 smithi192 ceph-mon[36700]: osdmap e91: 12 total, 12 up, 12 in 2024-03-08T23:05:42.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:41 smithi062 ceph-mon[35761]: osdmap e91: 12 total, 12 up, 12 in 2024-03-08T23:05:42.415 INFO:teuthology.orchestra.run.smithi046.stdout:34359738455 2024-03-08T23:05:42.416 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-08T23:05:42.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:42 smithi046 ceph-mon[31859]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-08T23:05:43.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:42 smithi192 ceph-mon[36700]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-08T23:05:43.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:42 smithi062 ceph-mon[35761]: pgmap v352: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-08T23:05:43.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 -- 172.21.15.46:0/2126741224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438102090 msgr2=0x7f2438102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 --2- 172.21.15.46:0/2126741224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438102090 0x7f2438102470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f24280099c0 tx=0x7f2428009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 -- 172.21.15.46:0/2126741224 shutdown_connections 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 --2- 172.21.15.46:0/2126741224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2438107160 0x7f243810ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 --2- 172.21.15.46:0/2126741224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438102090 0x7f2438102470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 --2- 172.21.15.46:0/2126741224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102a40 0x7f2438106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 -- 172.21.15.46:0/2126741224 >> 172.21.15.46:0/2126741224 conn(0x7f2438078f40 msgr2=0x7f2438079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 -- 172.21.15.46:0/2126741224 shutdown_connections 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.299+0000 7f243d600700 1 -- 172.21.15.46:0/2126741224 wait complete. 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.300+0000 7f243d600700 1 Processor -- start 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.300+0000 7f243d600700 1 -- start start 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 0x7f24381a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2438102a40 0x7f24381a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438107160 0x7f243819a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438112160 con 0x7f2438102a40 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2438111fe0 con 0x7f2438102090 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f243d600700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24381122e0 con 0x7f2438107160 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 0x7f24381a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 0x7f24381a05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53218/0 (socket says 172.21.15.46:53218) 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 -- 172.21.15.46:0/1114894852 learned_addr learned my addr 172.21.15.46:0/1114894852 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f242e000700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2438102a40 0x7f24381a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2437400700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438107160 0x7f243819a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 -- 172.21.15.46:0/1114894852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438107160 msgr2=0x7f243819a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2438107160 0x7f243819a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 -- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2438102a40 msgr2=0x7f24381a0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2438102a40 0x7f24381a0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.301+0000 7f2436a00700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243819af70 con 0x7f2438102090 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f2436a00700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 0x7f24381a05c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f24280059d0 tx=0x7f242800fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2428027070 con 0x7f2438102090 2024-03-08T23:05:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f243819b200 con 0x7f2438102090 2024-03-08T23:05:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2428003680 con 0x7f2438102090 2024-03-08T23:05:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24280173f0 con 0x7f2438102090 2024-03-08T23:05:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.302+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24381a7540 con 0x7f2438102090 2024-03-08T23:05:43.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.303+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2438106400 con 0x7f2438102090 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.305+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f242802a070 con 0x7f2438102090 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.306+0000 7f242fe00700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f241c0a29a0 0x7f241c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f24280cfe20 con 0x7f2438102090 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f242e000700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f241c0a29a0 0x7f241c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f242fe00700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] conn(0x7f241c0ada80 0x7f241c0aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2428010ea0 con 0x7f241c0ada80 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f24280d01d0 con 0x7f2438102090 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.307+0000 7f2437400700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] conn(0x7f241c0ada80 0x7f241c0aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.308+0000 7f242e000700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f241c0a29a0 0x7f241c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f243819be50 tx=0x7f242000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.308+0000 7f2437400700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] conn(0x7f241c0ada80 0x7f241c0aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.309+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== osd.6 v2:172.21.15.46:6818/2295220560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2428010ea0 con 0x7f241c0ada80 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.327+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 --> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2438103560 con 0x7f241c0ada80 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f242fe00700 1 -- 172.21.15.46:0/1114894852 <== osd.6 v2:172.21.15.46:6818/2295220560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2438103560 con 0x7f241c0ada80 2024-03-08T23:05:43.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] conn(0x7f241c0ada80 msgr2=0x7f241c0aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] conn(0x7f241c0ada80 0x7f241c0aff20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f241c0a29a0 msgr2=0x7f241c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f241c0a29a0 0x7f241c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f243819be50 tx=0x7f242000b3c0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 msgr2=0x7f24381a05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 --2- 172.21.15.46:0/1114894852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2438102090 0x7f24381a05c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f24280059d0 tx=0x7f242800fa00 comp rx=0 tx=0).stop 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f2437400700 1 -- 172.21.15.46:0/1114894852 reap_dead start 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 shutdown_connections 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 >> 172.21.15.46:0/1114894852 conn(0x7f2438078f40 msgr2=0x7f2438105a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.328+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 shutdown_connections 2024-03-08T23:05:43.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.329+0000 7f243d600700 1 -- 172.21.15.46:0/1114894852 wait complete. 2024-03-08T23:05:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 -- 172.21.15.46:0/3997806274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 msgr2=0x7fcc94104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 --2- 172.21.15.46:0/3997806274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc94104610 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcc840099c0 tx=0x7fcc84009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 -- 172.21.15.46:0/3997806274 shutdown_connections 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 --2- 172.21.15.46:0/3997806274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94109310 0x7fcc94110ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 --2- 172.21.15.46:0/3997806274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc94104610 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 --2- 172.21.15.46:0/3997806274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc94104be0 0x7fcc94108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 -- 172.21.15.46:0/3997806274 >> 172.21.15.46:0/3997806274 conn(0x7fcc940ffac0 msgr2=0x7fcc94101ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 -- 172.21.15.46:0/3997806274 shutdown_connections 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.412+0000 7fcc9a400700 1 -- 172.21.15.46:0/3997806274 wait complete. 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.414+0000 7fcc9a400700 1 Processor -- start 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.414+0000 7fcc9a400700 1 -- start start 2024-03-08T23:05:43.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.414+0000 7fcc9a400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc941a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.414+0000 7fcc9a400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc94104be0 0x7fcc941a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.414+0000 7fcc9a400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94109310 0x7fcc9419a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc9a400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc94114340 con 0x7fcc94104be0 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc9a400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc941141c0 con 0x7fcc94109310 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc9a400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc941144c0 con 0x7fcc94104230 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc941a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc92a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc94104be0 0x7fcc941a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc941a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37318/0 (socket says 172.21.15.46:37318) 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 -- 172.21.15.46:0/1683072781 learned_addr learned my addr 172.21.15.46:0/1683072781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93e00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94109310 0x7fcc9419a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 -- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94109310 msgr2=0x7fcc9419a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc94109310 0x7fcc9419a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 -- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc94104be0 msgr2=0x7fcc941a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc94104be0 0x7fcc941a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.415+0000 7fcc93400700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9419ad00 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc93400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc941a03e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcc84006010 tx=0x7fcc84003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc84003e50 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc840055a0 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc8400fe30 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc9419af90 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.416+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc941a72b0 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.417+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcc94062570 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.419+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcc8400f4f0 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.420+0000 7fcc8be00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcc780a2b40 0x7fcc780a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fcc840cfff0 con 0x7fcc94104230 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc92a00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcc780a2b40 0x7fcc780a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc8be00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] conn(0x7fcc780ad800 0x7fcc780afc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc8400b790 con 0x7fcc780ad800 2024-03-08T23:05:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc840d0420 con 0x7fcc94104230 2024-03-08T23:05:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc92a00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcc780a2b40 0x7fcc780a5000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009680 tx=0x7fcc7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.421+0000 7fcc93e00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] conn(0x7fcc780ad800 0x7fcc780afc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.422+0000 7fcc93e00700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] conn(0x7fcc780ad800 0x7fcc780afc40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.423+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== osd.7 v2:172.21.15.62:6816/1007444458 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcc8400b790 con 0x7fcc780ad800 2024-03-08T23:05:43.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.443+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 --> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcc94062970 con 0x7fcc780ad800 2024-03-08T23:05:43.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.444+0000 7fcc8be00700 1 -- 172.21.15.46:0/1683072781 <== osd.7 v2:172.21.15.62:6816/1007444458 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcc94062970 con 0x7fcc780ad800 2024-03-08T23:05:43.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] conn(0x7fcc780ad800 msgr2=0x7fcc780afc40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] conn(0x7fcc780ad800 0x7fcc780afc40 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcc780a2b40 msgr2=0x7fcc780a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcc780a2b40 0x7fcc780a5000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c009680 tx=0x7fcc7c009260 comp rx=0 tx=0).stop 2024-03-08T23:05:43.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 msgr2=0x7fcc941a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 --2- 172.21.15.46:0/1683072781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc94104230 0x7fcc941a03e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcc84006010 tx=0x7fcc84003a90 comp rx=0 tx=0).stop 2024-03-08T23:05:43.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc93e00700 1 -- 172.21.15.46:0/1683072781 reap_dead start 2024-03-08T23:05:43.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 shutdown_connections 2024-03-08T23:05:43.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 >> 172.21.15.46:0/1683072781 conn(0x7fcc940ffac0 msgr2=0x7fcc94101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 shutdown_connections 2024-03-08T23:05:43.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:39.445+0000 7fcc9a400700 1 -- 172.21.15.46:0/1683072781 wait complete. 2024-03-08T23:05:43.803 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:43 smithi046 ceph-mon[31859]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 -- 172.21.15.46:0/2608913660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348102a40 msgr2=0x7f0348106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 --2- 172.21.15.46:0/2608913660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348102a40 0x7f0348106a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f0334009920 tx=0x7f0334009c30 comp rx=0 tx=0).stop 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 -- 172.21.15.46:0/2608913660 shutdown_connections 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 --2- 172.21.15.46:0/2608913660 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348107160 0x7f034810ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 --2- 172.21.15.46:0/2608913660 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 0x7f0348102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 --2- 172.21.15.46:0/2608913660 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348102a40 0x7f0348106a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.227+0000 7f034ce00700 1 -- 172.21.15.46:0/2608913660 >> 172.21.15.46:0/2608913660 conn(0x7f0348078f40 msgr2=0x7f0348079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.228+0000 7f034ce00700 1 -- 172.21.15.46:0/2608913660 shutdown_connections 2024-03-08T23:05:43.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.228+0000 7f034ce00700 1 -- 172.21.15.46:0/2608913660 wait complete. 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.229+0000 7f034ce00700 1 Processor -- start 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.229+0000 7f034ce00700 1 -- start start 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 0x7f03481a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 0x7f03481a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348107160 0x7f034819a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348112170 con 0x7f0348102a40 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0348111ff0 con 0x7f0348102090 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f034ce00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03481122f0 con 0x7f0348107160 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f0345400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 0x7f03481a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f0345400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 0x7f03481a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51616/0 (socket says 172.21.15.46:51616) 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f0345400700 1 -- 172.21.15.46:0/808543494 learned_addr learned my addr 172.21.15.46:0/808543494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:43.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.230+0000 7f0345e00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 0x7f03481a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0346800700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348107160 0x7f034819a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 -- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348107160 msgr2=0x7f034819a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348107160 0x7f034819a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 -- 172.21.15.46:0/808543494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 msgr2=0x7f03481a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 0x7f03481a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034819aff0 con 0x7f0348102a40 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0346800700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0348107160 0x7f034819a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345e00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0348102090 0x7f03481a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.231+0000 7f0345400700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 0x7f03481a0b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f033400a010 tx=0x7f033400fb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.232+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334005330 con 0x7f0348102a40 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.232+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034819b280 con 0x7f0348102a40 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.232+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f033400ba40 con 0x7f0348102a40 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.232+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0334003c20 con 0x7f0348102a40 2024-03-08T23:05:43.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.232+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03481a73f0 con 0x7f0348102a40 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.233+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f034810e390 con 0x7f0348102a40 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.235+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f03340054d0 con 0x7f0348102a40 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f033ea00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f032c0a2920 0x7f032c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f03340d02d0 con 0x7f0348102a40 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f0345e00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f032c0a2920 0x7f032c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f033ea00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] conn(0x7f032c0ad5e0 0x7f032c0afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f032c0b00f0 con 0x7f032c0ad5e0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.238+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03340d0720 con 0x7f0348102a40 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.239+0000 7f0345e00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f032c0a2920 0x7f032c0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0338007fd0 tx=0x7f03380073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.239+0000 7f0346800700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] conn(0x7f032c0ad5e0 0x7f032c0afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.239+0000 7f0346800700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] conn(0x7f032c0ad5e0 0x7f032c0afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.240+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== osd.11 v2:172.21.15.192:6824/3216803010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f032c0b00f0 con 0x7f032c0ad5e0 2024-03-08T23:05:43.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.259+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 --> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0348103560 con 0x7f032c0ad5e0 2024-03-08T23:05:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.260+0000 7f033ea00700 1 -- 172.21.15.46:0/808543494 <== osd.11 v2:172.21.15.192:6824/3216803010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0348103560 con 0x7f032c0ad5e0 2024-03-08T23:05:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] conn(0x7f032c0ad5e0 msgr2=0x7f032c0afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] conn(0x7f032c0ad5e0 0x7f032c0afa20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f032c0a2920 msgr2=0x7f032c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f032c0a2920 0x7f032c0a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0338007fd0 tx=0x7f03380073d0 comp rx=0 tx=0).stop 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 msgr2=0x7f03481a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 --2- 172.21.15.46:0/808543494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0348102a40 0x7f03481a0b80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f033400a010 tx=0x7f033400fb40 comp rx=0 tx=0).stop 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f0346800700 1 -- 172.21.15.46:0/808543494 reap_dead start 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 shutdown_connections 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.261+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 >> 172.21.15.46:0/808543494 conn(0x7f0348078f40 msgr2=0x7f03481001e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.262+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 shutdown_connections 2024-03-08T23:05:43.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:38.262+0000 7f034ce00700 1 -- 172.21.15.46:0/808543494 wait complete. 2024-03-08T23:05:44.015 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:43 smithi062 ceph-mon[35761]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:05:44.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:43 smithi192 ceph-mon[36700]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:05:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 -- 172.21.15.46:0/203666817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec102a70 msgr2=0x7f60ec106a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 --2- 172.21.15.46:0/203666817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec102a70 0x7f60ec106a10 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098f0 tx=0x7f60d8009c00 comp rx=0 tx=0).stop 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 -- 172.21.15.46:0/203666817 shutdown_connections 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 --2- 172.21.15.46:0/203666817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 0x7f60ec10ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 --2- 172.21.15.46:0/203666817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec1020c0 0x7f60ec1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 --2- 172.21.15.46:0/203666817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec102a70 0x7f60ec106a10 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.529+0000 7f60f2c00700 1 -- 172.21.15.46:0/203666817 >> 172.21.15.46:0/203666817 conn(0x7f60ec078e70 msgr2=0x7f60ec079280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.530+0000 7f60f2c00700 1 -- 172.21.15.46:0/203666817 shutdown_connections 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.530+0000 7f60f2c00700 1 -- 172.21.15.46:0/203666817 wait complete. 2024-03-08T23:05:44.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.531+0000 7f60f2c00700 1 Processor -- start 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.531+0000 7f60f2c00700 1 -- start start 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec1020c0 0x7f60ec1a05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 0x7f60ec1a0af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 0x7f60ec19a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec1120e0 con 0x7f60ec102a70 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec111f60 con 0x7f60ec1070f0 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f2c00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec112260 con 0x7f60ec1020c0 2024-03-08T23:05:44.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60eb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 0x7f60ec1a0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60ebe00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec1020c0 0x7f60ec1a05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60eb400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 0x7f60ec1a0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46730/0 (socket says 172.21.15.46:46730) 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60eb400700 1 -- 172.21.15.46:0/218388913 learned_addr learned my addr 172.21.15.46:0/218388913 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.532+0000 7f60f0c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 0x7f60ec19a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 -- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec1020c0 msgr2=0x7f60ec1a05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60ec1020c0 0x7f60ec1a05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 -- 172.21.15.46:0/218388913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 msgr2=0x7f60ec19a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 0x7f60ec19a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec19af60 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60f0c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f60ec1070f0 0x7f60ec19a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.533+0000 7f60eb400700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 0x7f60ec1a0af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60d80094f0 tx=0x7f60d800fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.534+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8027070 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.534+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60d8005620 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.534+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec19b1f0 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.534+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8004930 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.534+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec1a7360 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.535+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f60ec0752b0 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.537+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f60d802a070 con 0x7f60ec102a70 2024-03-08T23:05:44.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.538+0000 7f60e8c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60d00a2920 0x7f60d00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f60d80cffc0 con 0x7f60ec102a70 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60ebe00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60d00a2920 0x7f60d00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60e8c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] conn(0x7f60d00ada00 0x7f60d00afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f60d8029e50 con 0x7f60d00ada00 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60d80d0370 con 0x7f60ec102a70 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.539+0000 7f60f0c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] conn(0x7f60d00ada00 0x7f60d00afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.540+0000 7f60ebe00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60d00a2920 0x7f60d00a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60dc005fd0 tx=0x7f60dc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.540+0000 7f60f0c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] conn(0x7f60d00ada00 0x7f60d00afea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.541+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== osd.5 v2:172.21.15.192:6808/2546200396 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f60d8029e50 con 0x7f60d00ada00 2024-03-08T23:05:44.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.560+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 --> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f60ec1034f0 con 0x7f60d00ada00 2024-03-08T23:05:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.560+0000 7f60e8c00700 1 -- 172.21.15.46:0/218388913 <== osd.5 v2:172.21.15.192:6808/2546200396 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f60ec1034f0 con 0x7f60d00ada00 2024-03-08T23:05:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] conn(0x7f60d00ada00 msgr2=0x7f60d00afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] conn(0x7f60d00ada00 0x7f60d00afea0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:44.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60d00a2920 msgr2=0x7f60d00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f60d00a2920 0x7f60d00a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f60dc005fd0 tx=0x7f60dc009310 comp rx=0 tx=0).stop 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 msgr2=0x7f60ec1a0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f2c00700 1 --2- 172.21.15.46:0/218388913 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60ec102a70 0x7f60ec1a0af0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f60d80094f0 tx=0x7f60d800fa00 comp rx=0 tx=0).stop 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.561+0000 7f60f0c00700 1 -- 172.21.15.46:0/218388913 reap_dead start 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.562+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 shutdown_connections 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.562+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 >> 172.21.15.46:0/218388913 conn(0x7f60ec078e70 msgr2=0x7f60ec108270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.562+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 shutdown_connections 2024-03-08T23:05:44.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:43.562+0000 7f60f2c00700 1 -- 172.21.15.46:0/218388913 wait complete. 2024-03-08T23:05:44.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:44 smithi046 ceph-mon[31859]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:45.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 -- 172.21.15.46:0/3661787704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb810e480 msgr2=0x7fedb810e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 --2- 172.21.15.46:0/3661787704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb810e480 0x7fedb810e860 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feda800b0a0 tx=0x7feda800b3b0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 -- 172.21.15.46:0/3661787704 shutdown_connections 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 --2- 172.21.15.46:0/3661787704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedb8108db0 0x7fedb81152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 --2- 172.21.15.46:0/3661787704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb810e480 0x7fedb810e860 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 --2- 172.21.15.46:0/3661787704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb8108870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 -- 172.21.15.46:0/3661787704 >> 172.21.15.46:0/3661787704 conn(0x7fedb80755b0 msgr2=0x7fedb80779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 -- 172.21.15.46:0/3661787704 shutdown_connections 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.150+0000 7fedbe000700 1 -- 172.21.15.46:0/3661787704 wait complete. 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.151+0000 7fedbe000700 1 Processor -- start 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.151+0000 7fedbe000700 1 -- start start 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb81a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb8108db0 0x7fedb81a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedb810e480 0x7fedb819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb8118730 con 0x7fedb81083f0 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb81185b0 con 0x7fedb810e480 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedbe000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb81188b0 con 0x7fedb8108db0 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb81a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb81a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51660/0 (socket says 172.21.15.46:51660) 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 -- 172.21.15.46:0/4120722810 learned_addr learned my addr 172.21.15.46:0/4120722810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7e00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedb810e480 0x7fedb819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb6a00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb8108db0 0x7fedb81a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 -- 172.21.15.46:0/4120722810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb8108db0 msgr2=0x7fedb81a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fedb8108db0 0x7fedb81a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 -- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedb810e480 msgr2=0x7fedb819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.152+0000 7fedb7400700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fedb810e480 0x7fedb819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedb7400700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb819f150 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedb7400700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb81a4830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feda800b070 tx=0x7feda8004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8021070 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feda8003c00 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb819f3e0 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feda8004750 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.153+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb81ab700 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.154+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fedb8062570 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.157+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7feda8003db0 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.158+0000 7fedafe00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fed9c0a2920 0x7fed9c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.158+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7feda800f020 con 0x7fedb81083f0 2024-03-08T23:05:45.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.158+0000 7fedb6a00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fed9c0a2920 0x7fed9c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.159+0000 7fedafe00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] conn(0x7fed9c0ada00 0x7fed9c0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.159+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fed9c0b0570 con 0x7fed9c0ada00 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.159+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7feda80d0950 con 0x7fedb81083f0 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.159+0000 7fedb7e00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] conn(0x7fed9c0ada00 0x7fed9c0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.159+0000 7fedb6a00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fed9c0a2920 0x7fed9c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fedb8077ba0 tx=0x7feda4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.160+0000 7fedb7e00700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] conn(0x7fed9c0ada00 0x7fed9c0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.160+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== osd.1 v2:172.21.15.62:6800/1681427025 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fed9c0b0570 con 0x7fed9c0ada00 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.179+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 --> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fedb80629f0 con 0x7fed9c0ada00 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.180+0000 7fedafe00700 1 -- 172.21.15.46:0/4120722810 <== osd.1 v2:172.21.15.62:6800/1681427025 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fedb80629f0 con 0x7fed9c0ada00 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.180+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] conn(0x7fed9c0ada00 msgr2=0x7fed9c0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.180+0000 7fedbe000700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] conn(0x7fed9c0ada00 0x7fed9c0afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.180+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fed9c0a2920 msgr2=0x7fed9c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fed9c0a2920 0x7fed9c0a4de0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fedb8077ba0 tx=0x7feda4009260 comp rx=0 tx=0).stop 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 msgr2=0x7fedb81a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 --2- 172.21.15.46:0/4120722810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fedb81083f0 0x7fedb81a4830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feda800b070 tx=0x7feda8004c80 comp rx=0 tx=0).stop 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedb7e00700 1 -- 172.21.15.46:0/4120722810 reap_dead start 2024-03-08T23:05:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 shutdown_connections 2024-03-08T23:05:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 >> 172.21.15.46:0/4120722810 conn(0x7fedb80755b0 msgr2=0x7fedb8075cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 shutdown_connections 2024-03-08T23:05:45.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:42.181+0000 7fedbe000700 1 -- 172.21.15.46:0/4120722810 wait complete. 2024-03-08T23:05:45.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:44 smithi192 ceph-mon[36700]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:45.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:44 smithi062 ceph-mon[35761]: pgmap v353: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:45.997 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:45 smithi046 ceph-mon[31859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-08T23:05:45.997 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:45 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:05:46.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:45 smithi192 ceph-mon[36700]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-08T23:05:46.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:45 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:05:46.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:45 smithi062 ceph-mon[35761]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-08T23:05:46.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:45 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:05:46.841 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:46 smithi046 ceph-mon[31859]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:46.900 INFO:teuthology.orchestra.run.smithi046.stdout:219043332144 2024-03-08T23:05:46.900 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-08T23:05:47.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:46 smithi192 ceph-mon[36700]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:47.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:46 smithi062 ceph-mon[35761]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:47.211 INFO:teuthology.orchestra.run.smithi046.stdout:163208757308 2024-03-08T23:05:47.212 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 -- 172.21.15.46:0/1157672973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0074180 msgr2=0x7f54a0074560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 --2- 172.21.15.46:0/1157672973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0074180 0x7f54a0074560 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5490009990 tx=0x7f5490009ca0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 -- 172.21.15.46:0/1157672973 shutdown_connections 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 --2- 172.21.15.46:0/1157672973 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0112dd0 0x7f54a01151c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 --2- 172.21.15.46:0/1157672973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0074180 0x7f54a0074560 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.405+0000 7f54a5c00700 1 --2- 172.21.15.46:0/1157672973 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074aa0 0x7f54a0112890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.406+0000 7f54a5c00700 1 -- 172.21.15.46:0/1157672973 >> 172.21.15.46:0/1157672973 conn(0x7f54a00fd890 msgr2=0x7f54a00ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.406+0000 7f54a5c00700 1 -- 172.21.15.46:0/1157672973 shutdown_connections 2024-03-08T23:05:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.406+0000 7f54a5c00700 1 -- 172.21.15.46:0/1157672973 wait complete. 2024-03-08T23:05:47.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.407+0000 7f54a5c00700 1 Processor -- start 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.407+0000 7f54a5c00700 1 -- start start 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.407+0000 7f54a5c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074180 0x7f54a01a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f54a5c00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 0x7f54a01a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f54a5c00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0112dd0 0x7f54a019e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f54a5c00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0118700 con 0x7f54a0074180 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f54a5c00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0118580 con 0x7f54a0074aa0 2024-03-08T23:05:47.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f54a5c00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a0118880 con 0x7f54a0112dd0 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 0x7f54a01a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 0x7f54a01a4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53248/0 (socket says 172.21.15.46:53248) 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 -- 172.21.15.46:0/2060938135 learned_addr learned my addr 172.21.15.46:0/2060938135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549ea00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074180 0x7f54a01a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 -- 172.21.15.46:0/2060938135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0112dd0 msgr2=0x7f54a019e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54a0112dd0 0x7f54a019e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 -- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074180 msgr2=0x7f54a01a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.408+0000 7f549e000700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074180 0x7f54a01a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f549e000700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a019f100 con 0x7f54a0074aa0 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f549ea00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54a0074180 0x7f54a01a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f549e000700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 0x7f54a01a4d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f548c00eb30 tx=0x7f548c00eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c00cd40 con 0x7f54a0074aa0 2024-03-08T23:05:47.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a019f3f0 con 0x7f54a0074aa0 2024-03-08T23:05:47.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f548c004500 con 0x7f54a0074aa0 2024-03-08T23:05:47.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c010430 con 0x7f54a0074aa0 2024-03-08T23:05:47.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.409+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a01ab6b0 con 0x7f54a0074aa0 2024-03-08T23:05:47.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.410+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f54a0062570 con 0x7f54a0074aa0 2024-03-08T23:05:47.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.413+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f548c004000 con 0x7f54a0074aa0 2024-03-08T23:05:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f5497400700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54840a29a0 0x7f54840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f548c0cf3d0 con 0x7f54a0074aa0 2024-03-08T23:05:47.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f549ea00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54840a29a0 0x7f54840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f5497400700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] conn(0x7f54840ada80 0x7f54840aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f548c023e50 con 0x7f54840ada80 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.414+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f548c0cf780 con 0x7f54a0074aa0 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.415+0000 7f549f400700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] conn(0x7f54840ada80 0x7f54840aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.415+0000 7f549ea00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54840a29a0 0x7f54840a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5490006010 tx=0x7f549001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:47.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.415+0000 7f549f400700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] conn(0x7f54840ada80 0x7f54840aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:47.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.417+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== osd.9 v2:172.21.15.46:6826/1699784429 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f548c023e50 con 0x7f54840ada80 2024-03-08T23:05:47.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.436+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 --> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f54a0062970 con 0x7f54840ada80 2024-03-08T23:05:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f5497400700 1 -- 172.21.15.46:0/2060938135 <== osd.9 v2:172.21.15.46:6826/1699784429 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f54a0062970 con 0x7f54840ada80 2024-03-08T23:05:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] conn(0x7f54840ada80 msgr2=0x7f54840aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] conn(0x7f54840ada80 0x7f54840aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54840a29a0 msgr2=0x7f54840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f54840a29a0 0x7f54840a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5490006010 tx=0x7f549001a040 comp rx=0 tx=0).stop 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 msgr2=0x7f54a01a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f54a5c00700 1 --2- 172.21.15.46:0/2060938135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54a0074aa0 0x7f54a01a4d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f548c00eb30 tx=0x7f548c00eef0 comp rx=0 tx=0).stop 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.437+0000 7f549f400700 1 -- 172.21.15.46:0/2060938135 reap_dead start 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.438+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 shutdown_connections 2024-03-08T23:05:47.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.438+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 >> 172.21.15.46:0/2060938135 conn(0x7f54a00fd890 msgr2=0x7f54a00fe500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.438+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 shutdown_connections 2024-03-08T23:05:47.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:47.438+0000 7f54a5c00700 1 -- 172.21.15.46:0/2060938135 wait complete. 2024-03-08T23:05:47.839 INFO:teuthology.orchestra.run.smithi046.stdout:55834574930 2024-03-08T23:05:47.840 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-08T23:05:48.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:48 smithi046 ceph-mon[31859]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:49.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:48 smithi192 ceph-mon[36700]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:49.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:48 smithi062 ceph-mon[35761]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:49.294 INFO:teuthology.orchestra.run.smithi046.stdout:197568495671 2024-03-08T23:05:49.295 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-08T23:05:49.878 INFO:teuthology.orchestra.run.smithi046.stdout:343597383706 2024-03-08T23:05:49.878 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-08T23:05:50.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 -- 172.21.15.46:0/651873159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f370806a430 msgr2=0x7f370810ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 --2- 172.21.15.46:0/651873159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f370806a430 0x7f370810ec70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f370000b0d0 tx=0x7f370000b3e0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 -- 172.21.15.46:0/651873159 shutdown_connections 2024-03-08T23:05:50.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 --2- 172.21.15.46:0/651873159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f370806a430 0x7f370810ec70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 --2- 172.21.15.46:0/651873159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3708107e30 0x7f3708108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 --2- 172.21.15.46:0/651873159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3708069a90 0x7f3708069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 -- 172.21.15.46:0/651873159 >> 172.21.15.46:0/651873159 conn(0x7f3708077c10 msgr2=0x7f3708078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 -- 172.21.15.46:0/651873159 shutdown_connections 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.523+0000 7f370ca00700 1 -- 172.21.15.46:0/651873159 wait complete. 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.524+0000 7f370ca00700 1 Processor -- start 2024-03-08T23:05:50.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.524+0000 7f370ca00700 1 -- start start 2024-03-08T23:05:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3708069a90 0x7f370819a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f370806a430 0x7f370819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3708107e30 0x7f370819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37081120b0 con 0x7f3708107e30 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3708111f30 con 0x7f370806a430 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f370ca00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3708112230 con 0x7f3708069a90 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f3706800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3708107e30 0x7f370819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f3706800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3708107e30 0x7f370819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46778/0 (socket says 172.21.15.46:46778) 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f3706800700 1 -- 172.21.15.46:0/2394945784 learned_addr learned my addr 172.21.15.46:0/2394945784 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f36fd400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f370806a430 0x7f370819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.525+0000 7f3705e00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3708069a90 0x7f370819a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 -- 172.21.15.46:0/2394945784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3708069a90 msgr2=0x7f370819a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3708069a90 0x7f370819a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 -- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3708107e30 msgr2=0x7f370819efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3708107e30 0x7f370819efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370819f6f0 con 0x7f370806a430 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.526+0000 7f36fd400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f370806a430 0x7f370819ac70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e9f0 tx=0x7f36f000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.527+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0004220 con 0x7f370806a430 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.527+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370819f9e0 con 0x7f370806a430 2024-03-08T23:05:50.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.527+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36f0004d10 con 0x7f370806a430 2024-03-08T23:05:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.527+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36f0005020 con 0x7f370806a430 2024-03-08T23:05:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.527+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37081a7260 con 0x7f370806a430 2024-03-08T23:05:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.528+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3708100180 con 0x7f370806a430 2024-03-08T23:05:50.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.530+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f36f0004750 con 0x7f370806a430 2024-03-08T23:05:50.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f36ff400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f36ec0a2920 0x7f36ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f36f00cf2e0 con 0x7f370806a430 2024-03-08T23:05:50.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f3705e00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f36ec0a2920 0x7f36ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f36ff400700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] conn(0x7f36ec0ada00 0x7f36ec0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36ec0b0570 con 0x7f36ec0ada00 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.532+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36f00cf690 con 0x7f370806a430 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.533+0000 7f3706800700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] conn(0x7f36ec0ada00 0x7f36ec0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.533+0000 7f3705e00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f36ec0a2920 0x7f36ec0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009be0 tx=0x7f36f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.533+0000 7f3706800700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] conn(0x7f36ec0ada00 0x7f36ec0afea0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.534+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== osd.3 v2:172.21.15.46:6810/2986867677 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f36ec0b0570 con 0x7f36ec0ada00 2024-03-08T23:05:50.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.553+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 --> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f37081034e0 con 0x7f36ec0ada00 2024-03-08T23:05:50.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.553+0000 7f36ff400700 1 -- 172.21.15.46:0/2394945784 <== osd.3 v2:172.21.15.46:6810/2986867677 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f37081034e0 con 0x7f36ec0ada00 2024-03-08T23:05:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] conn(0x7f36ec0ada00 msgr2=0x7f36ec0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] conn(0x7f36ec0ada00 0x7f36ec0afea0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f36ec0a2920 msgr2=0x7f36ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f36ec0a2920 0x7f36ec0a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f36f8009be0 tx=0x7f36f8009260 comp rx=0 tx=0).stop 2024-03-08T23:05:50.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f370806a430 msgr2=0x7f370819ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 --2- 172.21.15.46:0/2394945784 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f370806a430 0x7f370819ac70 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e9f0 tx=0x7f36f000ed00 comp rx=0 tx=0).stop 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f3706800700 1 -- 172.21.15.46:0/2394945784 reap_dead start 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 shutdown_connections 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 >> 172.21.15.46:0/2394945784 conn(0x7f3708077c10 msgr2=0x7f37080ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 shutdown_connections 2024-03-08T23:05:50.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.554+0000 7f370ca00700 1 -- 172.21.15.46:0/2394945784 wait complete. 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 -- 172.21.15.46:0/2765859187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0101920 msgr2=0x7f22d01058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 --2- 172.21.15.46:0/2765859187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0101920 0x7f22d01058e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f22bc0099c0 tx=0x7f22bc009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 -- 172.21.15.46:0/2765859187 shutdown_connections 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 --2- 172.21.15.46:0/2765859187 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22d0105fc0 0x7f22d010db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 --2- 172.21.15.46:0/2765859187 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d0100f70 0x7f22d0101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 --2- 172.21.15.46:0/2765859187 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0101920 0x7f22d01058e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 -- 172.21.15.46:0/2765859187 >> 172.21.15.46:0/2765859187 conn(0x7f22d0076d30 msgr2=0x7f22d0077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 -- 172.21.15.46:0/2765859187 shutdown_connections 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.511+0000 7f22d5200700 1 -- 172.21.15.46:0/2765859187 wait complete. 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 Processor -- start 2024-03-08T23:05:50.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 -- start start 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 0x7f22d01a0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d0101920 0x7f22d01a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22d0105fc0 0x7f22d019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0110ff0 con 0x7f22d0100f70 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0110e70 con 0x7f22d0105fc0 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22d5200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22d0111170 con 0x7f22d0101920 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22cde00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 0x7f22d01a0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.513+0000 7f22cde00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 0x7f22d01a0440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46746/0 (socket says 172.21.15.46:46746) 2024-03-08T23:05:50.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 -- 172.21.15.46:0/3924674792 learned_addr learned my addr 172.21.15.46:0/3924674792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22ce800700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22d0105fc0 0x7f22d019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cd400700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d0101920 0x7f22d01a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 -- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d0101920 msgr2=0x7f22d01a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22d0101920 0x7f22d01a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 -- 172.21.15.46:0/3924674792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22d0105fc0 msgr2=0x7f22d019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f22d0105fc0 0x7f22d019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d019ad60 con 0x7f22d0100f70 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.514+0000 7f22cde00700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 0x7f22d01a0440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22c000e9c0 tx=0x7f22c000ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.515+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000cbd0 con 0x7f22d0100f70 2024-03-08T23:05:50.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.515+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22c000cd70 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.515+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22d019b050 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.515+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0010430 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.515+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22d01a7310 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.517+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f22d00ffee0 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.519+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f22c0004750 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.521+0000 7f22c6a00700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f22b40a2920 0x7f22b40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.521+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f22c00cf210 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.521+0000 7f22cd400700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f22b40a2920 0x7f22b40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.522+0000 7f22c6a00700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] conn(0x7f22b40ada00 0x7f22b40afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.522+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f22b40b0570 con 0x7f22b40ada00 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.522+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22c00d6050 con 0x7f22d0100f70 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.522+0000 7f22cd400700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f22b40a2920 0x7f22b40a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22bc000c00 tx=0x7f22bc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.522+0000 7f22ce800700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] conn(0x7f22b40ada00 0x7f22b40afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.523+0000 7f22ce800700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] conn(0x7f22b40ada00 0x7f22b40afea0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.524+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== osd.8 v2:172.21.15.192:6816/3623660745 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f22b40b0570 con 0x7f22b40ada00 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.542+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 --> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f22d01023c0 con 0x7f22b40ada00 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.543+0000 7f22c6a00700 1 -- 172.21.15.46:0/3924674792 <== osd.8 v2:172.21.15.192:6816/3623660745 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f22d01023c0 con 0x7f22b40ada00 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.543+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] conn(0x7f22b40ada00 msgr2=0x7f22b40afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.543+0000 7f22d5200700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] conn(0x7f22b40ada00 0x7f22b40afea0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f22b40a2920 msgr2=0x7f22b40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f22b40a2920 0x7f22b40a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22bc000c00 tx=0x7f22bc01a040 comp rx=0 tx=0).stop 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 msgr2=0x7f22d01a0440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 --2- 172.21.15.46:0/3924674792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f22d0100f70 0x7f22d01a0440 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f22c000e9c0 tx=0x7f22c000ed80 comp rx=0 tx=0).stop 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22ce800700 1 -- 172.21.15.46:0/3924674792 reap_dead start 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 shutdown_connections 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 >> 172.21.15.46:0/3924674792 conn(0x7f22d0076d30 msgr2=0x7f22d006a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 shutdown_connections 2024-03-08T23:05:50.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:49.544+0000 7f22d5200700 1 -- 172.21.15.46:0/3924674792 wait complete. 2024-03-08T23:05:50.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:50 smithi046 ceph-mon[31859]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:50 smithi192 ceph-mon[36700]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:50 smithi062 ceph-mon[35761]: pgmap v356: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 -- 172.21.15.46:0/3732214945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc0101e30 msgr2=0x7f4fc01022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3732214945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc0101e30 0x7f4fc01022b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4fac0099c0 tx=0x7f4fac009cd0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 -- 172.21.15.46:0/3732214945 shutdown_connections 2024-03-08T23:05:51.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3732214945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc01027f0 0x7f4fc010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3732214945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0107e10 0x7f4fc01081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3732214945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc0101e30 0x7f4fc01022b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 -- 172.21.15.46:0/3732214945 >> 172.21.15.46:0/3732214945 conn(0x7f4fc0078f10 msgr2=0x7f4fc0079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 -- 172.21.15.46:0/3732214945 shutdown_connections 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.433+0000 7f4fc7800700 1 -- 172.21.15.46:0/3732214945 wait complete. 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.435+0000 7f4fc7800700 1 Processor -- start 2024-03-08T23:05:51.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.435+0000 7f4fc7800700 1 -- start start 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 0x7f4fc00740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc01027f0 0x7f4fc0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0107e10 0x7f4fc0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc01120e0 con 0x7f4fc01027f0 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0111f60 con 0x7f4fc0107e10 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc7800700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fc0112260 con 0x7f4fc0101e30 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 0x7f4fc00740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:51.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fbfe00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc01027f0 0x7f4fc0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 0x7f4fc00740e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36904/0 (socket says 172.21.15.46:36904) 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 -- 172.21.15.46:0/3615854336 learned_addr learned my addr 172.21.15.46:0/3615854336 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc5800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0107e10 0x7f4fc0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 -- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0107e10 msgr2=0x7f4fc0074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4fc0107e10 0x7f4fc0074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 -- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc01027f0 msgr2=0x7f4fc0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc01027f0 0x7f4fc0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fc4e00700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc01a6a70 con 0x7f4fc0101e30 2024-03-08T23:05:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.436+0000 7f4fbfe00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4fc01027f0 0x7f4fc0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:05:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fc4e00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 0x7f4fc00740e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e9a0 tx=0x7f4fb400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:51.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb400ca10 con 0x7f4fc0101e30 2024-03-08T23:05:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4fb400cbb0 con 0x7f4fc0101e30 2024-03-08T23:05:51.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fc01a6d60 con 0x7f4fc0101e30 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb4018680 con 0x7f4fc0101e30 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.437+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fc01a7210 con 0x7f4fc0101e30 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.438+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4fc00624f0 con 0x7f4fc0101e30 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.441+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4fb4047a90 con 0x7f4fc0101e30 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbd600700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fa80a2920 0x7f4fa80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:51.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4fb40cf260 con 0x7f4fc0101e30 2024-03-08T23:05:51.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbfe00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fa80a2920 0x7f4fa80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:51.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbd600700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] conn(0x7f4fa80ada00 0x7f4fa80afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:51.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fa80b0570 con 0x7f4fa80ada00 2024-03-08T23:05:51.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.442+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4fb40cf610 con 0x7f4fc0101e30 2024-03-08T23:05:51.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.443+0000 7f4fc5800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] conn(0x7f4fa80ada00 0x7f4fa80afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:51.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.443+0000 7f4fbfe00700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fa80a2920 0x7f4fa80a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0075b20 tx=0x7f4fac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:51.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.443+0000 7f4fc5800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] conn(0x7f4fa80ada00 0x7f4fa80afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:51.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.444+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== osd.10 v2:172.21.15.62:6824/3230310440 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4fa80b0570 con 0x7f4fa80ada00 2024-03-08T23:05:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.463+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 --> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4fc0062970 con 0x7f4fa80ada00 2024-03-08T23:05:51.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.463+0000 7f4fbd600700 1 -- 172.21.15.46:0/3615854336 <== osd.10 v2:172.21.15.62:6824/3230310440 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4fc0062970 con 0x7f4fa80ada00 2024-03-08T23:05:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] conn(0x7f4fa80ada00 msgr2=0x7f4fa80afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] conn(0x7f4fa80ada00 0x7f4fa80afea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fa80a2920 msgr2=0x7f4fa80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fa80a2920 0x7f4fa80a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4fc0075b20 tx=0x7f4fac005fb0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 msgr2=0x7f4fc00740e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 --2- 172.21.15.46:0/3615854336 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4fc0101e30 0x7f4fc00740e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4fb400e9a0 tx=0x7f4fb400ecb0 comp rx=0 tx=0).stop 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc5800700 1 -- 172.21.15.46:0/3615854336 reap_dead start 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 shutdown_connections 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 >> 172.21.15.46:0/3615854336 conn(0x7f4fc0078f10 msgr2=0x7f4fc0100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:51.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.464+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 shutdown_connections 2024-03-08T23:05:51.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:50.465+0000 7f4fc7800700 1 -- 172.21.15.46:0/3615854336 wait complete. 2024-03-08T23:05:52.251 INFO:teuthology.orchestra.run.smithi046.stdout:253403070510 2024-03-08T23:05:52.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-08T23:05:52.898 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:52 smithi046 ceph-mon[31859]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:53.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:52 smithi192 ceph-mon[36700]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:53.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:52 smithi062 ceph-mon[35761]: pgmap v357: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 -- 172.21.15.46:0/2427256011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 msgr2=0x7f500410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 --2- 172.21.15.46:0/2427256011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500410dfc0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4ff4009990 tx=0x7f4ff4009ca0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 -- 172.21.15.46:0/2427256011 shutdown_connections 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 --2- 172.21.15.46:0/2427256011 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f500410e500 0x7f5004110970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 --2- 172.21.15.46:0/2427256011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5004074210 0x7f50040745f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 --2- 172.21.15.46:0/2427256011 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500410dfc0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 -- 172.21.15.46:0/2427256011 >> 172.21.15.46:0/2427256011 conn(0x7f50040fd7b0 msgr2=0x7f50040ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 -- 172.21.15.46:0/2427256011 shutdown_connections 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.582+0000 7f500b200700 1 -- 172.21.15.46:0/2427256011 wait complete. 2024-03-08T23:05:53.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 Processor -- start 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 -- start start 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5004074210 0x7f500419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f500410e500 0x7f500419eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50041141f0 con 0x7f5004074b30 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004114070 con 0x7f5004074210 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.583+0000 7f500b200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5004114370 con 0x7f500410e500 2024-03-08T23:05:53.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:46798/0 (socket says 172.21.15.46:46798) 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 -- 172.21.15.46:0/1948640163 learned_addr learned my addr 172.21.15.46:0/1948640163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f500a800700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f500410e500 0x7f500419eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009e00700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5004074210 0x7f500419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 -- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f500410e500 msgr2=0x7f500419eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f500410e500 0x7f500419eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 -- 172.21.15.46:0/1948640163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5004074210 msgr2=0x7f500419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5004074210 0x7f500419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.584+0000 7f5009400700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f500419f620 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f5009400700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500419aba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400bae0 tx=0x7f4ff400bb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4026070 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ff4003d70 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f500419f8b0 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff4021410 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.585+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50041a7190 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.586+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5004062570 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.589+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f4ff4029070 con 0x7f5004074b30 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.590+0000 7f4ffea00700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fec0a2920 0x7f4fec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:53.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f4ff40d05e0 con 0x7f5004074b30 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f5009e00700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fec0a2920 0x7f4fec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f4ffea00700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] conn(0x7f4fec0ada00 0x7f4fec0afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4fec0b0570 con 0x7f4fec0ada00 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ff40d0a30 con 0x7f5004074b30 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f500a800700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] conn(0x7f4fec0ada00 0x7f4fec0afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.591+0000 7f5009e00700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fec0a2920 0x7f4fec0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009c60 tx=0x7f4ff8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.592+0000 7f500a800700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] conn(0x7f4fec0ada00 0x7f4fec0afea0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.593+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== osd.2 v2:172.21.15.192:6800/2982587479 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f4fec0b0570 con 0x7f4fec0ada00 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.611+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 --> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50040629f0 con 0x7f4fec0ada00 2024-03-08T23:05:53.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.612+0000 7f4ffea00700 1 -- 172.21.15.46:0/1948640163 <== osd.2 v2:172.21.15.192:6800/2982587479 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f50040629f0 con 0x7f4fec0ada00 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] conn(0x7f4fec0ada00 msgr2=0x7f4fec0afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] conn(0x7f4fec0ada00 0x7f4fec0afea0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fec0a2920 msgr2=0x7f4fec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f4fec0a2920 0x7f4fec0a4de0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4ff8009c60 tx=0x7f4ff8009260 comp rx=0 tx=0).stop 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 msgr2=0x7f500419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 --2- 172.21.15.46:0/1948640163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5004074b30 0x7f500419aba0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f4ff400bae0 tx=0x7f4ff400bb10 comp rx=0 tx=0).stop 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500a800700 1 -- 172.21.15.46:0/1948640163 reap_dead start 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 shutdown_connections 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 >> 172.21.15.46:0/1948640163 conn(0x7f50040fd7b0 msgr2=0x7f50040fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 shutdown_connections 2024-03-08T23:05:53.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:51.613+0000 7f500b200700 1 -- 172.21.15.46:0/1948640163 wait complete. 2024-03-08T23:05:53.880 INFO:teuthology.orchestra.run.smithi046.stdout:77309411408 2024-03-08T23:05:53.881 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-08T23:05:54.143 INFO:teuthology.orchestra.run.smithi046.stdout:287762808870 2024-03-08T23:05:54.144 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-08T23:05:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 -- 172.21.15.46:0/2306922134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c075e10 msgr2=0x7f5e5c074010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 --2- 172.21.15.46:0/2306922134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c075e10 0x7f5e5c074010 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009990 tx=0x7f5e50009ca0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 -- 172.21.15.46:0/2306922134 shutdown_connections 2024-03-08T23:05:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 --2- 172.21.15.46:0/2306922134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e5c112a70 0x7f5e5c114e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 --2- 172.21.15.46:0/2306922134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c075e10 0x7f5e5c074010 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 --2- 172.21.15.46:0/2306922134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c0749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 -- 172.21.15.46:0/2306922134 >> 172.21.15.46:0/2306922134 conn(0x7f5e5c078f10 msgr2=0x7f5e5c079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.893+0000 7f5e63800700 1 -- 172.21.15.46:0/2306922134 shutdown_connections 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.894+0000 7f5e63800700 1 -- 172.21.15.46:0/2306922134 wait complete. 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.895+0000 7f5e63800700 1 Processor -- start 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.895+0000 7f5e63800700 1 -- start start 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c1a4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e5c075e10 0x7f5e5c1a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c112a70 0x7f5e5c19e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c1186d0 con 0x7f5e5c074550 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c118550 con 0x7f5e5c112a70 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e63800700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c118850 con 0x7f5e5c075e10 2024-03-08T23:05:54.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c1a4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c1a4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50920/0 (socket says 172.21.15.46:50920) 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 -- 172.21.15.46:0/3927131214 learned_addr learned my addr 172.21.15.46:0/3927131214 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e5be00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e5c075e10 0x7f5e5c1a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e61800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c112a70 0x7f5e5c19e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 -- 172.21.15.46:0/3927131214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e5c075e10 msgr2=0x7f5e5c1a4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5e5c075e10 0x7f5e5c1a4d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 -- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c112a70 msgr2=0x7f5e5c19e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.896+0000 7f5e60e00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5e5c112a70 0x7f5e5c19e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e60e00700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5c19f150 con 0x7f5e5c074550 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e60e00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c1a4830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5e50000c00 tx=0x7f5e5000bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50027070 con 0x7f5e5c074550 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c19f3e0 con 0x7f5e5c074550 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e50003d70 con 0x7f5e5c074550 2024-03-08T23:05:54.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e5000f460 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.897+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c1ab700 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.898+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5e5c112210 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.901+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5e5002a070 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.902+0000 7f5e59600700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5e440a2920 0x7f5e440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.902+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f5e500d0320 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.902+0000 7f5e5be00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5e440a2920 0x7f5e440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.902+0000 7f5e59600700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] conn(0x7f5e440ada00 0x7f5e440afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.903+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5e500056d0 con 0x7f5e440ada00 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.903+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5e500d06d0 con 0x7f5e5c074550 2024-03-08T23:05:54.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.903+0000 7f5e61800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] conn(0x7f5e440ada00 0x7f5e440afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:05:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.903+0000 7f5e5be00700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5e440a2920 0x7f5e440a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009680 tx=0x7f5e48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.904+0000 7f5e61800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] conn(0x7f5e440ada00 0x7f5e440afea0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:05:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.905+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== osd.4 v2:172.21.15.62:6808/170069406 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5e500056d0 con 0x7f5e440ada00 2024-03-08T23:05:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.923+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 --> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5e5c0756c0 con 0x7f5e440ada00 2024-03-08T23:05:54.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e59600700 1 -- 172.21.15.46:0/3927131214 <== osd.4 v2:172.21.15.62:6808/170069406 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5e5c0756c0 con 0x7f5e440ada00 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] conn(0x7f5e440ada00 msgr2=0x7f5e440afea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] conn(0x7f5e440ada00 0x7f5e440afea0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5e440a2920 msgr2=0x7f5e440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5e440a2920 0x7f5e440a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5e48009680 tx=0x7f5e48009260 comp rx=0 tx=0).stop 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 msgr2=0x7f5e5c1a4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.924+0000 7f5e63800700 1 --2- 172.21.15.46:0/3927131214 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5e5c074550 0x7f5e5c1a4830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5e50000c00 tx=0x7f5e5000bab0 comp rx=0 tx=0).stop 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.925+0000 7f5e61800700 1 -- 172.21.15.46:0/3927131214 reap_dead start 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.925+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 shutdown_connections 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.925+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 >> 172.21.15.46:0/3927131214 conn(0x7f5e5c078f10 msgr2=0x7f5e5c111830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.925+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 shutdown_connections 2024-03-08T23:05:54.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:05:52.925+0000 7f5e63800700 1 -- 172.21.15.46:0/3927131214 wait complete. 2024-03-08T23:05:54.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:54 smithi046 ceph-mon[31859]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:55.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:54 smithi192 ceph-mon[36700]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:55.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:54 smithi062 ceph-mon[35761]: pgmap v358: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:05:55.392 INFO:teuthology.orchestra.run.smithi046.stdout:141733920837 2024-03-08T23:05:55.392 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-08T23:05:55.435 INFO:teuthology.orchestra.run.smithi046.stdout:107374182472 2024-03-08T23:05:55.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-08T23:05:55.973 INFO:teuthology.orchestra.run.smithi046.stdout:309237645345 2024-03-08T23:05:55.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-08T23:05:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:55.985 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:56.001 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:56.009 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:56.051 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:56.095 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:57.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:56 smithi192 ceph-mon[36700]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:05:57.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:56 smithi062 ceph-mon[35761]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:05:57.161 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:56 smithi046 ceph-mon[31859]: pgmap v359: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:05:57.889 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:57.895 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:05:58.909 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:05:58 smithi046 ceph-mon[31859]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:05:59.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:05:58 smithi192 ceph-mon[36700]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:05:59.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:05:58 smithi062 ceph-mon[35761]: pgmap v360: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:06:01.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:00 smithi192 ceph-mon[36700]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-08T23:06:01.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:00 smithi062 ceph-mon[35761]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-08T23:06:01.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:00 smithi046 ceph-mon[31859]: pgmap v361: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-08T23:06:03.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:02 smithi192 ceph-mon[36700]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:03.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:02 smithi062 ceph-mon[35761]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:03.155 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:02 smithi046 ceph-mon[31859]: pgmap v362: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:03.165 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:03 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1917871703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-08T23:06:04.209 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:03 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1917871703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-08T23:06:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:03 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1917871703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-08T23:06:04.412 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.884+0000 7fdd4ac00700 1 -- 172.21.15.46:0/2480724544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102a40 msgr2=0x7fdd44106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.884+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/2480724544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102a40 0x7fdd44106a80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c0098f0 tx=0x7fdd2c009c00 comp rx=0 tx=0).stop 2024-03-08T23:06:04.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 -- 172.21.15.46:0/2480724544 shutdown_connections 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/2480724544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44107160 0x7fdd4410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/2480724544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44102090 0x7fdd44102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/2480724544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102a40 0x7fdd44106a80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 -- 172.21.15.46:0/2480724544 >> 172.21.15.46:0/2480724544 conn(0x7fdd44078f40 msgr2=0x7fdd44079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 -- 172.21.15.46:0/2480724544 shutdown_connections 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.885+0000 7fdd4ac00700 1 -- 172.21.15.46:0/2480724544 wait complete. 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 Processor -- start 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 -- start start 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44102a40 0x7fdd441a0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 0x7fdd4419a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44112120 con 0x7fdd44107160 2024-03-08T23:06:04.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44111fa0 con 0x7fdd44102090 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.886+0000 7fdd4ac00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd441122a0 con 0x7fdd44102a40 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd48c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 0x7fdd4419a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43608/0 (socket says 172.21.15.46:43608) 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd3b400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44102a40 0x7fdd441a0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 -- 172.21.15.46:0/26128543 learned_addr learned my addr 172.21.15.46:0/26128543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 -- 172.21.15.46:0/26128543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44102a40 msgr2=0x7fdd441a0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44102a40 0x7fdd441a0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 -- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 msgr2=0x7fdd4419a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 0x7fdd4419a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 -- 172.21.15.46:0/26128543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4419afa0 con 0x7fdd44102090 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd48c00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 0x7fdd4419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:04.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.887+0000 7fdd43e00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ec10 tx=0x7fdd3400ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.888+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3400cc90 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.888+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd3400ce30 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.888+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4419b290 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.888+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd34010690 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.888+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd441a7570 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.889+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd44103560 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.891+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdd34003680 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.892+0000 7fdd41600700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 0x7fdd280a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.892+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fdd340cf2b0 con 0x7fdd44102090 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.892+0000 7fdd3b400700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 0x7fdd280a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.893+0000 7fdd3b400700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 0x7fdd280a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00c010 tx=0x7fdd2c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:04.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:04.897+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdd3409be50 con 0x7fdd44102090 2024-03-08T23:06:05.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.126+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdd441a7740 con 0x7fdd44102090 2024-03-08T23:06:05.131 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:04 smithi046 ceph-mon[31859]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:06:05.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.127+0000 7fdd41600700 1 -- 172.21.15.46:0/26128543 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdd3409b750 con 0x7fdd44102090 2024-03-08T23:06:05.132 INFO:teuthology.orchestra.run.smithi046.stdout:219043332148 2024-03-08T23:06:05.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 msgr2=0x7fdd280a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:05.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 0x7fdd280a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00c010 tx=0x7fdd2c01a040 comp rx=0 tx=0).stop 2024-03-08T23:06:05.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 msgr2=0x7fdd441a05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:05.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400ec10 tx=0x7fdd3400ef20 comp rx=0 tx=0).stop 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 shutdown_connections 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdd280a29a0 0x7fdd280a4e60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdd44107160 0x7fdd4419a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdd44102090 0x7fdd441a05f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 --2- 172.21.15.46:0/26128543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdd44102a40 0x7fdd441a0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.130+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 >> 172.21.15.46:0/26128543 conn(0x7fdd44078f40 msgr2=0x7fdd440ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.131+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 shutdown_connections 2024-03-08T23:06:05.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:05.131+0000 7fdd4ac00700 1 -- 172.21.15.46:0/26128543 wait complete. 2024-03-08T23:06:05.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:04 smithi062 ceph-mon[35761]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:06:05.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:04 smithi192 ceph-mon[36700]: pgmap v363: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:06:06.087 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:05 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/26128543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-08T23:06:06.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:05 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/26128543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-08T23:06:06.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:05 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/26128543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-08T23:06:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 -- 172.21.15.46:0/773114159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac010e7e0 msgr2=0x7feac0110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:07.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 --2- 172.21.15.46:0/773114159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac010e7e0 0x7feac0110bd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7feab800b0a0 tx=0x7feab800b3b0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 -- 172.21.15.46:0/773114159 shutdown_connections 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 --2- 172.21.15.46:0/773114159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac010e7e0 0x7feac0110bd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 --2- 172.21.15.46:0/773114159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac0074010 0x7feac00743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 --2- 172.21.15.46:0/773114159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074930 0x7feac010e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.875+0000 7feac7a00700 1 -- 172.21.15.46:0/773114159 >> 172.21.15.46:0/773114159 conn(0x7feac0078f10 msgr2=0x7feac0079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.876+0000 7feac7a00700 1 -- 172.21.15.46:0/773114159 shutdown_connections 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.876+0000 7feac7a00700 1 -- 172.21.15.46:0/773114159 wait complete. 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 Processor -- start 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 -- start start 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074010 0x7feac019a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac0074930 0x7feac019ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac01142c0 con 0x7feac010e7e0 2024-03-08T23:06:07.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0114140 con 0x7feac0074930 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.877+0000 7feac7a00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac0114440 con 0x7feac0074010 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59500/0 (socket says 172.21.15.46:59500) 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 -- 172.21.15.46:0/1917871703 learned_addr learned my addr 172.21.15.46:0/1917871703 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5000700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074010 0x7feac019a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feabfe00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac0074930 0x7feac019ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 -- 172.21.15.46:0/1917871703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074010 msgr2=0x7feac019a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074010 0x7feac019a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 -- 172.21.15.46:0/1917871703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac0074930 msgr2=0x7feac019ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac0074930 0x7feac019ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 -- 172.21.15.46:0/1917871703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac019f720 con 0x7feac010e7e0 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.878+0000 7feac5a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feab800b070 tx=0x7feab8004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:07.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.879+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8021070 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.879+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac019f9b0 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.879+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feab8003c00 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.879+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab80047a0 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.879+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac01a7290 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.880+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac00756b0 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.882+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7feab8007980 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.883+0000 7feabd600700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 0x7feaa80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.884+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7feab80cfe70 con 0x7feac010e7e0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.884+0000 7feac5000700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 0x7feaa80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.884+0000 7feac5000700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 0x7feaa80a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feab4009c60 tx=0x7feab4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:07.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:02.888+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feab809caa0 con 0x7feac010e7e0 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stdout:163208757312 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.147+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7feac002d830 con 0x7feac010e7e0 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.148+0000 7feabd600700 1 -- 172.21.15.46:0/1917871703 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7feab809c3a0 con 0x7feac010e7e0 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.152+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 msgr2=0x7feaa80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.152+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 0x7feaa80a4de0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7feab4009c60 tx=0x7feab4009260 comp rx=0 tx=0).stop 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.152+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 msgr2=0x7feac019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:07.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.152+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7feab800b070 tx=0x7feab8004cf0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 shutdown_connections 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7feaa80a2920 0x7feaa80a4de0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feac010e7e0 0x7feac019efe0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feac0074010 0x7feac019a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 --2- 172.21.15.46:0/1917871703 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feac0074930 0x7feac019ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 >> 172.21.15.46:0/1917871703 conn(0x7feac0078f10 msgr2=0x7feac00ff050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 shutdown_connections 2024-03-08T23:06:07.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:03.154+0000 7feac7a00700 1 -- 172.21.15.46:0/1917871703 wait complete. 2024-03-08T23:06:07.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:06 smithi062 ceph-mon[35761]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-08T23:06:07.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:06 smithi046 ceph-mon[31859]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-08T23:06:07.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:06 smithi192 ceph-mon[36700]: pgmap v364: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-08T23:06:08.754 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757308 got 163208757312 for osd.5 2024-03-08T23:06:08.754 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:08.970 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:08 smithi046 ceph-mon[31859]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:09.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:08 smithi062 ceph-mon[35761]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:09.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:08 smithi192 ceph-mon[36700]: pgmap v365: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 -- 172.21.15.46:0/2944117457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924074930 msgr2=0x7fc924112a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 --2- 172.21.15.46:0/2944117457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924074930 0x7fc924112a70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc9100099c0 tx=0x7fc910009cd0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 -- 172.21.15.46:0/2944117457 shutdown_connections 2024-03-08T23:06:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 --2- 172.21.15.46:0/2944117457 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924112fb0 0x7fc9241153a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 --2- 172.21.15.46:0/2944117457 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074010 0x7fc9240743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 --2- 172.21.15.46:0/2944117457 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924074930 0x7fc924112a70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 -- 172.21.15.46:0/2944117457 >> 172.21.15.46:0/2944117457 conn(0x7fc924078f40 msgr2=0x7fc924079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.880+0000 7fc929200700 1 -- 172.21.15.46:0/2944117457 shutdown_connections 2024-03-08T23:06:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.881+0000 7fc929200700 1 -- 172.21.15.46:0/2944117457 wait complete. 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 Processor -- start 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 -- start start 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074930 0x7fc9241a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924112fb0 0x7fc92419ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9241187e0 con 0x7fc924112fb0 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc924118660 con 0x7fc924074930 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.882+0000 7fc929200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc924118960 con 0x7fc924074010 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc922800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924112fb0 0x7fc92419ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074930 0x7fc9241a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:54288/0 (socket says 172.21.15.46:54288) 2024-03-08T23:06:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 -- 172.21.15.46:0/2979138845 learned_addr learned my addr 172.21.15.46:0/2979138845 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:10.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 -- 172.21.15.46:0/2979138845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074930 msgr2=0x7fc9241a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074930 0x7fc9241a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 -- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924112fb0 msgr2=0x7fc92419ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924112fb0 0x7fc92419ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.883+0000 7fc921e00700 1 -- 172.21.15.46:0/2979138845 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92419f180 con 0x7fc924074010 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc921e00700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc91400eb30 tx=0x7fc91400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc91400cbd0 con 0x7fc924074010 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc914004500 con 0x7fc924074010 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92419f470 con 0x7fc924074010 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc914010430 con 0x7fc924074010 2024-03-08T23:06:10.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.884+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9241ab730 con 0x7fc924074010 2024-03-08T23:06:10.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.886+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc924068aa0 con 0x7fc924074010 2024-03-08T23:06:10.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.887+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc914003f00 con 0x7fc924074010 2024-03-08T23:06:10.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.889+0000 7fc91aa00700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 0x7fc9080a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:10.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.889+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc9140cf0f0 con 0x7fc924074010 2024-03-08T23:06:10.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.889+0000 7fc921400700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 0x7fc9080a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:10.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.890+0000 7fc921400700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 0x7fc9080a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc910000c00 tx=0x7fc910005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:10.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.893+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc91409be30 con 0x7fc924074010 2024-03-08T23:06:11.036 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:10 smithi046 ceph-mon[31859]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:11.036 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:10 smithi046 ceph-mon[31859]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-08T23:06:11.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.135+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc9241ab960 con 0x7fc924074010 2024-03-08T23:06:11.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.136+0000 7fc91aa00700 1 -- 172.21.15.46:0/2979138845 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc91409b730 con 0x7fc924074010 2024-03-08T23:06:11.140 INFO:teuthology.orchestra.run.smithi046.stdout:107374182476 2024-03-08T23:06:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:10 smithi062 ceph-mon[35761]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:10 smithi062 ceph-mon[35761]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-08T23:06:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.139+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 msgr2=0x7fc9080a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.139+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 0x7fc9080a4d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc910000c00 tx=0x7fc910005fb0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.139+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 msgr2=0x7fc9241a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.139+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc91400eb30 tx=0x7fc91400ee40 comp rx=0 tx=0).stop 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 shutdown_connections 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9080a2890 0x7fc9080a4d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc924112fb0 0x7fc92419ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc924074010 0x7fc9241a4860 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 --2- 172.21.15.46:0/2979138845 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc924074930 0x7fc9241a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 >> 172.21.15.46:0/2979138845 conn(0x7fc924078f40 msgr2=0x7fc924111d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 shutdown_connections 2024-03-08T23:06:11.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.140+0000 7fc929200700 1 -- 172.21.15.46:0/2979138845 wait complete. 2024-03-08T23:06:11.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 -- 172.21.15.46:0/2392461075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 msgr2=0x7f50b806a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 --2- 172.21.15.46:0/2392461075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b806a430 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f50a40098f0 tx=0x7f50a4009c00 comp rx=0 tx=0).stop 2024-03-08T23:06:11.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 -- 172.21.15.46:0/2392461075 shutdown_connections 2024-03-08T23:06:11.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 --2- 172.21.15.46:0/2392461075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b81047e0 0x7f50b8104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 --2- 172.21.15.46:0/2392461075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b8106cd0 0x7f50b8069a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 --2- 172.21.15.46:0/2392461075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b806a430 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.686+0000 7f50bf400700 1 -- 172.21.15.46:0/2392461075 >> 172.21.15.46:0/2392461075 conn(0x7f50b8077c10 msgr2=0x7f50b8078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.687+0000 7f50bf400700 1 -- 172.21.15.46:0/2392461075 shutdown_connections 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.687+0000 7f50bf400700 1 -- 172.21.15.46:0/2392461075 wait complete. 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.689+0000 7f50bf400700 1 Processor -- start 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.689+0000 7f50bf400700 1 -- start start 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.689+0000 7f50bf400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.689+0000 7f50bf400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81047e0 0x7f50b81a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bf400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b8106cd0 0x7f50b819e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bf400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b8118670 con 0x7f50b8106cd0 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bf400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81184f0 con 0x7f50b8069fd0 2024-03-08T23:06:11.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bf400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b81187f0 con 0x7f50b81047e0 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bca00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43620/0 (socket says 172.21.15.46:43620) 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bca00700 1 -- 172.21.15.46:0/220744166 learned_addr learned my addr 172.21.15.46:0/220744166 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50bd400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b8106cd0 0x7f50b819e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.690+0000 7f50b7e00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81047e0 0x7f50b81a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 -- 172.21.15.46:0/220744166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81047e0 msgr2=0x7f50b81a4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81047e0 0x7f50b81a4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 -- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b8106cd0 msgr2=0x7f50b819e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b8106cd0 0x7f50b819e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 -- 172.21.15.46:0/220744166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b819f130 con 0x7f50b8069fd0 2024-03-08T23:06:11.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bca00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00ec10 tx=0x7f50ac00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac00cc90 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50ac00ce30 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ac010690 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.691+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b819f420 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.692+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b806b800 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.693+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b81012c0 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.697+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f50ac003680 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.699+0000 7f50b5600700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 0x7f50a00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.699+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f50ac0cf640 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.699+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f50ac01d060 con 0x7f50b8069fd0 2024-03-08T23:06:11.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.699+0000 7f50b7e00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 0x7f50a00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:11.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.700+0000 7f50b7e00700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 0x7f50a00a4dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f50a40098c0 tx=0x7f50a4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:11.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.946+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f50b806bb20 con 0x7f50b8069fd0 2024-03-08T23:06:11.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.947+0000 7f50b5600700 1 -- 172.21.15.46:0/220744166 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f50ac01f2f0 con 0x7f50b8069fd0 2024-03-08T23:06:11.277 INFO:teuthology.orchestra.run.smithi046.stdout:197568495676 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.949+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 msgr2=0x7f50a00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.949+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 0x7f50a00a4dd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f50a40098c0 tx=0x7f50a4019040 comp rx=0 tx=0).stop 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.949+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 msgr2=0x7f50b81a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.949+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50ac00ec10 tx=0x7f50ac00ef20 comp rx=0 tx=0).stop 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.949+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 shutdown_connections 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50a00a2910 0x7f50a00a4dd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50b8106cd0 0x7f50b819e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50b8069fd0 0x7f50b81a4810 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 --2- 172.21.15.46:0/220744166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50b81047e0 0x7f50b81a4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 >> 172.21.15.46:0/220744166 conn(0x7f50b8077c10 msgr2=0x7f50b81037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 shutdown_connections 2024-03-08T23:06:11.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.950+0000 7f50bf400700 1 -- 172.21.15.46:0/220744166 wait complete. 2024-03-08T23:06:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:10 smithi192 ceph-mon[36700]: pgmap v366: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:10 smithi192 ceph-mon[36700]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-08T23:06:12.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/220744166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-08T23:06:12.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1736938238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-08T23:06:12.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2979138845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-08T23:06:12.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/220744166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-08T23:06:12.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1736938238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-08T23:06:12.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2979138845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-08T23:06:12.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/220744166' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-08T23:06:12.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1736938238' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-08T23:06:12.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2979138845' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-08T23:06:12.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332144 got 219043332148 for osd.7 2024-03-08T23:06:12.933 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:13.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:12 smithi062 ceph-mon[35761]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-08T23:06:13.180 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:12 smithi046 ceph-mon[31859]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-08T23:06:13.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:12 smithi192 ceph-mon[36700]: pgmap v367: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-03-08T23:06:13.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 -- 172.21.15.46:0/16346731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 msgr2=0x7fbbc4102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 --2- 172.21.15.46:0/16346731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4102470 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbbb40099c0 tx=0x7fbbb4009cd0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 -- 172.21.15.46:0/16346731 shutdown_connections 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 --2- 172.21.15.46:0/16346731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 0x7fbbc410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 --2- 172.21.15.46:0/16346731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4102470 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 --2- 172.21.15.46:0/16346731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 -- 172.21.15.46:0/16346731 >> 172.21.15.46:0/16346731 conn(0x7fbbc4078f40 msgr2=0x7fbbc4079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 -- 172.21.15.46:0/16346731 shutdown_connections 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.430+0000 7fbbc9800700 1 -- 172.21.15.46:0/16346731 wait complete. 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 Processor -- start 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 -- start start 2024-03-08T23:06:13.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 0x7fbbc419cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41121c0 con 0x7fbbc4102090 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4112040 con 0x7fbbc4102a40 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.432+0000 7fbbc9800700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4112340 con 0x7fbbc4107160 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4198650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54710/0 (socket says 172.21.15.46:54710) 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2a00700 1 -- 172.21.15.46:0/404768920 learned_addr learned my addr 172.21.15.46:0/404768920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc3400700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 0x7fbbc419cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 -- 172.21.15.46:0/404768920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 msgr2=0x7fbbc419cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 0x7fbbc419cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 -- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 msgr2=0x7fbbc4198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:13.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.433+0000 7fbbc2000700 1 -- 172.21.15.46:0/404768920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc419d660 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbc2000700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4198b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e9c0 tx=0x7fbbb000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb000c9c0 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb000cb60 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc419d950 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0018680 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.434+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41a51d0 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.435+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc4068aa0 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.437+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbbb0010430 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.438+0000 7fbbbb400700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 0x7fbba80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.438+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fbbb00cf120 con 0x7fbbc4102a40 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.439+0000 7fbbc2a00700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 0x7fbba80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.439+0000 7fbbc2a00700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 0x7fbba80a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4000c00 tx=0x7fbbb4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:13.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.441+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbbb009beb0 con 0x7fbbc4102a40 2024-03-08T23:06:13.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.674+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbbc402d830 con 0x7fbbc4102a40 2024-03-08T23:06:13.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.675+0000 7fbbbb400700 1 -- 172.21.15.46:0/404768920 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbbb009b7b0 con 0x7fbbc4102a40 2024-03-08T23:06:13.829 INFO:teuthology.orchestra.run.smithi046.stdout:77309411412 2024-03-08T23:06:13.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 msgr2=0x7fbba80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:13.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 0x7fbba80a4d50 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4000c00 tx=0x7fbbb4005fb0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 msgr2=0x7fbbc4198b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4198b90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fbbb000e9c0 tx=0x7fbbb000ecd0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 shutdown_connections 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fbba80a2890 0x7fbba80a4d50 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbbc4107160 0x7fbbc419cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbbc4102090 0x7fbbc4198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 --2- 172.21.15.46:0/404768920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbbc4102a40 0x7fbbc4198b90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 >> 172.21.15.46:0/404768920 conn(0x7fbbc4078f40 msgr2=0x7fbbc4100040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 shutdown_connections 2024-03-08T23:06:13.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:13.678+0000 7fbbc9800700 1 -- 172.21.15.46:0/404768920 wait complete. 2024-03-08T23:06:13.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:13 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/404768920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-08T23:06:14.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:13 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/404768920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-08T23:06:14.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:13 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/404768920' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-08T23:06:14.681 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495671 got 197568495676 for osd.6 2024-03-08T23:06:14.681 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 -- 172.21.15.46:0/2407407613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 msgr2=0x7fc0fc108870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 --2- 172.21.15.46:0/2407407613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc108870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc0e80098f0 tx=0x7fc0e8009c00 comp rx=0 tx=0).stop 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 -- 172.21.15.46:0/2407407613 shutdown_connections 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 --2- 172.21.15.46:0/2407407613 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 --2- 172.21.15.46:0/2407407613 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc10e480 0x7fc0fc10e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 --2- 172.21.15.46:0/2407407613 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc108870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.758+0000 7fc100e00700 1 -- 172.21.15.46:0/2407407613 >> 172.21.15.46:0/2407407613 conn(0x7fc0fc0755b0 msgr2=0x7fc0fc0779d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:14.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.759+0000 7fc100e00700 1 -- 172.21.15.46:0/2407407613 shutdown_connections 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.759+0000 7fc100e00700 1 -- 172.21.15.46:0/2407407613 wait complete. 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.760+0000 7fc100e00700 1 Processor -- start 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.760+0000 7fc100e00700 1 -- start start 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc1a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc10e480 0x7fc0fc19ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc118730 con 0x7fc0fc108db0 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc1185b0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc100e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc1188b0 con 0x7fc0fc10e480 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.761+0000 7fc0f9400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1a4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:59544/0 (socket says 172.21.15.46:59544) 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9400700 1 -- 172.21.15.46:0/1736938238 learned_addr learned my addr 172.21.15.46:0/1736938238 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc1a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:14.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0fa800700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc10e480 0x7fc0fc19ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 -- 172.21.15.46:0/1736938238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc10e480 msgr2=0x7fc0fc19ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc10e480 0x7fc0fc19ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 -- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 msgr2=0x7fc0fc1a4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1a4da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.762+0000 7fc0f9e00700 1 -- 172.21.15.46:0/1736938238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0fc19f1d0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc0f9e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc1a4860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00eb90 tx=0x7fc0ec00eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec00cc90 con 0x7fc0fc1083f0 2024-03-08T23:06:14.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0ec00ce30 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ec010430 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc19f4c0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.763+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc1ab730 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.765+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc068aa0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.768+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc0ec003680 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.769+0000 7fc0f2a00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 0x7fc0e00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.769+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fc0ec0cefd0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.770+0000 7fc0f9400700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 0x7fc0e00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:14.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.770+0000 7fc0f9400700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 0x7fc0e00a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800c010 tx=0x7fc0e8019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:14.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:10.773+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc0ec09bcd0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.061+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fc0fc02d830 con 0x7fc0fc1083f0 2024-03-08T23:06:14.736 INFO:teuthology.orchestra.run.smithi046.stdout:55834574935 2024-03-08T23:06:14.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.062+0000 7fc0f2a00700 1 -- 172.21.15.46:0/1736938238 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0ec09b5d0 con 0x7fc0fc1083f0 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 msgr2=0x7fc0e00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 0x7fc0e00a4dd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fc0e800c010 tx=0x7fc0e8019040 comp rx=0 tx=0).stop 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 msgr2=0x7fc0fc1a4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc1a4860 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec00eb90 tx=0x7fc0ec00eea0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 shutdown_connections 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0e00a2910 0x7fc0e00a4dd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc0fc10e480 0x7fc0fc19ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc0fc1083f0 0x7fc0fc1a4860 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 --2- 172.21.15.46:0/1736938238 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc0fc108db0 0x7fc0fc1a4da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:14.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 >> 172.21.15.46:0/1736938238 conn(0x7fc0fc0755b0 msgr2=0x7fc0fc075bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:14.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 shutdown_connections 2024-03-08T23:06:14.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:11.065+0000 7fc100e00700 1 -- 172.21.15.46:0/1736938238 wait complete. 2024-03-08T23:06:15.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.001+0000 7fe222e00700 1 -- 172.21.15.46:0/2000084529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06a580 msgr2=0x7fe21c06a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.001+0000 7fe222e00700 1 --2- 172.21.15.46:0/2000084529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06a580 0x7fe21c06a960 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009920 tx=0x7fe20c009c30 comp rx=0 tx=0).stop 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.001+0000 7fe222e00700 1 -- 172.21.15.46:0/2000084529 shutdown_connections 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.001+0000 7fe222e00700 1 --2- 172.21.15.46:0/2000084529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c10ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.001+0000 7fe222e00700 1 --2- 172.21.15.46:0/2000084529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06a580 0x7fe21c06a960 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.002+0000 7fe222e00700 1 --2- 172.21.15.46:0/2000084529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c06af30 0x7fe21c1069e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.002+0000 7fe222e00700 1 -- 172.21.15.46:0/2000084529 >> 172.21.15.46:0/2000084529 conn(0x7fe21c076d30 msgr2=0x7fe21c077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.002+0000 7fe222e00700 1 -- 172.21.15.46:0/2000084529 shutdown_connections 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.002+0000 7fe222e00700 1 -- 172.21.15.46:0/2000084529 wait complete. 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 Processor -- start 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 -- start start 2024-03-08T23:06:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 0x7fe21c1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c19a910 0x7fe21c19ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c112110 con 0x7fe21c1070c0 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c111f90 con 0x7fe21c19a910 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.003+0000 7fe222e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe21c112290 con 0x7fe21c06af30 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54736/0 (socket says 172.21.15.46:54736) 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 -- 172.21.15.46:0/3238078814 learned_addr learned my addr 172.21.15.46:0/3238078814 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21be00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 0x7fe21c1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe220e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c19a910 0x7fe21c19ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 -- 172.21.15.46:0/3238078814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 msgr2=0x7fe21c1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 0x7fe21c1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 -- 172.21.15.46:0/3238078814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c19a910 msgr2=0x7fe21c19ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c19a910 0x7fe21c19ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21b400700 1 -- 172.21.15.46:0/3238078814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe21c19b490 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.004+0000 7fe21be00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 0x7fe21c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe21b400700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e9f0 tx=0x7fe20800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe208004a30 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe21c1a7620 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe20800cca0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2080056c0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.005+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe21c1a79c0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.006+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe21c068aa0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.008+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe20800c7c0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.010+0000 7fe218c00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 0x7fe2000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.010+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe2080cefa0 con 0x7fe21c1070c0 2024-03-08T23:06:15.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.010+0000 7fe21be00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 0x7fe2000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:15.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.011+0000 7fe21be00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 0x7fe2000a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009230 tx=0x7fe20c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:15.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.014+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe20809bca0 con 0x7fe21c1070c0 2024-03-08T23:06:15.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:14 smithi062 ceph-mon[35761]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-08T23:06:15.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:14 smithi046 ceph-mon[31859]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-08T23:06:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.259+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe21c1a7bf0 con 0x7fe21c1070c0 2024-03-08T23:06:15.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.259+0000 7fe218c00700 1 -- 172.21.15.46:0/3238078814 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe20809b5a0 con 0x7fe21c1070c0 2024-03-08T23:06:15.264 INFO:teuthology.orchestra.run.smithi046.stdout:34359738462 2024-03-08T23:06:15.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 msgr2=0x7fe2000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 0x7fe2000a4d50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe20c009230 tx=0x7fe20c019040 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 msgr2=0x7fe21c1a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe20800e9f0 tx=0x7fe20800ed00 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 shutdown_connections 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe2000a2890 0x7fe2000a4d50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe21c19a910 0x7fe21c19ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe21c06af30 0x7fe21c1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 --2- 172.21.15.46:0/3238078814 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe21c1070c0 0x7fe21c1a0c90 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 >> 172.21.15.46:0/3238078814 conn(0x7fe21c076d30 msgr2=0x7fe21c0fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 shutdown_connections 2024-03-08T23:06:15.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:15.262+0000 7fe222e00700 1 -- 172.21.15.46:0/3238078814 wait complete. 2024-03-08T23:06:15.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:14 smithi192 ceph-mon[36700]: pgmap v368: 129 pgs: 129 active+clean; 642 KiB data, 353 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2024-03-08T23:06:15.897 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:15 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3238078814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-08T23:06:15.941 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738462 for osd.0 2024-03-08T23:06:15.942 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:16.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:15 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3238078814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-08T23:06:16.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:15 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3238078814' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-08T23:06:16.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 -- 172.21.15.46:0/3494853816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c107100 msgr2=0x7f133c10ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:16.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 --2- 172.21.15.46:0/3494853816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c107100 0x7f133c10ecd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f133400b0d0 tx=0x7f133400b3e0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 -- 172.21.15.46:0/3494853816 shutdown_connections 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 --2- 172.21.15.46:0/3494853816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c107100 0x7f133c10ecd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 --2- 172.21.15.46:0/3494853816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f133c102020 0x7f133c102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 --2- 172.21.15.46:0/3494853816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c1029d0 0x7f133c106a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 -- 172.21.15.46:0/3494853816 >> 172.21.15.46:0/3494853816 conn(0x7f133c0fd890 msgr2=0x7f133c0ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 -- 172.21.15.46:0/3494853816 shutdown_connections 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.619+0000 7f1343400700 1 -- 172.21.15.46:0/3494853816 wait complete. 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.620+0000 7f1343400700 1 Processor -- start 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 -- start start 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f133c1029d0 0x7f133c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c107100 0x7f133c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c112160 con 0x7f133c102020 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c111fe0 con 0x7f133c1029d0 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1343400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c1122e0 con 0x7f133c107100 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1340a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1340a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54756/0 (socket says 172.21.15.46:54756) 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.621+0000 7f1340a00700 1 -- 172.21.15.46:0/2329476149 learned_addr learned my addr 172.21.15.46:0/2329476149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1341400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c107100 0x7f133c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 -- 172.21.15.46:0/2329476149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c107100 msgr2=0x7f133c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c107100 0x7f133c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 -- 172.21.15.46:0/2329476149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f133c1029d0 msgr2=0x7f133c19acd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f133c1029d0 0x7f133c19acd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 -- 172.21.15.46:0/2329476149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c19f750 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.622+0000 7f1340a00700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f133000e9a0 tx=0x7f133000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.623+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133000cbd0 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.623+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f133000cd70 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.623+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c19fa40 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.623+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330018750 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.623+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c1a72c0 con 0x7f133c102020 2024-03-08T23:06:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.624+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c068aa0 con 0x7f133c102020 2024-03-08T23:06:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.626+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1330010430 con 0x7f133c102020 2024-03-08T23:06:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.627+0000 7f1339600700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 0x7f13240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.628+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f13300cf0d0 con 0x7f133c102020 2024-03-08T23:06:16.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.628+0000 7f133be00700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 0x7f13240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:16.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.629+0000 7f133be00700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 0x7f13240a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f133c0ffd60 tx=0x7f1328009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.632+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f133009bdd0 con 0x7f133c102020 2024-03-08T23:06:16.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.864+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f133c1a74c0 con 0x7f133c102020 2024-03-08T23:06:16.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.865+0000 7f1339600700 1 -- 172.21.15.46:0/2329476149 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f13300860e0 con 0x7f133c102020 2024-03-08T23:06:16.869 INFO:teuthology.orchestra.run.smithi046.stdout:343597383713 2024-03-08T23:06:16.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 msgr2=0x7f13240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 0x7f13240a4d50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f133c0ffd60 tx=0x7f1328009260 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 msgr2=0x7f133c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f133000e9a0 tx=0x7f133000ecb0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 shutdown_connections 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f13240a2890 0x7f13240a4d50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f133c107100 0x7f133c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f133c102020 0x7f133c19a790 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 --2- 172.21.15.46:0/2329476149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f133c1029d0 0x7f133c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.867+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 >> 172.21.15.46:0/2329476149 conn(0x7f133c0fd890 msgr2=0x7f133c10d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.868+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 shutdown_connections 2024-03-08T23:06:16.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:16.868+0000 7f1343400700 1 -- 172.21.15.46:0/2329476149 wait complete. 2024-03-08T23:06:17.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:16 smithi046 ceph-mon[31859]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-08T23:06:17.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:16 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2329476149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-08T23:06:17.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:16 smithi192 ceph-mon[36700]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-08T23:06:17.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:16 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2329476149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-08T23:06:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:16 smithi062 ceph-mon[35761]: pgmap v369: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-08T23:06:17.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:16 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2329476149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-08T23:06:17.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182472 got 107374182476 for osd.3 2024-03-08T23:06:17.436 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 -- 172.21.15.46:0/3784041112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c06a3b0 msgr2=0x7fec7c10e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 --2- 172.21.15.46:0/3784041112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c06a3b0 0x7fec7c10e350 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec68009990 tx=0x7fec68009ca0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 -- 172.21.15.46:0/3784041112 shutdown_connections 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 --2- 172.21.15.46:0/3784041112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c110c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 --2- 172.21.15.46:0/3784041112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c069a90 0x7fec7c069e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 --2- 172.21.15.46:0/3784041112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c06a3b0 0x7fec7c10e350 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 -- 172.21.15.46:0/3784041112 >> 172.21.15.46:0/3784041112 conn(0x7fec7c077c10 msgr2=0x7fec7c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:17.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 -- 172.21.15.46:0/3784041112 shutdown_connections 2024-03-08T23:06:17.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.668+0000 7fec82e00700 1 -- 172.21.15.46:0/3784041112 wait complete. 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 Processor -- start 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 -- start start 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 0x7fec7c1a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c06a3b0 0x7fec7c1a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c114320 con 0x7fec7c069a90 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1141a0 con 0x7fec7c06a3b0 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.670+0000 7fec82e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c1144a0 con 0x7fec7c10e890 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec7be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 0x7fec7c1a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:17.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:17.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50250/0 (socket says 172.21.15.46:50250) 2024-03-08T23:06:17.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 -- 172.21.15.46:0/1164637637 learned_addr learned my addr 172.21.15.46:0/1164637637 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:17.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec7be00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 0x7fec7c1a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54780/0 (socket says 172.21.15.46:54780) 2024-03-08T23:06:17.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec7b400700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c06a3b0 0x7fec7c1a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:17.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 -- 172.21.15.46:0/1164637637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c06a3b0 msgr2=0x7fec7c1a0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:17.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c06a3b0 0x7fec7c1a0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 -- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 msgr2=0x7fec7c1a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 0x7fec7c1a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.671+0000 7fec80e00700 1 -- 172.21.15.46:0/1164637637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c19af10 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec80e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec7400ef90 tx=0x7fec7400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec740092e0 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec74004bb0 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec7c19b1a0 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec74004280 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.672+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7c1a7430 con 0x7fec7c10e890 2024-03-08T23:06:17.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.673+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7c068aa0 con 0x7fec7c10e890 2024-03-08T23:06:17.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.675+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fec7401b030 con 0x7fec7c10e890 2024-03-08T23:06:17.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.676+0000 7fec78c00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 0x7fec600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.677+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fec740cf270 con 0x7fec7c10e890 2024-03-08T23:06:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.677+0000 7fec7be00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 0x7fec600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:17.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.677+0000 7fec7be00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 0x7fec600a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec6c007be0 tx=0x7fec6c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:17.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.679+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fec74022020 con 0x7fec7c10e890 2024-03-08T23:06:17.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.899+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fec7c02d830 con 0x7fec7c10e890 2024-03-08T23:06:17.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.900+0000 7fec78c00700 1 -- 172.21.15.46:0/1164637637 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fec74010040 con 0x7fec7c10e890 2024-03-08T23:06:17.904 INFO:teuthology.orchestra.run.smithi046.stdout:141733920842 2024-03-08T23:06:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 msgr2=0x7fec600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:17.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 0x7fec600a4d50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec6c007be0 tx=0x7fec6c0073d0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 msgr2=0x7fec7c19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fec7400ef90 tx=0x7fec7400c3a0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 shutdown_connections 2024-03-08T23:06:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fec600a2890 0x7fec600a4d50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec7c10e890 0x7fec7c19a770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec7c069a90 0x7fec7c1a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 --2- 172.21.15.46:0/1164637637 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec7c06a3b0 0x7fec7c1a0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 >> 172.21.15.46:0/1164637637 conn(0x7fec7c077c10 msgr2=0x7fec7c10fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 shutdown_connections 2024-03-08T23:06:17.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:17.903+0000 7fec82e00700 1 -- 172.21.15.46:0/1164637637 wait complete. 2024-03-08T23:06:18.123 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574930 got 55834574935 for osd.1 2024-03-08T23:06:18.123 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:19.153 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:18 smithi046 ceph-mon[31859]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:19.153 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:18 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1164637637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-08T23:06:19.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:18 smithi192 ceph-mon[36700]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:19.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:18 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1164637637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-08T23:06:19.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:18 smithi062 ceph-mon[35761]: pgmap v370: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:19.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:18 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1164637637' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-08T23:06:19.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411408 got 77309411412 for osd.2 2024-03-08T23:06:19.775 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:19.852 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:19.888 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920842 for osd.4 2024-03-08T23:06:19.888 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:21.128 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:20 smithi046 ceph-mon[31859]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:21.135 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383706 got 343597383713 for osd.11 2024-03-08T23:06:21.135 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:21.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:20 smithi192 ceph-mon[36700]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:21.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:20 smithi062 ceph-mon[35761]: pgmap v371: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 -- 172.21.15.46:0/2327398034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c06a430 msgr2=0x7f598c10ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 --2- 172.21.15.46:0/2327398034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c06a430 0x7f598c10ec70 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f598400b0d0 tx=0x7f598400b3e0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 -- 172.21.15.46:0/2327398034 shutdown_connections 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 --2- 172.21.15.46:0/2327398034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c06a430 0x7f598c10ec70 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 --2- 172.21.15.46:0/2327398034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 --2- 172.21.15.46:0/2327398034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c069a90 0x7f598c069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 -- 172.21.15.46:0/2327398034 >> 172.21.15.46:0/2327398034 conn(0x7f598c077c10 msgr2=0x7f598c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 -- 172.21.15.46:0/2327398034 shutdown_connections 2024-03-08T23:06:21.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.863+0000 7f5994600700 1 -- 172.21.15.46:0/2327398034 wait complete. 2024-03-08T23:06:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.865+0000 7f5994600700 1 Processor -- start 2024-03-08T23:06:21.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.865+0000 7f5994600700 1 -- start start 2024-03-08T23:06:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c06a430 0x7f598c1012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:21.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c112080 con 0x7f598c069a90 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c111f00 con 0x7f598c107e30 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5994600700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c112200 con 0x7f598c06a430 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5991c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5991c00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54800/0 (socket says 172.21.15.46:54800) 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5991c00700 1 -- 172.21.15.46:0/1719283316 learned_addr learned my addr 172.21.15.46:0/1719283316 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5991200700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c06a430 0x7f598c1012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.866+0000 7f5992600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 -- 172.21.15.46:0/1719283316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c06a430 msgr2=0x7f598c1012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c06a430 0x7f598c1012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 -- 172.21.15.46:0/1719283316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 msgr2=0x7f598c101830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c101830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 -- 172.21.15.46:0/1719283316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c1a6a70 con 0x7f598c069a90 2024-03-08T23:06:21.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5992600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:21.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.867+0000 7f5991c00700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9a0 tx=0x7f598000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:21.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.868+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000ca10 con 0x7f598c069a90 2024-03-08T23:06:21.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.868+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c1a6d60 con 0x7f598c069a90 2024-03-08T23:06:21.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.868+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c1a7260 con 0x7f598c069a90 2024-03-08T23:06:21.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.868+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f598000cbb0 con 0x7f598c069a90 2024-03-08T23:06:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.868+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5980018680 con 0x7f598c069a90 2024-03-08T23:06:21.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.869+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596c005330 con 0x7f598c069a90 2024-03-08T23:06:21.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.871+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5980018820 con 0x7f598c069a90 2024-03-08T23:06:21.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.873+0000 7f598a000700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 0x7f59740a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:21.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.873+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f59800cee50 con 0x7f598c069a90 2024-03-08T23:06:21.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.873+0000 7f5991200700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 0x7f59740a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:21.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.874+0000 7f5991200700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 0x7f59740a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f598c078110 tx=0x7f5978009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:21.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:21.877+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f598009bbb0 con 0x7f598c069a90 2024-03-08T23:06:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.127+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f596c005620 con 0x7f598c069a90 2024-03-08T23:06:22.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.128+0000 7f598a000700 1 -- 172.21.15.46:0/1719283316 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f598009b4b0 con 0x7f598c069a90 2024-03-08T23:06:22.133 INFO:teuthology.orchestra.run.smithi046.stdout:287762808877 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 msgr2=0x7f59740a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 0x7f59740a4dd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f598c078110 tx=0x7f5978009260 comp rx=0 tx=0).stop 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 msgr2=0x7f598c100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f598000e9a0 tx=0x7f598000ecb0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 shutdown_connections 2024-03-08T23:06:22.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59740a2910 0x7f59740a4dd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f598c107e30 0x7f598c101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f598c069a90 0x7f598c100db0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 --2- 172.21.15.46:0/1719283316 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f598c06a430 0x7f598c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 >> 172.21.15.46:0/1719283316 conn(0x7f598c077c10 msgr2=0x7f598c0ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 shutdown_connections 2024-03-08T23:06:22.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.131+0000 7f5994600700 1 -- 172.21.15.46:0/1719283316 wait complete. 2024-03-08T23:06:22.405 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:22 smithi046 ceph-mon[31859]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-08T23:06:22.405 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.405 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:22 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.405 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:22 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1719283316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-08T23:06:22.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:22 smithi192 ceph-mon[36700]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-08T23:06:22.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:22 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:22 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1719283316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-08T23:06:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:22 smithi062 ceph-mon[35761]: pgmap v372: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 7 op/s 2024-03-08T23:06:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:22 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:22 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1719283316' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-08T23:06:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.480+0000 7fe9f2400700 1 -- 172.21.15.46:0/1167497753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074010 msgr2=0x7fe9ec0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.480+0000 7fe9f2400700 1 --2- 172.21.15.46:0/1167497753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074010 0x7fe9ec0743f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009990 tx=0x7fe9dc009ca0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 -- 172.21.15.46:0/1167497753 shutdown_connections 2024-03-08T23:06:22.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 --2- 172.21.15.46:0/1167497753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 --2- 172.21.15.46:0/1167497753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074010 0x7fe9ec0743f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 --2- 172.21.15.46:0/1167497753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec074930 0x7fe9ec10e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 -- 172.21.15.46:0/1167497753 >> 172.21.15.46:0/1167497753 conn(0x7fe9ec078f10 msgr2=0x7fe9ec079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 -- 172.21.15.46:0/1167497753 shutdown_connections 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.481+0000 7fe9f2400700 1 -- 172.21.15.46:0/1167497753 wait complete. 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.482+0000 7fe9f2400700 1 Processor -- start 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.482+0000 7fe9f2400700 1 -- start start 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec074010 0x7fe9ec1a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec19a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:22.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec114290 con 0x7fe9ec074930 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec114110 con 0x7fe9ec10e7e0 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9f2400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec114410 con 0x7fe9ec074010 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54816/0 (socket says 172.21.15.46:54816) 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 -- 172.21.15.46:0/2184193650 learned_addr learned my addr 172.21.15.46:0/2184193650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9ebe00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec19a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9eb400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec074010 0x7fe9ec1a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 -- 172.21.15.46:0/2184193650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec074010 msgr2=0x7fe9ec1a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec074010 0x7fe9ec1a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 -- 172.21.15.46:0/2184193650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 msgr2=0x7fe9ec19a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec19a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9e3e00700 1 -- 172.21.15.46:0/2184193650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ec19add0 con 0x7fe9ec074930 2024-03-08T23:06:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.483+0000 7fe9ebe00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9e3e00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400eb30 tx=0x7fe9d400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d400cbd0 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9d4004500 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d4010430 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec19b0c0 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.484+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec1a7330 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.485+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec068aa0 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.487+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe9d4003f00 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.489+0000 7fe9e8c00700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 0x7fe9d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.489+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fe9d40cef70 con 0x7fe9ec074930 2024-03-08T23:06:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.489+0000 7fe9eb400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 0x7fe9d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stdout:253403070516 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.490+0000 7fe9eb400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 0x7fe9d00a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0098e0 tx=0x7fe9dc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.493+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe9d409bc70 con 0x7fe9ec074930 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.724+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fe9ec19bb50 con 0x7fe9ec074930 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.725+0000 7fe9e8c00700 1 -- 172.21.15.46:0/2184193650 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe9d409b570 con 0x7fe9ec074930 2024-03-08T23:06:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.728+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 msgr2=0x7fe9d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.728+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 0x7fe9d00a4dd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc0098e0 tx=0x7fe9dc005960 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.728+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 msgr2=0x7fe9ec1a09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.728+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe9d400eb30 tx=0x7fe9d400ee40 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 shutdown_connections 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9d00a2910 0x7fe9d00a4dd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe9ec10e7e0 0x7fe9ec19a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe9ec074010 0x7fe9ec1a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 --2- 172.21.15.46:0/2184193650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe9ec074930 0x7fe9ec1a09a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 >> 172.21.15.46:0/2184193650 conn(0x7fe9ec078f10 msgr2=0x7fe9ec100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 shutdown_connections 2024-03-08T23:06:22.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:22.729+0000 7fe9f2400700 1 -- 172.21.15.46:0/2184193650 wait complete. 2024-03-08T23:06:23.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:23 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2184193650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-08T23:06:23.585 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:23 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2184193650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-08T23:06:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:23 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2184193650' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-08T23:06:24.338 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808877 for osd.9 2024-03-08T23:06:24.338 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:24 smithi192 ceph-mon[36700]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:06:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:06:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:06:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:24.587 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:24 smithi046 ceph-mon[31859]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:06:24.587 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:06:24.587 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:06:24.587 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:24 smithi062 ceph-mon[35761]: pgmap v373: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:06:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:06:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:06:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:06:25.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070510 got 253403070516 for osd.8 2024-03-08T23:06:25.037 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:25.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 -- 172.21.15.46:0/3868434360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0102a40 msgr2=0x7efcc0106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3868434360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0102a40 0x7efcc0106a80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7efca80099c0 tx=0x7efca8009cd0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 -- 172.21.15.46:0/3868434360 shutdown_connections 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3868434360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0107160 0x7efcc010ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3868434360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102090 0x7efcc0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3868434360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0102a40 0x7efcc0106a80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 -- 172.21.15.46:0/3868434360 >> 172.21.15.46:0/3868434360 conn(0x7efcc0078f40 msgr2=0x7efcc0079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 -- 172.21.15.46:0/3868434360 shutdown_connections 2024-03-08T23:06:25.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.604+0000 7efcc6e00700 1 -- 172.21.15.46:0/3868434360 wait complete. 2024-03-08T23:06:25.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.606+0000 7efcc6e00700 1 Processor -- start 2024-03-08T23:06:25.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.606+0000 7efcc6e00700 1 -- start start 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.606+0000 7efcc6e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc6e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102a40 0x7efcc01a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc6e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0107160 0x7efcc019a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc6e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0112170 con 0x7efcc0107160 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc6e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc0111ff0 con 0x7efcc0102a40 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc6e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc01122f0 con 0x7efcc0102090 2024-03-08T23:06:25.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:25.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44768/0 (socket says 172.21.15.46:44768) 2024-03-08T23:06:25.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 -- 172.21.15.46:0/3530586234 learned_addr learned my addr 172.21.15.46:0/3530586234 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:25.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcb7400700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102a40 0x7efcc01a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:25.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcc4e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0107160 0x7efcc019a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 -- 172.21.15.46:0/3530586234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102a40 msgr2=0x7efcc01a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102a40 0x7efcc01a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 -- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0107160 msgr2=0x7efcc019a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0107160 0x7efcc019a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.607+0000 7efcbfe00700 1 -- 172.21.15.46:0/3530586234 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc019af50 con 0x7efcc0102090 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcbfe00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcb000ef10 tx=0x7efcb000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb000c9c0 con 0x7efcc0102090 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc019b1e0 con 0x7efcc0102090 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcb0019070 con 0x7efcc0102090 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcb0014480 con 0x7efcc0102090 2024-03-08T23:06:25.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.608+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc01a7520 con 0x7efcc0102090 2024-03-08T23:06:25.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.609+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc0103560 con 0x7efcc0102090 2024-03-08T23:06:25.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.611+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efcb0012050 con 0x7efcc0102090 2024-03-08T23:06:25.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.613+0000 7efcbd600700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 0x7efca40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:25.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.613+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7efcb00cead0 con 0x7efcc0102090 2024-03-08T23:06:25.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.613+0000 7efcb7400700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 0x7efca40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:25.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.614+0000 7efcb7400700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 0x7efca40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:25.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.617+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efcb0024020 con 0x7efcc0102090 2024-03-08T23:06:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.846+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7efcc019bbd0 con 0x7efcc0102090 2024-03-08T23:06:25.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.847+0000 7efcbd600700 1 -- 172.21.15.46:0/3530586234 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7efcc019bbd0 con 0x7efcc0102090 2024-03-08T23:06:25.855 INFO:teuthology.orchestra.run.smithi046.stdout:309237645352 2024-03-08T23:06:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 msgr2=0x7efca40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 0x7efca40a4d50 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7efca8000c00 tx=0x7efca8005fb0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 msgr2=0x7efcc01a05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:25.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7efcb000ef10 tx=0x7efcb000c3a0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 shutdown_connections 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efca40a2890 0x7efca40a4d50 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcc0107160 0x7efcc019a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcc0102090 0x7efcc01a05a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 --2- 172.21.15.46:0/3530586234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcc0102a40 0x7efcc01a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 >> 172.21.15.46:0/3530586234 conn(0x7efcc0078f40 msgr2=0x7efcc01000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 shutdown_connections 2024-03-08T23:06:25.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:25.850+0000 7efcc6e00700 1 -- 172.21.15.46:0/3530586234 wait complete. 2024-03-08T23:06:26.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:26 smithi192 ceph-mon[36700]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:06:26.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:26 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3530586234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-08T23:06:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:26 smithi062 ceph-mon[35761]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:06:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:26 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3530586234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-08T23:06:26.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:26 smithi046 ceph-mon[31859]: pgmap v374: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:06:26.721 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:26 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3530586234' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-08T23:06:26.790 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645345 got 309237645352 for osd.10 2024-03-08T23:06:26.790 DEBUG:teuthology.parallel:result is None 2024-03-08T23:06:26.791 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-08T23:06:26.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-08T23:06:27.261 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:28.451 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:28 smithi046 ceph-mon[31859]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:28.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:28 smithi192 ceph-mon[36700]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:28.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:28 smithi062 ceph-mon[35761]: pgmap v375: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.337+0000 7f5f89400700 1 -- 172.21.15.46:0/548871623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84112fb0 msgr2=0x7f5f841153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.337+0000 7f5f89400700 1 --2- 172.21.15.46:0/548871623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84112fb0 0x7f5f841153a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00b0d0 tx=0x7f5f7c00b3e0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.337+0000 7f5f89400700 1 -- 172.21.15.46:0/548871623 shutdown_connections 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 --2- 172.21.15.46:0/548871623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84112fb0 0x7f5f841153a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 --2- 172.21.15.46:0/548871623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84074010 0x7f5f840743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 --2- 172.21.15.46:0/548871623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074930 0x7f5f84112a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 -- 172.21.15.46:0/548871623 >> 172.21.15.46:0/548871623 conn(0x7f5f84078f40 msgr2=0x7f5f84079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:29.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 -- 172.21.15.46:0/548871623 shutdown_connections 2024-03-08T23:06:29.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.338+0000 7f5f89400700 1 -- 172.21.15.46:0/548871623 wait complete. 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.339+0000 7f5f89400700 1 Processor -- start 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.339+0000 7f5f89400700 1 -- start start 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.339+0000 7f5f89400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f89400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84074930 0x7f5f841a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f89400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 0x7f5f8419e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f89400700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f841187e0 con 0x7f5f84074930 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f89400700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84118660 con 0x7f5f84112fb0 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f89400700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f84118960 con 0x7f5f84074010 2024-03-08T23:06:29.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f82800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f82800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44772/0 (socket says 172.21.15.46:44772) 2024-03-08T23:06:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f82800700 1 -- 172.21.15.46:0/2114353546 learned_addr learned my addr 172.21.15.46:0/2114353546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f81e00700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84074930 0x7f5f841a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f83200700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 0x7f5f8419e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f83200700 1 -- 172.21.15.46:0/2114353546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 msgr2=0x7f5f841a47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.340+0000 7f5f83200700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f83200700 1 -- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84074930 msgr2=0x7f5f841a4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f83200700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84074930 0x7f5f841a4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f83200700 1 -- 172.21.15.46:0/2114353546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f8419f100 con 0x7f5f84112fb0 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f82800700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f83200700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 0x7f5f8419e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00d010 tx=0x7f5f7c0079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c003c00 con 0x7f5f84112fb0 2024-03-08T23:06:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f8419f390 con 0x7f5f84112fb0 2024-03-08T23:06:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f7c022070 con 0x7f5f84112fb0 2024-03-08T23:06:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.341+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c01b460 con 0x7f5f84112fb0 2024-03-08T23:06:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.342+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f841ab6b0 con 0x7f5f84112fb0 2024-03-08T23:06:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.343+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f84068aa0 con 0x7f5f84112fb0 2024-03-08T23:06:29.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.345+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5f7c009b80 con 0x7f5f84112fb0 2024-03-08T23:06:29.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.346+0000 7f5f7b400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 0x7f5f680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:29.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.347+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f5f7c00f020 con 0x7f5f84112fb0 2024-03-08T23:06:29.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.347+0000 7f5f82800700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 0x7f5f680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:29.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.347+0000 7f5f82800700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 0x7f5f680a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5f8419fae0 tx=0x7f5f74009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:29.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.349+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5f7c09c5f0 con 0x7f5f84112fb0 2024-03-08T23:06:29.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.573+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f84112410 con 0x7f5f680a2910 2024-03-08T23:06:29.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.602+0000 7f5f7b400700 1 -- 172.21.15.46:0/2114353546 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340775 (secure 0 0 0) 0x7f5f84112410 con 0x7f5f680a2910 2024-03-08T23:06:29.614 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 msgr2=0x7f5f680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 0x7f5f680a4dd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5f8419fae0 tx=0x7f5f74009260 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 msgr2=0x7f5f8419e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 0x7f5f8419e9a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00d010 tx=0x7f5f7c0079a0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 shutdown_connections 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5f680a2910 0x7f5f680a4dd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5f84112fb0 0x7f5f8419e9a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5f84074010 0x7f5f841a47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 --2- 172.21.15.46:0/2114353546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5f84074930 0x7f5f841a4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.611+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 >> 172.21.15.46:0/2114353546 conn(0x7f5f84078f40 msgr2=0x7f5f84111d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:29.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.612+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 shutdown_connections 2024-03-08T23:06:29.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:29.612+0000 7f5f89400700 1 -- 172.21.15.46:0/2114353546 wait complete. 2024-03-08T23:06:29.618 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:06:30.394 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":376,"stamp":"2024-03-08T23:06:29.222623+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657597,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":649,"num_read_kb":448,"num_write":354,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365464,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576360,"statfs":{"total":1151940427776,"available":1151566192640,"internally_reserved":0,"allocated":12443648,"data_stored":7957841,"data_compressed":361056,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005802"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197423+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:06:09.197423+0000","last_peered":"2024-03-08T23:06:09.197423+0000","last_clean":"2024-03-08T23:06:09.197423+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:06:09.197423+0000","last_undegraded":"2024-03-08T23:06:09.197423+0000","last_fullsized":"2024-03-08T23:06:09.197423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200014+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:06:09.200014+0000","last_peered":"2024-03-08T23:06:09.200014+0000","last_clean":"2024-03-08T23:06:09.200014+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:06:09.200014+0000","last_undegraded":"2024-03-08T23:06:09.200014+0000","last_fullsized":"2024-03-08T23:06:09.200014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200340+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:06:09.200340+0000","last_peered":"2024-03-08T23:06:09.200340+0000","last_clean":"2024-03-08T23:06:09.200340+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:06:09.200340+0000","last_undegraded":"2024-03-08T23:06:09.200340+0000","last_fullsized":"2024-03-08T23:06:09.200340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.176978+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:06:09.176978+0000","last_peered":"2024-03-08T23:06:09.176978+0000","last_clean":"2024-03-08T23:06:09.176978+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:06:09.176978+0000","last_undegraded":"2024-03-08T23:06:09.176978+0000","last_fullsized":"2024-03-08T23:06:09.176978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181136+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:06:09.181136+0000","last_peered":"2024-03-08T23:06:09.181136+0000","last_clean":"2024-03-08T23:06:09.181136+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:06:09.181136+0000","last_undegraded":"2024-03-08T23:06:09.181136+0000","last_fullsized":"2024-03-08T23:06:09.181136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185368+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:06:09.185368+0000","last_peered":"2024-03-08T23:06:09.185368+0000","last_clean":"2024-03-08T23:06:09.185368+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:06:09.185368+0000","last_undegraded":"2024-03-08T23:06:09.185368+0000","last_fullsized":"2024-03-08T23:06:09.185368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180418+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:06:09.180418+0000","last_peered":"2024-03-08T23:06:09.180418+0000","last_clean":"2024-03-08T23:06:09.180418+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:06:09.180418+0000","last_undegraded":"2024-03-08T23:06:09.180418+0000","last_fullsized":"2024-03-08T23:06:09.180418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.187226+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:06:09.187226+0000","last_peered":"2024-03-08T23:06:09.187226+0000","last_clean":"2024-03-08T23:06:09.187226+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:06:09.187226+0000","last_undegraded":"2024-03-08T23:06:09.187226+0000","last_fullsized":"2024-03-08T23:06:09.187226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192440+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:06:09.192440+0000","last_peered":"2024-03-08T23:06:09.192440+0000","last_clean":"2024-03-08T23:06:09.192440+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:06:09.192440+0000","last_undegraded":"2024-03-08T23:06:09.192440+0000","last_fullsized":"2024-03-08T23:06:09.192440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180400+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:06:09.180400+0000","last_peered":"2024-03-08T23:06:09.180400+0000","last_clean":"2024-03-08T23:06:09.180400+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:06:09.180400+0000","last_undegraded":"2024-03-08T23:06:09.180400+0000","last_fullsized":"2024-03-08T23:06:09.180400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191719+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:06:09.191719+0000","last_peered":"2024-03-08T23:06:09.191719+0000","last_clean":"2024-03-08T23:06:09.191719+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:06:09.191719+0000","last_undegraded":"2024-03-08T23:06:09.191719+0000","last_fullsized":"2024-03-08T23:06:09.191719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.183365+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:06:09.183365+0000","last_peered":"2024-03-08T23:06:09.183365+0000","last_clean":"2024-03-08T23:06:09.183365+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:06:09.183365+0000","last_undegraded":"2024-03-08T23:06:09.183365+0000","last_fullsized":"2024-03-08T23:06:09.183365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191458+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:06:09.191458+0000","last_peered":"2024-03-08T23:06:09.191458+0000","last_clean":"2024-03-08T23:06:09.191458+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:06:09.191458+0000","last_undegraded":"2024-03-08T23:06:09.191458+0000","last_fullsized":"2024-03-08T23:06:09.191458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197044+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:06:09.197044+0000","last_peered":"2024-03-08T23:06:09.197044+0000","last_clean":"2024-03-08T23:06:09.197044+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:06:09.197044+0000","last_undegraded":"2024-03-08T23:06:09.197044+0000","last_fullsized":"2024-03-08T23:06:09.197044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.201322+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:06:09.201322+0000","last_peered":"2024-03-08T23:06:09.201322+0000","last_clean":"2024-03-08T23:06:09.201322+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:06:09.201322+0000","last_undegraded":"2024-03-08T23:06:09.201322+0000","last_fullsized":"2024-03-08T23:06:09.201322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190606+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:06:09.190606+0000","last_peered":"2024-03-08T23:06:09.190606+0000","last_clean":"2024-03-08T23:06:09.190606+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:06:09.190606+0000","last_undegraded":"2024-03-08T23:06:09.190606+0000","last_fullsized":"2024-03-08T23:06:09.190606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197573+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:06:09.197573+0000","last_peered":"2024-03-08T23:06:09.197573+0000","last_clean":"2024-03-08T23:06:09.197573+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:06:09.197573+0000","last_undegraded":"2024-03-08T23:06:09.197573+0000","last_fullsized":"2024-03-08T23:06:09.197573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192414+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:06:09.192414+0000","last_peered":"2024-03-08T23:06:09.192414+0000","last_clean":"2024-03-08T23:06:09.192414+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:06:09.192414+0000","last_undegraded":"2024-03-08T23:06:09.192414+0000","last_fullsized":"2024-03-08T23:06:09.192414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181120+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:06:09.181120+0000","last_peered":"2024-03-08T23:06:09.181120+0000","last_clean":"2024-03-08T23:06:09.181120+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:06:09.181120+0000","last_undegraded":"2024-03-08T23:06:09.181120+0000","last_fullsized":"2024-03-08T23:06:09.181120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.640979+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:05:40.640979+0000","last_peered":"2024-03-08T23:05:40.640979+0000","last_clean":"2024-03-08T23:05:40.640979+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:05:40.640979+0000","last_undegraded":"2024-03-08T23:05:40.640979+0000","last_fullsized":"2024-03-08T23:05:40.640979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.656756+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:05:40.656756+0000","last_peered":"2024-03-08T23:05:40.656756+0000","last_clean":"2024-03-08T23:05:40.656756+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:05:40.656756+0000","last_undegraded":"2024-03-08T23:05:40.656756+0000","last_fullsized":"2024-03-08T23:05:40.656756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.178888+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:06:09.178888+0000","last_peered":"2024-03-08T23:06:09.178888+0000","last_clean":"2024-03-08T23:06:09.178888+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:06:09.178888+0000","last_undegraded":"2024-03-08T23:06:09.178888+0000","last_fullsized":"2024-03-08T23:06:09.178888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643002+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:05:40.643002+0000","last_peered":"2024-03-08T23:05:40.643002+0000","last_clean":"2024-03-08T23:05:40.643002+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:05:40.643002+0000","last_undegraded":"2024-03-08T23:05:40.643002+0000","last_fullsized":"2024-03-08T23:05:40.643002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.649676+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:05:40.649676+0000","last_peered":"2024-03-08T23:05:40.649676+0000","last_clean":"2024-03-08T23:05:40.649676+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:05:40.649676+0000","last_undegraded":"2024-03-08T23:05:40.649676+0000","last_fullsized":"2024-03-08T23:05:40.649676+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205981+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:06:09.205981+0000","last_peered":"2024-03-08T23:06:09.205981+0000","last_clean":"2024-03-08T23:06:09.205981+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:06:09.205981+0000","last_undegraded":"2024-03-08T23:06:09.205981+0000","last_fullsized":"2024-03-08T23:06:09.205981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643462+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:05:40.643462+0000","last_peered":"2024-03-08T23:05:40.643462+0000","last_clean":"2024-03-08T23:05:40.643462+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:05:40.643462+0000","last_undegraded":"2024-03-08T23:05:40.643462+0000","last_fullsized":"2024-03-08T23:05:40.643462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.644175+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:05:40.644175+0000","last_peered":"2024-03-08T23:05:40.644175+0000","last_clean":"2024-03-08T23:05:40.644175+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:05:40.644175+0000","last_undegraded":"2024-03-08T23:05:40.644175+0000","last_fullsized":"2024-03-08T23:05:40.644175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.188540+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:06:09.188540+0000","last_peered":"2024-03-08T23:06:09.188540+0000","last_clean":"2024-03-08T23:06:09.188540+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:06:09.188540+0000","last_undegraded":"2024-03-08T23:06:09.188540+0000","last_fullsized":"2024-03-08T23:06:09.188540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643010+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:05:40.643010+0000","last_peered":"2024-03-08T23:05:40.643010+0000","last_clean":"2024-03-08T23:05:40.643010+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:05:40.643010+0000","last_undegraded":"2024-03-08T23:05:40.643010+0000","last_fullsized":"2024-03-08T23:05:40.643010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643666+0000","last_change":"2024-03-08T23:04:38.326247+0000","last_active":"2024-03-08T23:05:40.643666+0000","last_peered":"2024-03-08T23:05:40.643666+0000","last_clean":"2024-03-08T23:05:40.643666+0000","last_became_active":"2024-03-08T23:04:38.325843+0000","last_became_peered":"2024-03-08T23:04:38.325843+0000","last_unstale":"2024-03-08T23:05:40.643666+0000","last_undegraded":"2024-03-08T23:05:40.643666+0000","last_fullsized":"2024-03-08T23:05:40.643666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:38:07.047189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190480+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:06:09.190480+0000","last_peered":"2024-03-08T23:06:09.190480+0000","last_clean":"2024-03-08T23:06:09.190480+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:06:09.190480+0000","last_undegraded":"2024-03-08T23:06:09.190480+0000","last_fullsized":"2024-03-08T23:06:09.190480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641232+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:05:40.641232+0000","last_peered":"2024-03-08T23:05:40.641232+0000","last_clean":"2024-03-08T23:05:40.641232+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:05:40.641232+0000","last_undegraded":"2024-03-08T23:05:40.641232+0000","last_fullsized":"2024-03-08T23:05:40.641232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:27.215457+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:06:27.215457+0000","last_peered":"2024-03-08T23:06:27.215457+0000","last_clean":"2024-03-08T23:06:27.215457+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:06:27.215457+0000","last_undegraded":"2024-03-08T23:06:27.215457+0000","last_fullsized":"2024-03-08T23:06:27.215457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.172087+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:06:09.172087+0000","last_peered":"2024-03-08T23:06:09.172087+0000","last_clean":"2024-03-08T23:06:09.172087+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:06:09.172087+0000","last_undegraded":"2024-03-08T23:06:09.172087+0000","last_fullsized":"2024-03-08T23:06:09.172087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641148+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:05:40.641148+0000","last_peered":"2024-03-08T23:05:40.641148+0000","last_clean":"2024-03-08T23:05:40.641148+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:05:40.641148+0000","last_undegraded":"2024-03-08T23:05:40.641148+0000","last_fullsized":"2024-03-08T23:05:40.641148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645888+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:05:40.645888+0000","last_peered":"2024-03-08T23:05:40.645888+0000","last_clean":"2024-03-08T23:05:40.645888+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:05:40.645888+0000","last_undegraded":"2024-03-08T23:05:40.645888+0000","last_fullsized":"2024-03-08T23:05:40.645888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181806+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:06:09.181806+0000","last_peered":"2024-03-08T23:06:09.181806+0000","last_clean":"2024-03-08T23:06:09.181806+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:06:09.181806+0000","last_undegraded":"2024-03-08T23:06:09.181806+0000","last_fullsized":"2024-03-08T23:06:09.181806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645369+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:05:40.645369+0000","last_peered":"2024-03-08T23:05:40.645369+0000","last_clean":"2024-03-08T23:05:40.645369+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:05:40.645369+0000","last_undegraded":"2024-03-08T23:05:40.645369+0000","last_fullsized":"2024-03-08T23:05:40.645369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.655646+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:05:40.655646+0000","last_peered":"2024-03-08T23:05:40.655646+0000","last_clean":"2024-03-08T23:05:40.655646+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:05:40.655646+0000","last_undegraded":"2024-03-08T23:05:40.655646+0000","last_fullsized":"2024-03-08T23:05:40.655646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185980+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:06:09.185980+0000","last_peered":"2024-03-08T23:06:09.185980+0000","last_clean":"2024-03-08T23:06:09.185980+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:06:09.185980+0000","last_undegraded":"2024-03-08T23:06:09.185980+0000","last_fullsized":"2024-03-08T23:06:09.185980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.990386+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:05:40.990386+0000","last_peered":"2024-03-08T23:05:40.990386+0000","last_clean":"2024-03-08T23:05:40.990386+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:05:40.990386+0000","last_undegraded":"2024-03-08T23:05:40.990386+0000","last_fullsized":"2024-03-08T23:05:40.990386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146494+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:05:41.146494+0000","last_peered":"2024-03-08T23:05:41.146494+0000","last_clean":"2024-03-08T23:05:41.146494+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:05:41.146494+0000","last_undegraded":"2024-03-08T23:05:41.146494+0000","last_fullsized":"2024-03-08T23:05:41.146494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191415+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:06:09.191415+0000","last_peered":"2024-03-08T23:06:09.191415+0000","last_clean":"2024-03-08T23:06:09.191415+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:06:09.191415+0000","last_undegraded":"2024-03-08T23:06:09.191415+0000","last_fullsized":"2024-03-08T23:06:09.191415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650588+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:05:40.650588+0000","last_peered":"2024-03-08T23:05:40.650588+0000","last_clean":"2024-03-08T23:05:40.650588+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:05:40.650588+0000","last_undegraded":"2024-03-08T23:05:40.650588+0000","last_fullsized":"2024-03-08T23:05:40.650588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":110,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:23.209040+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:06:23.209040+0000","last_peered":"2024-03-08T23:06:23.209040+0000","last_clean":"2024-03-08T23:06:23.209040+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:06:23.209040+0000","last_undegraded":"2024-03-08T23:06:23.209040+0000","last_fullsized":"2024-03-08T23:06:23.209040+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":79,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190029+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:06:09.190029+0000","last_peered":"2024-03-08T23:06:09.190029+0000","last_clean":"2024-03-08T23:06:09.190029+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:06:09.190029+0000","last_undegraded":"2024-03-08T23:06:09.190029+0000","last_fullsized":"2024-03-08T23:06:09.190029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646899+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:05:40.646899+0000","last_peered":"2024-03-08T23:05:40.646899+0000","last_clean":"2024-03-08T23:05:40.646899+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:05:40.646899+0000","last_undegraded":"2024-03-08T23:05:40.646899+0000","last_fullsized":"2024-03-08T23:05:40.646899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646199+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:05:40.646199+0000","last_peered":"2024-03-08T23:05:40.646199+0000","last_clean":"2024-03-08T23:05:40.646199+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:05:40.646199+0000","last_undegraded":"2024-03-08T23:05:40.646199+0000","last_fullsized":"2024-03-08T23:05:40.646199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200351+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:06:09.200351+0000","last_peered":"2024-03-08T23:06:09.200351+0000","last_clean":"2024-03-08T23:06:09.200351+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:06:09.200351+0000","last_undegraded":"2024-03-08T23:06:09.200351+0000","last_fullsized":"2024-03-08T23:06:09.200351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.884963+0000","last_change":"2024-03-08T23:05:40.884963+0000","last_active":"2024-03-08T23:05:40.884963+0000","last_peered":"2024-03-08T23:05:40.884963+0000","last_clean":"2024-03-08T23:05:40.884963+0000","last_became_active":"2024-03-08T23:05:40.884402+0000","last_became_peered":"2024-03-08T23:05:40.884402+0000","last_unstale":"2024-03-08T23:05:40.884963+0000","last_undegraded":"2024-03-08T23:05:40.884963+0000","last_fullsized":"2024-03-08T23:05:40.884963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:26.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146436+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:05:41.146436+0000","last_peered":"2024-03-08T23:05:41.146436+0000","last_clean":"2024-03-08T23:05:41.146436+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:05:41.146436+0000","last_undegraded":"2024-03-08T23:05:41.146436+0000","last_fullsized":"2024-03-08T23:05:41.146436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185116+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:06:09.185116+0000","last_peered":"2024-03-08T23:06:09.185116+0000","last_clean":"2024-03-08T23:06:09.185116+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:06:09.185116+0000","last_undegraded":"2024-03-08T23:06:09.185116+0000","last_fullsized":"2024-03-08T23:06:09.185116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.148144+0000","last_change":"2024-03-08T23:05:41.148144+0000","last_active":"2024-03-08T23:05:41.148144+0000","last_peered":"2024-03-08T23:05:41.148144+0000","last_clean":"2024-03-08T23:05:41.148144+0000","last_became_active":"2024-03-08T23:05:41.147760+0000","last_became_peered":"2024-03-08T23:05:41.147760+0000","last_unstale":"2024-03-08T23:05:41.148144+0000","last_undegraded":"2024-03-08T23:05:41.148144+0000","last_fullsized":"2024-03-08T23:05:41.148144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:00.345274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'34","reported_seq":229,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:24.210418+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:06:24.210418+0000","last_peered":"2024-03-08T23:06:24.210418+0000","last_clean":"2024-03-08T23:06:24.210418+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:06:24.210418+0000","last_undegraded":"2024-03-08T23:06:24.210418+0000","last_fullsized":"2024-03-08T23:06:24.210418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62321,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":168,"num_read_kb":146,"num_write":35,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.171345+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:06:09.171345+0000","last_peered":"2024-03-08T23:06:09.171345+0000","last_clean":"2024-03-08T23:06:09.171345+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:06:09.171345+0000","last_undegraded":"2024-03-08T23:06:09.171345+0000","last_fullsized":"2024-03-08T23:06:09.171345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.659738+0000","last_change":"2024-03-08T23:05:40.659738+0000","last_active":"2024-03-08T23:05:40.659738+0000","last_peered":"2024-03-08T23:05:40.659738+0000","last_clean":"2024-03-08T23:05:40.659738+0000","last_became_active":"2024-03-08T23:05:40.659391+0000","last_became_peered":"2024-03-08T23:05:40.659391+0000","last_unstale":"2024-03-08T23:05:40.659738+0000","last_undegraded":"2024-03-08T23:05:40.659738+0000","last_fullsized":"2024-03-08T23:05:40.659738+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:42:03.247682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641425+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:05:40.641425+0000","last_peered":"2024-03-08T23:05:40.641425+0000","last_clean":"2024-03-08T23:05:40.641425+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:05:40.641425+0000","last_undegraded":"2024-03-08T23:05:40.641425+0000","last_fullsized":"2024-03-08T23:05:40.641425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.183853+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:06:09.183853+0000","last_peered":"2024-03-08T23:06:09.183853+0000","last_clean":"2024-03-08T23:06:09.183853+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:06:09.183853+0000","last_undegraded":"2024-03-08T23:06:09.183853+0000","last_fullsized":"2024-03-08T23:06:09.183853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645193+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:05:40.645193+0000","last_peered":"2024-03-08T23:05:40.645193+0000","last_clean":"2024-03-08T23:05:40.645193+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:05:40.645193+0000","last_undegraded":"2024-03-08T23:05:40.645193+0000","last_fullsized":"2024-03-08T23:05:40.645193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146818+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:05:41.146818+0000","last_peered":"2024-03-08T23:05:41.146818+0000","last_clean":"2024-03-08T23:05:41.146818+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:05:41.146818+0000","last_undegraded":"2024-03-08T23:05:41.146818+0000","last_fullsized":"2024-03-08T23:05:41.146818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.182986+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:06:09.182986+0000","last_peered":"2024-03-08T23:06:09.182986+0000","last_clean":"2024-03-08T23:06:09.182986+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:06:09.182986+0000","last_undegraded":"2024-03-08T23:06:09.182986+0000","last_fullsized":"2024-03-08T23:06:09.182986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.656114+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:05:40.656114+0000","last_peered":"2024-03-08T23:05:40.656114+0000","last_clean":"2024-03-08T23:05:40.656114+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:05:40.656114+0000","last_undegraded":"2024-03-08T23:05:40.656114+0000","last_fullsized":"2024-03-08T23:05:40.656114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:23.256343+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:06:23.256343+0000","last_peered":"2024-03-08T23:06:23.256343+0000","last_clean":"2024-03-08T23:06:23.256343+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:06:23.256343+0000","last_undegraded":"2024-03-08T23:06:23.256343+0000","last_fullsized":"2024-03-08T23:06:23.256343+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186482+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:06:09.186482+0000","last_peered":"2024-03-08T23:06:09.186482+0000","last_clean":"2024-03-08T23:06:09.186482+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:06:09.186482+0000","last_undegraded":"2024-03-08T23:06:09.186482+0000","last_fullsized":"2024-03-08T23:06:09.186482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.659109+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:05:40.659109+0000","last_peered":"2024-03-08T23:05:40.659109+0000","last_clean":"2024-03-08T23:05:40.659109+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:05:40.659109+0000","last_undegraded":"2024-03-08T23:05:40.659109+0000","last_fullsized":"2024-03-08T23:05:40.659109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643248+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:05:40.643248+0000","last_peered":"2024-03-08T23:05:40.643248+0000","last_clean":"2024-03-08T23:05:40.643248+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:05:40.643248+0000","last_undegraded":"2024-03-08T23:05:40.643248+0000","last_fullsized":"2024-03-08T23:05:40.643248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.167956+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:06:09.167956+0000","last_peered":"2024-03-08T23:06:09.167956+0000","last_clean":"2024-03-08T23:06:09.167956+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:06:09.167956+0000","last_undegraded":"2024-03-08T23:06:09.167956+0000","last_fullsized":"2024-03-08T23:06:09.167956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146341+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:05:41.146341+0000","last_peered":"2024-03-08T23:05:41.146341+0000","last_clean":"2024-03-08T23:05:41.146341+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:05:41.146341+0000","last_undegraded":"2024-03-08T23:05:41.146341+0000","last_fullsized":"2024-03-08T23:05:41.146341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643378+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:05:40.643378+0000","last_peered":"2024-03-08T23:05:40.643378+0000","last_clean":"2024-03-08T23:05:40.643378+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:05:40.643378+0000","last_undegraded":"2024-03-08T23:05:40.643378+0000","last_fullsized":"2024-03-08T23:05:40.643378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.165175+0000","last_change":"2024-03-08T23:05:41.148194+0000","last_active":"2024-03-08T23:06:09.165175+0000","last_peered":"2024-03-08T23:06:09.165175+0000","last_clean":"2024-03-08T23:06:09.165175+0000","last_became_active":"2024-03-08T23:05:41.147832+0000","last_became_peered":"2024-03-08T23:05:41.147832+0000","last_unstale":"2024-03-08T23:06:09.165175+0000","last_undegraded":"2024-03-08T23:06:09.165175+0000","last_fullsized":"2024-03-08T23:06:09.165175+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:38.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643581+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:05:40.643581+0000","last_peered":"2024-03-08T23:05:40.643581+0000","last_clean":"2024-03-08T23:05:40.643581+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:05:40.643581+0000","last_undegraded":"2024-03-08T23:05:40.643581+0000","last_fullsized":"2024-03-08T23:05:40.643581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.653241+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:05:40.653241+0000","last_peered":"2024-03-08T23:05:40.653241+0000","last_clean":"2024-03-08T23:05:40.653241+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:05:40.653241+0000","last_undegraded":"2024-03-08T23:05:40.653241+0000","last_fullsized":"2024-03-08T23:05:40.653241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185050+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:06:09.185050+0000","last_peered":"2024-03-08T23:06:09.185050+0000","last_clean":"2024-03-08T23:06:09.185050+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:06:09.185050+0000","last_undegraded":"2024-03-08T23:06:09.185050+0000","last_fullsized":"2024-03-08T23:06:09.185050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.658548+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:05:40.658548+0000","last_peered":"2024-03-08T23:05:40.658548+0000","last_clean":"2024-03-08T23:05:40.658548+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:05:40.658548+0000","last_undegraded":"2024-03-08T23:05:40.658548+0000","last_fullsized":"2024-03-08T23:05:40.658548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646508+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:05:40.646508+0000","last_peered":"2024-03-08T23:05:40.646508+0000","last_clean":"2024-03-08T23:05:40.646508+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:05:40.646508+0000","last_undegraded":"2024-03-08T23:05:40.646508+0000","last_fullsized":"2024-03-08T23:05:40.646508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.204821+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:06:09.204821+0000","last_peered":"2024-03-08T23:06:09.204821+0000","last_clean":"2024-03-08T23:06:09.204821+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:06:09.204821+0000","last_undegraded":"2024-03-08T23:06:09.204821+0000","last_fullsized":"2024-03-08T23:06:09.204821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.658990+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:05:40.658990+0000","last_peered":"2024-03-08T23:05:40.658990+0000","last_clean":"2024-03-08T23:05:40.658990+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:05:40.658990+0000","last_undegraded":"2024-03-08T23:05:40.658990+0000","last_fullsized":"2024-03-08T23:05:40.658990+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181912+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:06:09.181912+0000","last_peered":"2024-03-08T23:06:09.181912+0000","last_clean":"2024-03-08T23:06:09.181912+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:06:09.181912+0000","last_undegraded":"2024-03-08T23:06:09.181912+0000","last_fullsized":"2024-03-08T23:06:09.181912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":114,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:26.261999+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:06:26.261999+0000","last_peered":"2024-03-08T23:06:26.261999+0000","last_clean":"2024-03-08T23:06:26.261999+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:06:26.261999+0000","last_undegraded":"2024-03-08T23:06:26.261999+0000","last_fullsized":"2024-03-08T23:06:26.261999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646715+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:05:40.646715+0000","last_peered":"2024-03-08T23:05:40.646715+0000","last_clean":"2024-03-08T23:05:40.646715+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:05:40.646715+0000","last_undegraded":"2024-03-08T23:05:40.646715+0000","last_fullsized":"2024-03-08T23:05:40.646715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186677+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:06:09.186677+0000","last_peered":"2024-03-08T23:06:09.186677+0000","last_clean":"2024-03-08T23:06:09.186677+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:06:09.186677+0000","last_undegraded":"2024-03-08T23:06:09.186677+0000","last_fullsized":"2024-03-08T23:06:09.186677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643539+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:05:40.643539+0000","last_peered":"2024-03-08T23:05:40.643539+0000","last_clean":"2024-03-08T23:05:40.643539+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:05:40.643539+0000","last_undegraded":"2024-03-08T23:05:40.643539+0000","last_fullsized":"2024-03-08T23:05:40.643539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645314+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:05:40.645314+0000","last_peered":"2024-03-08T23:05:40.645314+0000","last_clean":"2024-03-08T23:05:40.645314+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:05:40.645314+0000","last_undegraded":"2024-03-08T23:05:40.645314+0000","last_fullsized":"2024-03-08T23:05:40.645314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146553+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:05:41.146553+0000","last_peered":"2024-03-08T23:05:41.146553+0000","last_clean":"2024-03-08T23:05:41.146553+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:05:41.146553+0000","last_undegraded":"2024-03-08T23:05:41.146553+0000","last_fullsized":"2024-03-08T23:05:41.146553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.179062+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:06:09.179062+0000","last_peered":"2024-03-08T23:06:09.179062+0000","last_clean":"2024-03-08T23:06:09.179062+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:06:09.179062+0000","last_undegraded":"2024-03-08T23:06:09.179062+0000","last_fullsized":"2024-03-08T23:06:09.179062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645321+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:05:40.645321+0000","last_peered":"2024-03-08T23:05:40.645321+0000","last_clean":"2024-03-08T23:05:40.645321+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:05:40.645321+0000","last_undegraded":"2024-03-08T23:05:40.645321+0000","last_fullsized":"2024-03-08T23:05:40.645321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645097+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:05:40.645097+0000","last_peered":"2024-03-08T23:05:40.645097+0000","last_clean":"2024-03-08T23:05:40.645097+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:05:40.645097+0000","last_undegraded":"2024-03-08T23:05:40.645097+0000","last_fullsized":"2024-03-08T23:05:40.645097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643982+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:05:40.643982+0000","last_peered":"2024-03-08T23:05:40.643982+0000","last_clean":"2024-03-08T23:05:40.643982+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:05:40.643982+0000","last_undegraded":"2024-03-08T23:05:40.643982+0000","last_fullsized":"2024-03-08T23:05:40.643982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.169973+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:06:09.169973+0000","last_peered":"2024-03-08T23:06:09.169973+0000","last_clean":"2024-03-08T23:06:09.169973+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:06:09.169973+0000","last_undegraded":"2024-03-08T23:06:09.169973+0000","last_fullsized":"2024-03-08T23:06:09.169973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650040+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:05:40.650040+0000","last_peered":"2024-03-08T23:05:40.650040+0000","last_clean":"2024-03-08T23:05:40.650040+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:05:40.650040+0000","last_undegraded":"2024-03-08T23:05:40.650040+0000","last_fullsized":"2024-03-08T23:05:40.650040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.655131+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:05:40.655131+0000","last_peered":"2024-03-08T23:05:40.655131+0000","last_clean":"2024-03-08T23:05:40.655131+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:05:40.655131+0000","last_undegraded":"2024-03-08T23:05:40.655131+0000","last_fullsized":"2024-03-08T23:05:40.655131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180152+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:06:09.180152+0000","last_peered":"2024-03-08T23:06:09.180152+0000","last_clean":"2024-03-08T23:06:09.180152+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:06:09.180152+0000","last_undegraded":"2024-03-08T23:06:09.180152+0000","last_fullsized":"2024-03-08T23:06:09.180152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.640881+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:05:40.640881+0000","last_peered":"2024-03-08T23:05:40.640881+0000","last_clean":"2024-03-08T23:05:40.640881+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:05:40.640881+0000","last_undegraded":"2024-03-08T23:05:40.640881+0000","last_fullsized":"2024-03-08T23:05:40.640881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643332+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:05:40.643332+0000","last_peered":"2024-03-08T23:05:40.643332+0000","last_clean":"2024-03-08T23:05:40.643332+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:05:40.643332+0000","last_undegraded":"2024-03-08T23:05:40.643332+0000","last_fullsized":"2024-03-08T23:05:40.643332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205327+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:06:09.205327+0000","last_peered":"2024-03-08T23:06:09.205327+0000","last_clean":"2024-03-08T23:06:09.205327+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:06:09.205327+0000","last_undegraded":"2024-03-08T23:06:09.205327+0000","last_fullsized":"2024-03-08T23:06:09.205327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643692+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:05:40.643692+0000","last_peered":"2024-03-08T23:05:40.643692+0000","last_clean":"2024-03-08T23:05:40.643692+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:05:40.643692+0000","last_undegraded":"2024-03-08T23:05:40.643692+0000","last_fullsized":"2024-03-08T23:05:40.643692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.883136+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:05:40.883136+0000","last_peered":"2024-03-08T23:05:40.883136+0000","last_clean":"2024-03-08T23:05:40.883136+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:05:40.883136+0000","last_undegraded":"2024-03-08T23:05:40.883136+0000","last_fullsized":"2024-03-08T23:05:40.883136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.138622+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:06:09.138622+0000","last_peered":"2024-03-08T23:06:09.138622+0000","last_clean":"2024-03-08T23:06:09.138622+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:06:09.138622+0000","last_undegraded":"2024-03-08T23:06:09.138622+0000","last_fullsized":"2024-03-08T23:06:09.138622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.148869+0000","last_change":"2024-03-08T23:05:41.148196+0000","last_active":"2024-03-08T23:05:41.148869+0000","last_peered":"2024-03-08T23:05:41.148869+0000","last_clean":"2024-03-08T23:05:41.148869+0000","last_became_active":"2024-03-08T23:05:41.147730+0000","last_became_peered":"2024-03-08T23:05:41.147730+0000","last_unstale":"2024-03-08T23:05:41.148869+0000","last_undegraded":"2024-03-08T23:05:41.148869+0000","last_fullsized":"2024-03-08T23:05:41.148869+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:21:05.284970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650718+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:05:40.650718+0000","last_peered":"2024-03-08T23:05:40.650718+0000","last_clean":"2024-03-08T23:05:40.650718+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:05:40.650718+0000","last_undegraded":"2024-03-08T23:05:40.650718+0000","last_fullsized":"2024-03-08T23:05:40.650718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.194371+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:06:09.194371+0000","last_peered":"2024-03-08T23:06:09.194371+0000","last_clean":"2024-03-08T23:06:09.194371+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:06:09.194371+0000","last_undegraded":"2024-03-08T23:06:09.194371+0000","last_fullsized":"2024-03-08T23:06:09.194371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643863+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:05:40.643863+0000","last_peered":"2024-03-08T23:05:40.643863+0000","last_clean":"2024-03-08T23:05:40.643863+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:05:40.643863+0000","last_undegraded":"2024-03-08T23:05:40.643863+0000","last_fullsized":"2024-03-08T23:05:40.643863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641034+0000","last_change":"2024-03-08T23:04:38.331866+0000","last_active":"2024-03-08T23:05:40.641034+0000","last_peered":"2024-03-08T23:05:40.641034+0000","last_clean":"2024-03-08T23:05:40.641034+0000","last_became_active":"2024-03-08T23:04:38.331506+0000","last_became_peered":"2024-03-08T23:04:38.331506+0000","last_unstale":"2024-03-08T23:05:40.641034+0000","last_undegraded":"2024-03-08T23:05:40.641034+0000","last_fullsized":"2024-03-08T23:05:40.641034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:02:10.577391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200743+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:06:09.200743+0000","last_peered":"2024-03-08T23:06:09.200743+0000","last_clean":"2024-03-08T23:06:09.200743+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:06:09.200743+0000","last_undegraded":"2024-03-08T23:06:09.200743+0000","last_fullsized":"2024-03-08T23:06:09.200743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643200+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:05:40.643200+0000","last_peered":"2024-03-08T23:05:40.643200+0000","last_clean":"2024-03-08T23:05:40.643200+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:05:40.643200+0000","last_undegraded":"2024-03-08T23:05:40.643200+0000","last_fullsized":"2024-03-08T23:05:40.643200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643297+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:05:40.643297+0000","last_peered":"2024-03-08T23:05:40.643297+0000","last_clean":"2024-03-08T23:05:40.643297+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:05:40.643297+0000","last_undegraded":"2024-03-08T23:05:40.643297+0000","last_fullsized":"2024-03-08T23:05:40.643297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191305+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:06:09.191305+0000","last_peered":"2024-03-08T23:06:09.191305+0000","last_clean":"2024-03-08T23:06:09.191305+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:06:09.191305+0000","last_undegraded":"2024-03-08T23:06:09.191305+0000","last_fullsized":"2024-03-08T23:06:09.191305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.883258+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:05:40.883258+0000","last_peered":"2024-03-08T23:05:40.883258+0000","last_clean":"2024-03-08T23:05:40.883258+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:05:40.883258+0000","last_undegraded":"2024-03-08T23:05:40.883258+0000","last_fullsized":"2024-03-08T23:05:40.883258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641438+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:05:40.641438+0000","last_peered":"2024-03-08T23:05:40.641438+0000","last_clean":"2024-03-08T23:05:40.641438+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:05:40.641438+0000","last_undegraded":"2024-03-08T23:05:40.641438+0000","last_fullsized":"2024-03-08T23:05:40.641438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191852+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:06:09.191852+0000","last_peered":"2024-03-08T23:06:09.191852+0000","last_clean":"2024-03-08T23:06:09.191852+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:06:09.191852+0000","last_undegraded":"2024-03-08T23:06:09.191852+0000","last_fullsized":"2024-03-08T23:06:09.191852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.989251+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:05:40.989251+0000","last_peered":"2024-03-08T23:05:40.989251+0000","last_clean":"2024-03-08T23:05:40.989251+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:05:40.989251+0000","last_undegraded":"2024-03-08T23:05:40.989251+0000","last_fullsized":"2024-03-08T23:05:40.989251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146777+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:05:41.146777+0000","last_peered":"2024-03-08T23:05:41.146777+0000","last_clean":"2024-03-08T23:05:41.146777+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:05:41.146777+0000","last_undegraded":"2024-03-08T23:05:41.146777+0000","last_fullsized":"2024-03-08T23:05:41.146777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.174007+0000","last_change":"2024-03-08T23:05:41.148243+0000","last_active":"2024-03-08T23:06:09.174007+0000","last_peered":"2024-03-08T23:06:09.174007+0000","last_clean":"2024-03-08T23:06:09.174007+0000","last_became_active":"2024-03-08T23:05:41.147924+0000","last_became_peered":"2024-03-08T23:05:41.147924+0000","last_unstale":"2024-03-08T23:06:09.174007+0000","last_undegraded":"2024-03-08T23:06:09.174007+0000","last_fullsized":"2024-03-08T23:06:09.174007+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:47.129092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146018+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:05:41.146018+0000","last_peered":"2024-03-08T23:05:41.146018+0000","last_clean":"2024-03-08T23:05:41.146018+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:05:41.146018+0000","last_undegraded":"2024-03-08T23:05:41.146018+0000","last_fullsized":"2024-03-08T23:05:41.146018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.661217+0000","last_change":"2024-03-08T23:05:40.661217+0000","last_active":"2024-03-08T23:05:40.661217+0000","last_peered":"2024-03-08T23:05:40.661217+0000","last_clean":"2024-03-08T23:05:40.661217+0000","last_became_active":"2024-03-08T23:05:40.657165+0000","last_became_peered":"2024-03-08T23:05:40.657165+0000","last_unstale":"2024-03-08T23:05:40.661217+0000","last_undegraded":"2024-03-08T23:05:40.661217+0000","last_fullsized":"2024-03-08T23:05:40.661217+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:16.125758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.173872+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:06:09.173872+0000","last_peered":"2024-03-08T23:06:09.173872+0000","last_clean":"2024-03-08T23:06:09.173872+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:06:09.173872+0000","last_undegraded":"2024-03-08T23:06:09.173872+0000","last_fullsized":"2024-03-08T23:06:09.173872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.171031+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:06:09.171031+0000","last_peered":"2024-03-08T23:06:09.171031+0000","last_clean":"2024-03-08T23:06:09.171031+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:06:09.171031+0000","last_undegraded":"2024-03-08T23:06:09.171031+0000","last_fullsized":"2024-03-08T23:06:09.171031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.179063+0000","last_change":"2024-03-08T23:05:40.651674+0000","last_active":"2024-03-08T23:06:09.179063+0000","last_peered":"2024-03-08T23:06:09.179063+0000","last_clean":"2024-03-08T23:06:09.179063+0000","last_became_active":"2024-03-08T23:05:40.651393+0000","last_became_peered":"2024-03-08T23:05:40.651393+0000","last_unstale":"2024-03-08T23:06:09.179063+0000","last_undegraded":"2024-03-08T23:06:09.179063+0000","last_fullsized":"2024-03-08T23:06:09.179063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:30:44.691787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197159+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:06:09.197159+0000","last_peered":"2024-03-08T23:06:09.197159+0000","last_clean":"2024-03-08T23:06:09.197159+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:06:09.197159+0000","last_undegraded":"2024-03-08T23:06:09.197159+0000","last_fullsized":"2024-03-08T23:06:09.197159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205178+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:06:09.205178+0000","last_peered":"2024-03-08T23:06:09.205178+0000","last_clean":"2024-03-08T23:06:09.205178+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:06:09.205178+0000","last_undegraded":"2024-03-08T23:06:09.205178+0000","last_fullsized":"2024-03-08T23:06:09.205178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197326+0000","last_change":"2024-03-08T23:05:40.993334+0000","last_active":"2024-03-08T23:06:09.197326+0000","last_peered":"2024-03-08T23:06:09.197326+0000","last_clean":"2024-03-08T23:06:09.197326+0000","last_became_active":"2024-03-08T23:05:40.991755+0000","last_became_peered":"2024-03-08T23:05:40.991755+0000","last_unstale":"2024-03-08T23:06:09.197326+0000","last_undegraded":"2024-03-08T23:06:09.197326+0000","last_fullsized":"2024-03-08T23:06:09.197326+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:31.085388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.166651+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:06:09.166651+0000","last_peered":"2024-03-08T23:06:09.166651+0000","last_clean":"2024-03-08T23:06:09.166651+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:06:09.166651+0000","last_undegraded":"2024-03-08T23:06:09.166651+0000","last_fullsized":"2024-03-08T23:06:09.166651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191349+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:06:09.191349+0000","last_peered":"2024-03-08T23:06:09.191349+0000","last_clean":"2024-03-08T23:06:09.191349+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:06:09.191349+0000","last_undegraded":"2024-03-08T23:06:09.191349+0000","last_fullsized":"2024-03-08T23:06:09.191349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.182229+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:06:09.182229+0000","last_peered":"2024-03-08T23:06:09.182229+0000","last_clean":"2024-03-08T23:06:09.182229+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:06:09.182229+0000","last_undegraded":"2024-03-08T23:06:09.182229+0000","last_fullsized":"2024-03-08T23:06:09.182229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186392+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:06:09.186392+0000","last_peered":"2024-03-08T23:06:09.186392+0000","last_clean":"2024-03-08T23:06:09.186392+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:06:09.186392+0000","last_undegraded":"2024-03-08T23:06:09.186392+0000","last_fullsized":"2024-03-08T23:06:09.186392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.177332+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:06:09.177332+0000","last_peered":"2024-03-08T23:06:09.177332+0000","last_clean":"2024-03-08T23:06:09.177332+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:06:09.177332+0000","last_undegraded":"2024-03-08T23:06:09.177332+0000","last_fullsized":"2024-03-08T23:06:09.177332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192885+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:06:09.192885+0000","last_peered":"2024-03-08T23:06:09.192885+0000","last_clean":"2024-03-08T23:06:09.192885+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:06:09.192885+0000","last_undegraded":"2024-03-08T23:06:09.192885+0000","last_fullsized":"2024-03-08T23:06:09.192885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.207893+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:06:09.207893+0000","last_peered":"2024-03-08T23:06:09.207893+0000","last_clean":"2024-03-08T23:06:09.207893+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:06:09.207893+0000","last_undegraded":"2024-03-08T23:06:09.207893+0000","last_fullsized":"2024-03-08T23:06:09.207893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185435+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:06:09.185435+0000","last_peered":"2024-03-08T23:06:09.185435+0000","last_clean":"2024-03-08T23:06:09.185435+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:06:09.185435+0000","last_undegraded":"2024-03-08T23:06:09.185435+0000","last_fullsized":"2024-03-08T23:06:09.185435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66995,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":513,"num_read_kb":329,"num_write":171,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":200985,"data_compressed":5463,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383716,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":500760,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1150881,"data_compressed":58609,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":505092,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":500916,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1097498,"data_compressed":56788,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495680,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":496172,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":558403,"data_compressed":22402,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":497560,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":560285,"data_compressed":22402,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496366,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":501354,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1092554,"data_compressed":56788,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62321,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64203,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64431,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:06:30.400 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-08T23:06:30.526 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:30 smithi046 ceph-mon[31859]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:30.526 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:30 smithi046 ceph-mon[31859]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:30 smithi192 ceph-mon[36700]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:30 smithi192 ceph-mon[36700]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:30 smithi062 ceph-mon[35761]: pgmap v376: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:30 smithi062 ceph-mon[35761]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:30.924 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:32.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:32 smithi192 ceph-mon[36700]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:32.628 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:32 smithi046 ceph-mon[31859]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:32.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:32 smithi062 ceph-mon[35761]: pgmap v377: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.128+0000 7f1c3f000700 1 -- 172.21.15.46:0/3490856325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 msgr2=0x7f1c381022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.128+0000 7f1c3f000700 1 --2- 172.21.15.46:0/3490856325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c381022b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c240099c0 tx=0x7f1c24009cd0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 -- 172.21.15.46:0/3490856325 shutdown_connections 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 --2- 172.21.15.46:0/3490856325 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c381027f0 0x7f1c3810ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 --2- 172.21.15.46:0/3490856325 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c38107e10 0x7f1c381081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 --2- 172.21.15.46:0/3490856325 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c381022b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 -- 172.21.15.46:0/3490856325 >> 172.21.15.46:0/3490856325 conn(0x7f1c38078f10 msgr2=0x7f1c38079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 -- 172.21.15.46:0/3490856325 shutdown_connections 2024-03-08T23:06:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.129+0000 7f1c3f000700 1 -- 172.21.15.46:0/3490856325 wait complete. 2024-03-08T23:06:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.130+0000 7f1c3f000700 1 Processor -- start 2024-03-08T23:06:33.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.130+0000 7f1c3f000700 1 -- start start 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c3819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c38107e10 0x7f1c3819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c381120b0 con 0x7f1c38107e10 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c38111f30 con 0x7f1c38101e30 2024-03-08T23:06:33.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c3f000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c38112230 con 0x7f1c381027f0 2024-03-08T23:06:33.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c37400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c37e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c3819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c37400700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:42768/0 (socket says 172.21.15.46:42768) 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.131+0000 7f1c37400700 1 -- 172.21.15.46:0/4126129264 learned_addr learned my addr 172.21.15.46:0/4126129264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c3d000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c38107e10 0x7f1c3819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 -- 172.21.15.46:0/4126129264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 msgr2=0x7f1c3819a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c3819a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 -- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c38107e10 msgr2=0x7f1c3819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c38107e10 0x7f1c3819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 -- 172.21.15.46:0/4126129264 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3819f710 con 0x7f1c381027f0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.132+0000 7f1c37400700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c24006010 tx=0x7f1c24003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.133+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c24003680 con 0x7f1c381027f0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.133+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c3819f9a0 con 0x7f1c381027f0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.133+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c24003e50 con 0x7f1c381027f0 2024-03-08T23:06:33.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.133+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c2400fe10 con 0x7f1c381027f0 2024-03-08T23:06:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.133+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c381a7280 con 0x7f1c381027f0 2024-03-08T23:06:33.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.134+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c38068aa0 con 0x7f1c381027f0 2024-03-08T23:06:33.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.136+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1c2400f460 con 0x7f1c381027f0 2024-03-08T23:06:33.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.138+0000 7f1c34c00700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 0x7f1c1c0a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.138+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f1c240cfa30 con 0x7f1c381027f0 2024-03-08T23:06:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.138+0000 7f1c37e00700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 0x7f1c1c0a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:33.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.139+0000 7f1c37e00700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 0x7f1c1c0a4ef0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009c60 tx=0x7f1c28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:33.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.140+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1c24013070 con 0x7f1c381027f0 2024-03-08T23:06:33.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.369+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c38075960 con 0x7f1c1c0a2a30 2024-03-08T23:06:33.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.398+0000 7f1c34c00700 1 -- 172.21.15.46:0/4126129264 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340775 (secure 0 0 0) 0x7f1c38075960 con 0x7f1c1c0a2a30 2024-03-08T23:06:33.407 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 msgr2=0x7f1c1c0a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 0x7f1c1c0a4ef0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1c28009c60 tx=0x7f1c28009260 comp rx=0 tx=0).stop 2024-03-08T23:06:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 msgr2=0x7f1c3819ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1c24006010 tx=0x7f1c24003a90 comp rx=0 tx=0).stop 2024-03-08T23:06:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 shutdown_connections 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f1c1c0a2a30 0x7f1c1c0a4ef0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c38107e10 0x7f1c3819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c38101e30 0x7f1c3819a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 --2- 172.21.15.46:0/4126129264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c381027f0 0x7f1c3819ac40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 >> 172.21.15.46:0/4126129264 conn(0x7f1c38078f10 msgr2=0x7f1c38074240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.404+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 shutdown_connections 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:33.405+0000 7f1c3f000700 1 -- 172.21.15.46:0/4126129264 wait complete. 2024-03-08T23:06:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:06:34.177 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-03-08T23:06:33.224717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657597,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":674,"num_read_kb":463,"num_write":354,"num_write_kb":3062,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":787,"ondisk_log_size":787,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365464,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576360,"statfs":{"total":1151940427776,"available":1151566192640,"internally_reserved":0,"allocated":12443648,"data_stored":7957841,"data_compressed":361056,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006097"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197423+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:06:09.197423+0000","last_peered":"2024-03-08T23:06:09.197423+0000","last_clean":"2024-03-08T23:06:09.197423+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:06:09.197423+0000","last_undegraded":"2024-03-08T23:06:09.197423+0000","last_fullsized":"2024-03-08T23:06:09.197423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200014+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:06:09.200014+0000","last_peered":"2024-03-08T23:06:09.200014+0000","last_clean":"2024-03-08T23:06:09.200014+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:06:09.200014+0000","last_undegraded":"2024-03-08T23:06:09.200014+0000","last_fullsized":"2024-03-08T23:06:09.200014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200340+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:06:09.200340+0000","last_peered":"2024-03-08T23:06:09.200340+0000","last_clean":"2024-03-08T23:06:09.200340+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:06:09.200340+0000","last_undegraded":"2024-03-08T23:06:09.200340+0000","last_fullsized":"2024-03-08T23:06:09.200340+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.176978+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:06:09.176978+0000","last_peered":"2024-03-08T23:06:09.176978+0000","last_clean":"2024-03-08T23:06:09.176978+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:06:09.176978+0000","last_undegraded":"2024-03-08T23:06:09.176978+0000","last_fullsized":"2024-03-08T23:06:09.176978+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181136+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:06:09.181136+0000","last_peered":"2024-03-08T23:06:09.181136+0000","last_clean":"2024-03-08T23:06:09.181136+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:06:09.181136+0000","last_undegraded":"2024-03-08T23:06:09.181136+0000","last_fullsized":"2024-03-08T23:06:09.181136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185368+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:06:09.185368+0000","last_peered":"2024-03-08T23:06:09.185368+0000","last_clean":"2024-03-08T23:06:09.185368+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:06:09.185368+0000","last_undegraded":"2024-03-08T23:06:09.185368+0000","last_fullsized":"2024-03-08T23:06:09.185368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180418+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:06:09.180418+0000","last_peered":"2024-03-08T23:06:09.180418+0000","last_clean":"2024-03-08T23:06:09.180418+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:06:09.180418+0000","last_undegraded":"2024-03-08T23:06:09.180418+0000","last_fullsized":"2024-03-08T23:06:09.180418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.187226+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:06:09.187226+0000","last_peered":"2024-03-08T23:06:09.187226+0000","last_clean":"2024-03-08T23:06:09.187226+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:06:09.187226+0000","last_undegraded":"2024-03-08T23:06:09.187226+0000","last_fullsized":"2024-03-08T23:06:09.187226+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192440+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:06:09.192440+0000","last_peered":"2024-03-08T23:06:09.192440+0000","last_clean":"2024-03-08T23:06:09.192440+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:06:09.192440+0000","last_undegraded":"2024-03-08T23:06:09.192440+0000","last_fullsized":"2024-03-08T23:06:09.192440+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180400+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:06:09.180400+0000","last_peered":"2024-03-08T23:06:09.180400+0000","last_clean":"2024-03-08T23:06:09.180400+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:06:09.180400+0000","last_undegraded":"2024-03-08T23:06:09.180400+0000","last_fullsized":"2024-03-08T23:06:09.180400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191719+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:06:09.191719+0000","last_peered":"2024-03-08T23:06:09.191719+0000","last_clean":"2024-03-08T23:06:09.191719+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:06:09.191719+0000","last_undegraded":"2024-03-08T23:06:09.191719+0000","last_fullsized":"2024-03-08T23:06:09.191719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.183365+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:06:09.183365+0000","last_peered":"2024-03-08T23:06:09.183365+0000","last_clean":"2024-03-08T23:06:09.183365+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:06:09.183365+0000","last_undegraded":"2024-03-08T23:06:09.183365+0000","last_fullsized":"2024-03-08T23:06:09.183365+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191458+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:06:09.191458+0000","last_peered":"2024-03-08T23:06:09.191458+0000","last_clean":"2024-03-08T23:06:09.191458+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:06:09.191458+0000","last_undegraded":"2024-03-08T23:06:09.191458+0000","last_fullsized":"2024-03-08T23:06:09.191458+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197044+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:06:09.197044+0000","last_peered":"2024-03-08T23:06:09.197044+0000","last_clean":"2024-03-08T23:06:09.197044+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:06:09.197044+0000","last_undegraded":"2024-03-08T23:06:09.197044+0000","last_fullsized":"2024-03-08T23:06:09.197044+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.201322+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:06:09.201322+0000","last_peered":"2024-03-08T23:06:09.201322+0000","last_clean":"2024-03-08T23:06:09.201322+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:06:09.201322+0000","last_undegraded":"2024-03-08T23:06:09.201322+0000","last_fullsized":"2024-03-08T23:06:09.201322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190606+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:06:09.190606+0000","last_peered":"2024-03-08T23:06:09.190606+0000","last_clean":"2024-03-08T23:06:09.190606+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:06:09.190606+0000","last_undegraded":"2024-03-08T23:06:09.190606+0000","last_fullsized":"2024-03-08T23:06:09.190606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197573+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:06:09.197573+0000","last_peered":"2024-03-08T23:06:09.197573+0000","last_clean":"2024-03-08T23:06:09.197573+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:06:09.197573+0000","last_undegraded":"2024-03-08T23:06:09.197573+0000","last_fullsized":"2024-03-08T23:06:09.197573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192414+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:06:09.192414+0000","last_peered":"2024-03-08T23:06:09.192414+0000","last_clean":"2024-03-08T23:06:09.192414+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:06:09.192414+0000","last_undegraded":"2024-03-08T23:06:09.192414+0000","last_fullsized":"2024-03-08T23:06:09.192414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181120+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:06:09.181120+0000","last_peered":"2024-03-08T23:06:09.181120+0000","last_clean":"2024-03-08T23:06:09.181120+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:06:09.181120+0000","last_undegraded":"2024-03-08T23:06:09.181120+0000","last_fullsized":"2024-03-08T23:06:09.181120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.640979+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:05:40.640979+0000","last_peered":"2024-03-08T23:05:40.640979+0000","last_clean":"2024-03-08T23:05:40.640979+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:05:40.640979+0000","last_undegraded":"2024-03-08T23:05:40.640979+0000","last_fullsized":"2024-03-08T23:05:40.640979+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.656756+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:05:40.656756+0000","last_peered":"2024-03-08T23:05:40.656756+0000","last_clean":"2024-03-08T23:05:40.656756+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:05:40.656756+0000","last_undegraded":"2024-03-08T23:05:40.656756+0000","last_fullsized":"2024-03-08T23:05:40.656756+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.178888+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:06:09.178888+0000","last_peered":"2024-03-08T23:06:09.178888+0000","last_clean":"2024-03-08T23:06:09.178888+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:06:09.178888+0000","last_undegraded":"2024-03-08T23:06:09.178888+0000","last_fullsized":"2024-03-08T23:06:09.178888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643002+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:05:40.643002+0000","last_peered":"2024-03-08T23:05:40.643002+0000","last_clean":"2024-03-08T23:05:40.643002+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:05:40.643002+0000","last_undegraded":"2024-03-08T23:05:40.643002+0000","last_fullsized":"2024-03-08T23:05:40.643002+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.649676+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:05:40.649676+0000","last_peered":"2024-03-08T23:05:40.649676+0000","last_clean":"2024-03-08T23:05:40.649676+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:05:40.649676+0000","last_undegraded":"2024-03-08T23:05:40.649676+0000","last_fullsized":"2024-03-08T23:05:40.649676+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205981+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:06:09.205981+0000","last_peered":"2024-03-08T23:06:09.205981+0000","last_clean":"2024-03-08T23:06:09.205981+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:06:09.205981+0000","last_undegraded":"2024-03-08T23:06:09.205981+0000","last_fullsized":"2024-03-08T23:06:09.205981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643462+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:05:40.643462+0000","last_peered":"2024-03-08T23:05:40.643462+0000","last_clean":"2024-03-08T23:05:40.643462+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:05:40.643462+0000","last_undegraded":"2024-03-08T23:05:40.643462+0000","last_fullsized":"2024-03-08T23:05:40.643462+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.644175+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:05:40.644175+0000","last_peered":"2024-03-08T23:05:40.644175+0000","last_clean":"2024-03-08T23:05:40.644175+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:05:40.644175+0000","last_undegraded":"2024-03-08T23:05:40.644175+0000","last_fullsized":"2024-03-08T23:05:40.644175+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.188540+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:06:09.188540+0000","last_peered":"2024-03-08T23:06:09.188540+0000","last_clean":"2024-03-08T23:06:09.188540+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:06:09.188540+0000","last_undegraded":"2024-03-08T23:06:09.188540+0000","last_fullsized":"2024-03-08T23:06:09.188540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643010+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:05:40.643010+0000","last_peered":"2024-03-08T23:05:40.643010+0000","last_clean":"2024-03-08T23:05:40.643010+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:05:40.643010+0000","last_undegraded":"2024-03-08T23:05:40.643010+0000","last_fullsized":"2024-03-08T23:05:40.643010+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643666+0000","last_change":"2024-03-08T23:04:38.326247+0000","last_active":"2024-03-08T23:05:40.643666+0000","last_peered":"2024-03-08T23:05:40.643666+0000","last_clean":"2024-03-08T23:05:40.643666+0000","last_became_active":"2024-03-08T23:04:38.325843+0000","last_became_peered":"2024-03-08T23:04:38.325843+0000","last_unstale":"2024-03-08T23:05:40.643666+0000","last_undegraded":"2024-03-08T23:05:40.643666+0000","last_fullsized":"2024-03-08T23:05:40.643666+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:38:07.047189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190480+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:06:09.190480+0000","last_peered":"2024-03-08T23:06:09.190480+0000","last_clean":"2024-03-08T23:06:09.190480+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:06:09.190480+0000","last_undegraded":"2024-03-08T23:06:09.190480+0000","last_fullsized":"2024-03-08T23:06:09.190480+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641232+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:05:40.641232+0000","last_peered":"2024-03-08T23:05:40.641232+0000","last_clean":"2024-03-08T23:05:40.641232+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:05:40.641232+0000","last_undegraded":"2024-03-08T23:05:40.641232+0000","last_fullsized":"2024-03-08T23:05:40.641232+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":121,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:27.215457+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:06:27.215457+0000","last_peered":"2024-03-08T23:06:27.215457+0000","last_clean":"2024-03-08T23:06:27.215457+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:06:27.215457+0000","last_undegraded":"2024-03-08T23:06:27.215457+0000","last_fullsized":"2024-03-08T23:06:27.215457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.172087+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:06:09.172087+0000","last_peered":"2024-03-08T23:06:09.172087+0000","last_clean":"2024-03-08T23:06:09.172087+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:06:09.172087+0000","last_undegraded":"2024-03-08T23:06:09.172087+0000","last_fullsized":"2024-03-08T23:06:09.172087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641148+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:05:40.641148+0000","last_peered":"2024-03-08T23:05:40.641148+0000","last_clean":"2024-03-08T23:05:40.641148+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:05:40.641148+0000","last_undegraded":"2024-03-08T23:05:40.641148+0000","last_fullsized":"2024-03-08T23:05:40.641148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645888+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:05:40.645888+0000","last_peered":"2024-03-08T23:05:40.645888+0000","last_clean":"2024-03-08T23:05:40.645888+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:05:40.645888+0000","last_undegraded":"2024-03-08T23:05:40.645888+0000","last_fullsized":"2024-03-08T23:05:40.645888+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181806+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:06:09.181806+0000","last_peered":"2024-03-08T23:06:09.181806+0000","last_clean":"2024-03-08T23:06:09.181806+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:06:09.181806+0000","last_undegraded":"2024-03-08T23:06:09.181806+0000","last_fullsized":"2024-03-08T23:06:09.181806+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645369+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:05:40.645369+0000","last_peered":"2024-03-08T23:05:40.645369+0000","last_clean":"2024-03-08T23:05:40.645369+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:05:40.645369+0000","last_undegraded":"2024-03-08T23:05:40.645369+0000","last_fullsized":"2024-03-08T23:05:40.645369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.655646+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:05:40.655646+0000","last_peered":"2024-03-08T23:05:40.655646+0000","last_clean":"2024-03-08T23:05:40.655646+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:05:40.655646+0000","last_undegraded":"2024-03-08T23:05:40.655646+0000","last_fullsized":"2024-03-08T23:05:40.655646+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185980+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:06:09.185980+0000","last_peered":"2024-03-08T23:06:09.185980+0000","last_clean":"2024-03-08T23:06:09.185980+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:06:09.185980+0000","last_undegraded":"2024-03-08T23:06:09.185980+0000","last_fullsized":"2024-03-08T23:06:09.185980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.990386+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:05:40.990386+0000","last_peered":"2024-03-08T23:05:40.990386+0000","last_clean":"2024-03-08T23:05:40.990386+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:05:40.990386+0000","last_undegraded":"2024-03-08T23:05:40.990386+0000","last_fullsized":"2024-03-08T23:05:40.990386+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146494+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:05:41.146494+0000","last_peered":"2024-03-08T23:05:41.146494+0000","last_clean":"2024-03-08T23:05:41.146494+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:05:41.146494+0000","last_undegraded":"2024-03-08T23:05:41.146494+0000","last_fullsized":"2024-03-08T23:05:41.146494+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191415+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:06:09.191415+0000","last_peered":"2024-03-08T23:06:09.191415+0000","last_clean":"2024-03-08T23:06:09.191415+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:06:09.191415+0000","last_undegraded":"2024-03-08T23:06:09.191415+0000","last_fullsized":"2024-03-08T23:06:09.191415+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650588+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:05:40.650588+0000","last_peered":"2024-03-08T23:05:40.650588+0000","last_clean":"2024-03-08T23:05:40.650588+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:05:40.650588+0000","last_undegraded":"2024-03-08T23:05:40.650588+0000","last_fullsized":"2024-03-08T23:05:40.650588+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":115,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:28.216589+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:06:28.216589+0000","last_peered":"2024-03-08T23:06:28.216589+0000","last_clean":"2024-03-08T23:06:28.216589+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:06:28.216589+0000","last_undegraded":"2024-03-08T23:06:28.216589+0000","last_fullsized":"2024-03-08T23:06:28.216589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":84,"num_read_kb":84,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.190029+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:06:09.190029+0000","last_peered":"2024-03-08T23:06:09.190029+0000","last_clean":"2024-03-08T23:06:09.190029+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:06:09.190029+0000","last_undegraded":"2024-03-08T23:06:09.190029+0000","last_fullsized":"2024-03-08T23:06:09.190029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646899+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:05:40.646899+0000","last_peered":"2024-03-08T23:05:40.646899+0000","last_clean":"2024-03-08T23:05:40.646899+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:05:40.646899+0000","last_undegraded":"2024-03-08T23:05:40.646899+0000","last_fullsized":"2024-03-08T23:05:40.646899+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646199+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:05:40.646199+0000","last_peered":"2024-03-08T23:05:40.646199+0000","last_clean":"2024-03-08T23:05:40.646199+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:05:40.646199+0000","last_undegraded":"2024-03-08T23:05:40.646199+0000","last_fullsized":"2024-03-08T23:05:40.646199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200351+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:06:09.200351+0000","last_peered":"2024-03-08T23:06:09.200351+0000","last_clean":"2024-03-08T23:06:09.200351+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:06:09.200351+0000","last_undegraded":"2024-03-08T23:06:09.200351+0000","last_fullsized":"2024-03-08T23:06:09.200351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.884963+0000","last_change":"2024-03-08T23:05:40.884963+0000","last_active":"2024-03-08T23:05:40.884963+0000","last_peered":"2024-03-08T23:05:40.884963+0000","last_clean":"2024-03-08T23:05:40.884963+0000","last_became_active":"2024-03-08T23:05:40.884402+0000","last_became_peered":"2024-03-08T23:05:40.884402+0000","last_unstale":"2024-03-08T23:05:40.884963+0000","last_undegraded":"2024-03-08T23:05:40.884963+0000","last_fullsized":"2024-03-08T23:05:40.884963+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:26.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146436+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:05:41.146436+0000","last_peered":"2024-03-08T23:05:41.146436+0000","last_clean":"2024-03-08T23:05:41.146436+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:05:41.146436+0000","last_undegraded":"2024-03-08T23:05:41.146436+0000","last_fullsized":"2024-03-08T23:05:41.146436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185116+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:06:09.185116+0000","last_peered":"2024-03-08T23:06:09.185116+0000","last_clean":"2024-03-08T23:06:09.185116+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:06:09.185116+0000","last_undegraded":"2024-03-08T23:06:09.185116+0000","last_fullsized":"2024-03-08T23:06:09.185116+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.148144+0000","last_change":"2024-03-08T23:05:41.148144+0000","last_active":"2024-03-08T23:05:41.148144+0000","last_peered":"2024-03-08T23:05:41.148144+0000","last_clean":"2024-03-08T23:05:41.148144+0000","last_became_active":"2024-03-08T23:05:41.147760+0000","last_became_peered":"2024-03-08T23:05:41.147760+0000","last_unstale":"2024-03-08T23:05:41.148144+0000","last_undegraded":"2024-03-08T23:05:41.148144+0000","last_fullsized":"2024-03-08T23:05:41.148144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:00.345274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"91'34","reported_seq":239,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:29.218293+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:06:29.218293+0000","last_peered":"2024-03-08T23:06:29.218293+0000","last_clean":"2024-03-08T23:06:29.218293+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:06:29.218293+0000","last_undegraded":"2024-03-08T23:06:29.218293+0000","last_fullsized":"2024-03-08T23:06:29.218293+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62321,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":178,"num_read_kb":151,"num_write":35,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.171345+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:06:09.171345+0000","last_peered":"2024-03-08T23:06:09.171345+0000","last_clean":"2024-03-08T23:06:09.171345+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:06:09.171345+0000","last_undegraded":"2024-03-08T23:06:09.171345+0000","last_fullsized":"2024-03-08T23:06:09.171345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.659738+0000","last_change":"2024-03-08T23:05:40.659738+0000","last_active":"2024-03-08T23:05:40.659738+0000","last_peered":"2024-03-08T23:05:40.659738+0000","last_clean":"2024-03-08T23:05:40.659738+0000","last_became_active":"2024-03-08T23:05:40.659391+0000","last_became_peered":"2024-03-08T23:05:40.659391+0000","last_unstale":"2024-03-08T23:05:40.659738+0000","last_undegraded":"2024-03-08T23:05:40.659738+0000","last_fullsized":"2024-03-08T23:05:40.659738+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:42:03.247682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641425+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:05:40.641425+0000","last_peered":"2024-03-08T23:05:40.641425+0000","last_clean":"2024-03-08T23:05:40.641425+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:05:40.641425+0000","last_undegraded":"2024-03-08T23:05:40.641425+0000","last_fullsized":"2024-03-08T23:05:40.641425+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.183853+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:06:09.183853+0000","last_peered":"2024-03-08T23:06:09.183853+0000","last_clean":"2024-03-08T23:06:09.183853+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:06:09.183853+0000","last_undegraded":"2024-03-08T23:06:09.183853+0000","last_fullsized":"2024-03-08T23:06:09.183853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645193+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:05:40.645193+0000","last_peered":"2024-03-08T23:05:40.645193+0000","last_clean":"2024-03-08T23:05:40.645193+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:05:40.645193+0000","last_undegraded":"2024-03-08T23:05:40.645193+0000","last_fullsized":"2024-03-08T23:05:40.645193+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146818+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:05:41.146818+0000","last_peered":"2024-03-08T23:05:41.146818+0000","last_clean":"2024-03-08T23:05:41.146818+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:05:41.146818+0000","last_undegraded":"2024-03-08T23:05:41.146818+0000","last_fullsized":"2024-03-08T23:05:41.146818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.182986+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:06:09.182986+0000","last_peered":"2024-03-08T23:06:09.182986+0000","last_clean":"2024-03-08T23:06:09.182986+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:06:09.182986+0000","last_undegraded":"2024-03-08T23:06:09.182986+0000","last_fullsized":"2024-03-08T23:06:09.182986+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.656114+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:05:40.656114+0000","last_peered":"2024-03-08T23:05:40.656114+0000","last_clean":"2024-03-08T23:05:40.656114+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:05:40.656114+0000","last_undegraded":"2024-03-08T23:05:40.656114+0000","last_fullsized":"2024-03-08T23:05:40.656114+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":120,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:28.268135+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:06:28.268135+0000","last_peered":"2024-03-08T23:06:28.268135+0000","last_clean":"2024-03-08T23:06:28.268135+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:06:28.268135+0000","last_undegraded":"2024-03-08T23:06:28.268135+0000","last_fullsized":"2024-03-08T23:06:28.268135+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186482+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:06:09.186482+0000","last_peered":"2024-03-08T23:06:09.186482+0000","last_clean":"2024-03-08T23:06:09.186482+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:06:09.186482+0000","last_undegraded":"2024-03-08T23:06:09.186482+0000","last_fullsized":"2024-03-08T23:06:09.186482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.659109+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:05:40.659109+0000","last_peered":"2024-03-08T23:05:40.659109+0000","last_clean":"2024-03-08T23:05:40.659109+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:05:40.659109+0000","last_undegraded":"2024-03-08T23:05:40.659109+0000","last_fullsized":"2024-03-08T23:05:40.659109+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643248+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:05:40.643248+0000","last_peered":"2024-03-08T23:05:40.643248+0000","last_clean":"2024-03-08T23:05:40.643248+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:05:40.643248+0000","last_undegraded":"2024-03-08T23:05:40.643248+0000","last_fullsized":"2024-03-08T23:05:40.643248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.167956+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:06:09.167956+0000","last_peered":"2024-03-08T23:06:09.167956+0000","last_clean":"2024-03-08T23:06:09.167956+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:06:09.167956+0000","last_undegraded":"2024-03-08T23:06:09.167956+0000","last_fullsized":"2024-03-08T23:06:09.167956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146341+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:05:41.146341+0000","last_peered":"2024-03-08T23:05:41.146341+0000","last_clean":"2024-03-08T23:05:41.146341+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:05:41.146341+0000","last_undegraded":"2024-03-08T23:05:41.146341+0000","last_fullsized":"2024-03-08T23:05:41.146341+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643378+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:05:40.643378+0000","last_peered":"2024-03-08T23:05:40.643378+0000","last_clean":"2024-03-08T23:05:40.643378+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:05:40.643378+0000","last_undegraded":"2024-03-08T23:05:40.643378+0000","last_fullsized":"2024-03-08T23:05:40.643378+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.165175+0000","last_change":"2024-03-08T23:05:41.148194+0000","last_active":"2024-03-08T23:06:09.165175+0000","last_peered":"2024-03-08T23:06:09.165175+0000","last_clean":"2024-03-08T23:06:09.165175+0000","last_became_active":"2024-03-08T23:05:41.147832+0000","last_became_peered":"2024-03-08T23:05:41.147832+0000","last_unstale":"2024-03-08T23:06:09.165175+0000","last_undegraded":"2024-03-08T23:06:09.165175+0000","last_fullsized":"2024-03-08T23:06:09.165175+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:38.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643581+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:05:40.643581+0000","last_peered":"2024-03-08T23:05:40.643581+0000","last_clean":"2024-03-08T23:05:40.643581+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:05:40.643581+0000","last_undegraded":"2024-03-08T23:05:40.643581+0000","last_fullsized":"2024-03-08T23:05:40.643581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.653241+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:05:40.653241+0000","last_peered":"2024-03-08T23:05:40.653241+0000","last_clean":"2024-03-08T23:05:40.653241+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:05:40.653241+0000","last_undegraded":"2024-03-08T23:05:40.653241+0000","last_fullsized":"2024-03-08T23:05:40.653241+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185050+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:06:09.185050+0000","last_peered":"2024-03-08T23:06:09.185050+0000","last_clean":"2024-03-08T23:06:09.185050+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:06:09.185050+0000","last_undegraded":"2024-03-08T23:06:09.185050+0000","last_fullsized":"2024-03-08T23:06:09.185050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.658548+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:05:40.658548+0000","last_peered":"2024-03-08T23:05:40.658548+0000","last_clean":"2024-03-08T23:05:40.658548+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:05:40.658548+0000","last_undegraded":"2024-03-08T23:05:40.658548+0000","last_fullsized":"2024-03-08T23:05:40.658548+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646508+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:05:40.646508+0000","last_peered":"2024-03-08T23:05:40.646508+0000","last_clean":"2024-03-08T23:05:40.646508+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:05:40.646508+0000","last_undegraded":"2024-03-08T23:05:40.646508+0000","last_fullsized":"2024-03-08T23:05:40.646508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.204821+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:06:09.204821+0000","last_peered":"2024-03-08T23:06:09.204821+0000","last_clean":"2024-03-08T23:06:09.204821+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:06:09.204821+0000","last_undegraded":"2024-03-08T23:06:09.204821+0000","last_fullsized":"2024-03-08T23:06:09.204821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.658990+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:05:40.658990+0000","last_peered":"2024-03-08T23:05:40.658990+0000","last_clean":"2024-03-08T23:05:40.658990+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:05:40.658990+0000","last_undegraded":"2024-03-08T23:05:40.658990+0000","last_fullsized":"2024-03-08T23:05:40.658990+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.181912+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:06:09.181912+0000","last_peered":"2024-03-08T23:06:09.181912+0000","last_clean":"2024-03-08T23:06:09.181912+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:06:09.181912+0000","last_undegraded":"2024-03-08T23:06:09.181912+0000","last_fullsized":"2024-03-08T23:06:09.181912+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:31.274823+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:06:31.274823+0000","last_peered":"2024-03-08T23:06:31.274823+0000","last_clean":"2024-03-08T23:06:31.274823+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:06:31.274823+0000","last_undegraded":"2024-03-08T23:06:31.274823+0000","last_fullsized":"2024-03-08T23:06:31.274823+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.646715+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:05:40.646715+0000","last_peered":"2024-03-08T23:05:40.646715+0000","last_clean":"2024-03-08T23:05:40.646715+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:05:40.646715+0000","last_undegraded":"2024-03-08T23:05:40.646715+0000","last_fullsized":"2024-03-08T23:05:40.646715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186677+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:06:09.186677+0000","last_peered":"2024-03-08T23:06:09.186677+0000","last_clean":"2024-03-08T23:06:09.186677+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:06:09.186677+0000","last_undegraded":"2024-03-08T23:06:09.186677+0000","last_fullsized":"2024-03-08T23:06:09.186677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643539+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:05:40.643539+0000","last_peered":"2024-03-08T23:05:40.643539+0000","last_clean":"2024-03-08T23:05:40.643539+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:05:40.643539+0000","last_undegraded":"2024-03-08T23:05:40.643539+0000","last_fullsized":"2024-03-08T23:05:40.643539+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":363,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645314+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:05:40.645314+0000","last_peered":"2024-03-08T23:05:40.645314+0000","last_clean":"2024-03-08T23:05:40.645314+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:05:40.645314+0000","last_undegraded":"2024-03-08T23:05:40.645314+0000","last_fullsized":"2024-03-08T23:05:40.645314+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146553+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:05:41.146553+0000","last_peered":"2024-03-08T23:05:41.146553+0000","last_clean":"2024-03-08T23:05:41.146553+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:05:41.146553+0000","last_undegraded":"2024-03-08T23:05:41.146553+0000","last_fullsized":"2024-03-08T23:05:41.146553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.179062+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:06:09.179062+0000","last_peered":"2024-03-08T23:06:09.179062+0000","last_clean":"2024-03-08T23:06:09.179062+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:06:09.179062+0000","last_undegraded":"2024-03-08T23:06:09.179062+0000","last_fullsized":"2024-03-08T23:06:09.179062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645321+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:05:40.645321+0000","last_peered":"2024-03-08T23:05:40.645321+0000","last_clean":"2024-03-08T23:05:40.645321+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:05:40.645321+0000","last_undegraded":"2024-03-08T23:05:40.645321+0000","last_fullsized":"2024-03-08T23:05:40.645321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.645097+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:05:40.645097+0000","last_peered":"2024-03-08T23:05:40.645097+0000","last_clean":"2024-03-08T23:05:40.645097+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:05:40.645097+0000","last_undegraded":"2024-03-08T23:05:40.645097+0000","last_fullsized":"2024-03-08T23:05:40.645097+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643982+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:05:40.643982+0000","last_peered":"2024-03-08T23:05:40.643982+0000","last_clean":"2024-03-08T23:05:40.643982+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:05:40.643982+0000","last_undegraded":"2024-03-08T23:05:40.643982+0000","last_fullsized":"2024-03-08T23:05:40.643982+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.169973+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:06:09.169973+0000","last_peered":"2024-03-08T23:06:09.169973+0000","last_clean":"2024-03-08T23:06:09.169973+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:06:09.169973+0000","last_undegraded":"2024-03-08T23:06:09.169973+0000","last_fullsized":"2024-03-08T23:06:09.169973+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650040+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:05:40.650040+0000","last_peered":"2024-03-08T23:05:40.650040+0000","last_clean":"2024-03-08T23:05:40.650040+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:05:40.650040+0000","last_undegraded":"2024-03-08T23:05:40.650040+0000","last_fullsized":"2024-03-08T23:05:40.650040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'10","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.655131+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:05:40.655131+0000","last_peered":"2024-03-08T23:05:40.655131+0000","last_clean":"2024-03-08T23:05:40.655131+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:05:40.655131+0000","last_undegraded":"2024-03-08T23:05:40.655131+0000","last_fullsized":"2024-03-08T23:05:40.655131+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.180152+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:06:09.180152+0000","last_peered":"2024-03-08T23:06:09.180152+0000","last_clean":"2024-03-08T23:06:09.180152+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:06:09.180152+0000","last_undegraded":"2024-03-08T23:06:09.180152+0000","last_fullsized":"2024-03-08T23:06:09.180152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.640881+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:05:40.640881+0000","last_peered":"2024-03-08T23:05:40.640881+0000","last_clean":"2024-03-08T23:05:40.640881+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:05:40.640881+0000","last_undegraded":"2024-03-08T23:05:40.640881+0000","last_fullsized":"2024-03-08T23:05:40.640881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643332+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:05:40.643332+0000","last_peered":"2024-03-08T23:05:40.643332+0000","last_clean":"2024-03-08T23:05:40.643332+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:05:40.643332+0000","last_undegraded":"2024-03-08T23:05:40.643332+0000","last_fullsized":"2024-03-08T23:05:40.643332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205327+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:06:09.205327+0000","last_peered":"2024-03-08T23:06:09.205327+0000","last_clean":"2024-03-08T23:06:09.205327+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:06:09.205327+0000","last_undegraded":"2024-03-08T23:06:09.205327+0000","last_fullsized":"2024-03-08T23:06:09.205327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643692+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:05:40.643692+0000","last_peered":"2024-03-08T23:05:40.643692+0000","last_clean":"2024-03-08T23:05:40.643692+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:05:40.643692+0000","last_undegraded":"2024-03-08T23:05:40.643692+0000","last_fullsized":"2024-03-08T23:05:40.643692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.883136+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:05:40.883136+0000","last_peered":"2024-03-08T23:05:40.883136+0000","last_clean":"2024-03-08T23:05:40.883136+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:05:40.883136+0000","last_undegraded":"2024-03-08T23:05:40.883136+0000","last_fullsized":"2024-03-08T23:05:40.883136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.138622+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:06:09.138622+0000","last_peered":"2024-03-08T23:06:09.138622+0000","last_clean":"2024-03-08T23:06:09.138622+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:06:09.138622+0000","last_undegraded":"2024-03-08T23:06:09.138622+0000","last_fullsized":"2024-03-08T23:06:09.138622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.148869+0000","last_change":"2024-03-08T23:05:41.148196+0000","last_active":"2024-03-08T23:05:41.148869+0000","last_peered":"2024-03-08T23:05:41.148869+0000","last_clean":"2024-03-08T23:05:41.148869+0000","last_became_active":"2024-03-08T23:05:41.147730+0000","last_became_peered":"2024-03-08T23:05:41.147730+0000","last_unstale":"2024-03-08T23:05:41.148869+0000","last_undegraded":"2024-03-08T23:05:41.148869+0000","last_fullsized":"2024-03-08T23:05:41.148869+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:21:05.284970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.650718+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:05:40.650718+0000","last_peered":"2024-03-08T23:05:40.650718+0000","last_clean":"2024-03-08T23:05:40.650718+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:05:40.650718+0000","last_undegraded":"2024-03-08T23:05:40.650718+0000","last_fullsized":"2024-03-08T23:05:40.650718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.194371+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:06:09.194371+0000","last_peered":"2024-03-08T23:06:09.194371+0000","last_clean":"2024-03-08T23:06:09.194371+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:06:09.194371+0000","last_undegraded":"2024-03-08T23:06:09.194371+0000","last_fullsized":"2024-03-08T23:06:09.194371+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643863+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:05:40.643863+0000","last_peered":"2024-03-08T23:05:40.643863+0000","last_clean":"2024-03-08T23:05:40.643863+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:05:40.643863+0000","last_undegraded":"2024-03-08T23:05:40.643863+0000","last_fullsized":"2024-03-08T23:05:40.643863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641034+0000","last_change":"2024-03-08T23:04:38.331866+0000","last_active":"2024-03-08T23:05:40.641034+0000","last_peered":"2024-03-08T23:05:40.641034+0000","last_clean":"2024-03-08T23:05:40.641034+0000","last_became_active":"2024-03-08T23:04:38.331506+0000","last_became_peered":"2024-03-08T23:04:38.331506+0000","last_unstale":"2024-03-08T23:05:40.641034+0000","last_undegraded":"2024-03-08T23:05:40.641034+0000","last_fullsized":"2024-03-08T23:05:40.641034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:02:10.577391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.200743+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:06:09.200743+0000","last_peered":"2024-03-08T23:06:09.200743+0000","last_clean":"2024-03-08T23:06:09.200743+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:06:09.200743+0000","last_undegraded":"2024-03-08T23:06:09.200743+0000","last_fullsized":"2024-03-08T23:06:09.200743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643200+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:05:40.643200+0000","last_peered":"2024-03-08T23:05:40.643200+0000","last_clean":"2024-03-08T23:05:40.643200+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:05:40.643200+0000","last_undegraded":"2024-03-08T23:05:40.643200+0000","last_fullsized":"2024-03-08T23:05:40.643200+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.643297+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:05:40.643297+0000","last_peered":"2024-03-08T23:05:40.643297+0000","last_clean":"2024-03-08T23:05:40.643297+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:05:40.643297+0000","last_undegraded":"2024-03-08T23:05:40.643297+0000","last_fullsized":"2024-03-08T23:05:40.643297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191305+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:06:09.191305+0000","last_peered":"2024-03-08T23:06:09.191305+0000","last_clean":"2024-03-08T23:06:09.191305+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:06:09.191305+0000","last_undegraded":"2024-03-08T23:06:09.191305+0000","last_fullsized":"2024-03-08T23:06:09.191305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.883258+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:05:40.883258+0000","last_peered":"2024-03-08T23:05:40.883258+0000","last_clean":"2024-03-08T23:05:40.883258+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:05:40.883258+0000","last_undegraded":"2024-03-08T23:05:40.883258+0000","last_fullsized":"2024-03-08T23:05:40.883258+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.641438+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:05:40.641438+0000","last_peered":"2024-03-08T23:05:40.641438+0000","last_clean":"2024-03-08T23:05:40.641438+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:05:40.641438+0000","last_undegraded":"2024-03-08T23:05:40.641438+0000","last_fullsized":"2024-03-08T23:05:40.641438+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191852+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:06:09.191852+0000","last_peered":"2024-03-08T23:06:09.191852+0000","last_clean":"2024-03-08T23:06:09.191852+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:06:09.191852+0000","last_undegraded":"2024-03-08T23:06:09.191852+0000","last_fullsized":"2024-03-08T23:06:09.191852+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.989251+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:05:40.989251+0000","last_peered":"2024-03-08T23:05:40.989251+0000","last_clean":"2024-03-08T23:05:40.989251+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:05:40.989251+0000","last_undegraded":"2024-03-08T23:05:40.989251+0000","last_fullsized":"2024-03-08T23:05:40.989251+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146777+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:05:41.146777+0000","last_peered":"2024-03-08T23:05:41.146777+0000","last_clean":"2024-03-08T23:05:41.146777+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:05:41.146777+0000","last_undegraded":"2024-03-08T23:05:41.146777+0000","last_fullsized":"2024-03-08T23:05:41.146777+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.174007+0000","last_change":"2024-03-08T23:05:41.148243+0000","last_active":"2024-03-08T23:06:09.174007+0000","last_peered":"2024-03-08T23:06:09.174007+0000","last_clean":"2024-03-08T23:06:09.174007+0000","last_became_active":"2024-03-08T23:05:41.147924+0000","last_became_peered":"2024-03-08T23:05:41.147924+0000","last_unstale":"2024-03-08T23:06:09.174007+0000","last_undegraded":"2024-03-08T23:06:09.174007+0000","last_fullsized":"2024-03-08T23:06:09.174007+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:47.129092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:41.146018+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:05:41.146018+0000","last_peered":"2024-03-08T23:05:41.146018+0000","last_clean":"2024-03-08T23:05:41.146018+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:05:41.146018+0000","last_undegraded":"2024-03-08T23:05:41.146018+0000","last_fullsized":"2024-03-08T23:05:41.146018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:05:40.661217+0000","last_change":"2024-03-08T23:05:40.661217+0000","last_active":"2024-03-08T23:05:40.661217+0000","last_peered":"2024-03-08T23:05:40.661217+0000","last_clean":"2024-03-08T23:05:40.661217+0000","last_became_active":"2024-03-08T23:05:40.657165+0000","last_became_peered":"2024-03-08T23:05:40.657165+0000","last_unstale":"2024-03-08T23:05:40.661217+0000","last_undegraded":"2024-03-08T23:05:40.661217+0000","last_fullsized":"2024-03-08T23:05:40.661217+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:16.125758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.173872+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:06:09.173872+0000","last_peered":"2024-03-08T23:06:09.173872+0000","last_clean":"2024-03-08T23:06:09.173872+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:06:09.173872+0000","last_undegraded":"2024-03-08T23:06:09.173872+0000","last_fullsized":"2024-03-08T23:06:09.173872+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.171031+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:06:09.171031+0000","last_peered":"2024-03-08T23:06:09.171031+0000","last_clean":"2024-03-08T23:06:09.171031+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:06:09.171031+0000","last_undegraded":"2024-03-08T23:06:09.171031+0000","last_fullsized":"2024-03-08T23:06:09.171031+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":43,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.179063+0000","last_change":"2024-03-08T23:05:40.651674+0000","last_active":"2024-03-08T23:06:09.179063+0000","last_peered":"2024-03-08T23:06:09.179063+0000","last_clean":"2024-03-08T23:06:09.179063+0000","last_became_active":"2024-03-08T23:05:40.651393+0000","last_became_peered":"2024-03-08T23:05:40.651393+0000","last_unstale":"2024-03-08T23:06:09.179063+0000","last_undegraded":"2024-03-08T23:06:09.179063+0000","last_fullsized":"2024-03-08T23:06:09.179063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:30:44.691787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197159+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:06:09.197159+0000","last_peered":"2024-03-08T23:06:09.197159+0000","last_clean":"2024-03-08T23:06:09.197159+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:06:09.197159+0000","last_undegraded":"2024-03-08T23:06:09.197159+0000","last_fullsized":"2024-03-08T23:06:09.197159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.205178+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:06:09.205178+0000","last_peered":"2024-03-08T23:06:09.205178+0000","last_clean":"2024-03-08T23:06:09.205178+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:06:09.205178+0000","last_undegraded":"2024-03-08T23:06:09.205178+0000","last_fullsized":"2024-03-08T23:06:09.205178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.197326+0000","last_change":"2024-03-08T23:05:40.993334+0000","last_active":"2024-03-08T23:06:09.197326+0000","last_peered":"2024-03-08T23:06:09.197326+0000","last_clean":"2024-03-08T23:06:09.197326+0000","last_became_active":"2024-03-08T23:05:40.991755+0000","last_became_peered":"2024-03-08T23:05:40.991755+0000","last_unstale":"2024-03-08T23:06:09.197326+0000","last_undegraded":"2024-03-08T23:06:09.197326+0000","last_fullsized":"2024-03-08T23:06:09.197326+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:31.085388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.166651+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:06:09.166651+0000","last_peered":"2024-03-08T23:06:09.166651+0000","last_clean":"2024-03-08T23:06:09.166651+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:06:09.166651+0000","last_undegraded":"2024-03-08T23:06:09.166651+0000","last_fullsized":"2024-03-08T23:06:09.166651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.191349+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:06:09.191349+0000","last_peered":"2024-03-08T23:06:09.191349+0000","last_clean":"2024-03-08T23:06:09.191349+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:06:09.191349+0000","last_undegraded":"2024-03-08T23:06:09.191349+0000","last_fullsized":"2024-03-08T23:06:09.191349+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.182229+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:06:09.182229+0000","last_peered":"2024-03-08T23:06:09.182229+0000","last_clean":"2024-03-08T23:06:09.182229+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:06:09.182229+0000","last_undegraded":"2024-03-08T23:06:09.182229+0000","last_fullsized":"2024-03-08T23:06:09.182229+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.186392+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:06:09.186392+0000","last_peered":"2024-03-08T23:06:09.186392+0000","last_clean":"2024-03-08T23:06:09.186392+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:06:09.186392+0000","last_undegraded":"2024-03-08T23:06:09.186392+0000","last_fullsized":"2024-03-08T23:06:09.186392+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.177332+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:06:09.177332+0000","last_peered":"2024-03-08T23:06:09.177332+0000","last_clean":"2024-03-08T23:06:09.177332+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:06:09.177332+0000","last_undegraded":"2024-03-08T23:06:09.177332+0000","last_fullsized":"2024-03-08T23:06:09.177332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.192885+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:06:09.192885+0000","last_peered":"2024-03-08T23:06:09.192885+0000","last_clean":"2024-03-08T23:06:09.192885+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:06:09.192885+0000","last_undegraded":"2024-03-08T23:06:09.192885+0000","last_fullsized":"2024-03-08T23:06:09.192885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.207893+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:06:09.207893+0000","last_peered":"2024-03-08T23:06:09.207893+0000","last_clean":"2024-03-08T23:06:09.207893+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:06:09.207893+0000","last_undegraded":"2024-03-08T23:06:09.207893+0000","last_fullsized":"2024-03-08T23:06:09.207893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-08T23:06:09.185435+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:06:09.185435+0000","last_peered":"2024-03-08T23:06:09.185435+0000","last_clean":"2024-03-08T23:06:09.185435+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:06:09.185435+0000","last_undegraded":"2024-03-08T23:06:09.185435+0000","last_fullsized":"2024-03-08T23:06:09.185435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66995,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":538,"num_read_kb":344,"num_write":171,"num_write_kb":122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":200985,"data_compressed":5463,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":148,"ondisk_log_size":148,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383717,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":500760,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645353,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1150881,"data_compressed":58609,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808879,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":505092,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070518,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":500916,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332154,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1060864,"data_stored":1097498,"data_compressed":56788,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495681,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":496172,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":558403,"data_compressed":22402,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1015808,"data_stored":497560,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":560285,"data_compressed":22402,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496366,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920845,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":501354,"data_compressed":20581,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1092554,"data_compressed":56788,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":62321,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64203,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":64431,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:06:34.182 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-08T23:06:34.183 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-08T23:06:34.183 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-08T23:06:34.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-08T23:06:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:34 smithi046 ceph-mon[31859]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:34 smithi046 ceph-mon[31859]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:34 smithi192 ceph-mon[36700]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:34 smithi192 ceph-mon[36700]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:34.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:34 smithi062 ceph-mon[35761]: pgmap v378: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:34.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:34 smithi062 ceph-mon[35761]: from='client.25066 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:06:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:36.400 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:36 smithi046 ceph-mon[31859]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:06:36.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:36 smithi192 ceph-mon[36700]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:06:36.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:36 smithi062 ceph-mon[35761]: pgmap v379: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 -- 172.21.15.46:0/2225660788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 msgr2=0x7fa2a80743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 --2- 172.21.15.46:0/2225660788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a80743f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa298009990 tx=0x7fa298009ca0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 -- 172.21.15.46:0/2225660788 shutdown_connections 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 --2- 172.21.15.46:0/2225660788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a810e7e0 0x7fa2a8110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 --2- 172.21.15.46:0/2225660788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a80743f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 --2- 172.21.15.46:0/2225660788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a8074930 0x7fa2a810e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 -- 172.21.15.46:0/2225660788 >> 172.21.15.46:0/2225660788 conn(0x7fa2a8078f10 msgr2=0x7fa2a8079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 -- 172.21.15.46:0/2225660788 shutdown_connections 2024-03-08T23:06:36.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.932+0000 7fa2acc00700 1 -- 172.21.15.46:0/2225660788 wait complete. 2024-03-08T23:06:36.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.934+0000 7fa2acc00700 1 Processor -- start 2024-03-08T23:06:36.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.934+0000 7fa2acc00700 1 -- start start 2024-03-08T23:06:36.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:36.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8074930 0x7fa2a81a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a810e7e0 0x7fa2a819a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a81142e0 con 0x7fa2a810e7e0 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8114160 con 0x7fa2a8074010 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.935+0000 7fa2acc00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8114460 con 0x7fa2a8074930 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43166/0 (socket says 172.21.15.46:43166) 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 -- 172.21.15.46:0/553059334 learned_addr learned my addr 172.21.15.46:0/553059334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5400700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8074930 0x7fa2a81a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:36.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a6800700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a810e7e0 0x7fa2a819a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:36.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 -- 172.21.15.46:0/553059334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8074930 msgr2=0x7fa2a81a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:36.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8074930 0x7fa2a81a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 -- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a810e7e0 msgr2=0x7fa2a819a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a810e7e0 0x7fa2a819a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.936+0000 7fa2a5e00700 1 -- 172.21.15.46:0/553059334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a819ad00 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa2a5e00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa298005910 tx=0x7fa29800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa298027070 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa298004650 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa29800f460 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a819af90 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.937+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a81a72b0 con 0x7fa2a8074010 2024-03-08T23:06:36.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.938+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a8068aa0 con 0x7fa2a8074010 2024-03-08T23:06:36.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.941+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa29802a070 con 0x7fa2a8074010 2024-03-08T23:06:36.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.942+0000 7fa29ea00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 0x7fa28c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:36.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.942+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fa2980d0060 con 0x7fa2a8074010 2024-03-08T23:06:36.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.942+0000 7fa2a5400700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 0x7fa28c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:36.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.943+0000 7fa2a5400700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 0x7fa28c0a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa294009680 tx=0x7fa294009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:36.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:36.944+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa298031070 con 0x7fa2a8074010 2024-03-08T23:06:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.219+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa2a81a7480 con 0x7fa2a8074010 2024-03-08T23:06:37.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.220+0000 7fa29ea00700 1 -- 172.21.15.46:0/553059334 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa298013070 con 0x7fa2a8074010 2024-03-08T23:06:37.224 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:37.224 INFO:teuthology.orchestra.run.smithi046.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-08T23:06:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 msgr2=0x7fa28c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 0x7fa28c0a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa294009680 tx=0x7fa294009260 comp rx=0 tx=0).stop 2024-03-08T23:06:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 msgr2=0x7fa2a81a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa298005910 tx=0x7fa29800bab0 comp rx=0 tx=0).stop 2024-03-08T23:06:37.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 shutdown_connections 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa28c0a2910 0x7fa28c0a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2a810e7e0 0x7fa2a819a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2a8074010 0x7fa2a81a03e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 --2- 172.21.15.46:0/553059334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2a8074930 0x7fa2a81a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 >> 172.21.15.46:0/553059334 conn(0x7fa2a8078f10 msgr2=0x7fa2a80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.223+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 shutdown_connections 2024-03-08T23:06:37.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:37.224+0000 7fa2acc00700 1 -- 172.21.15.46:0/553059334 wait complete. 2024-03-08T23:06:37.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:37 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/553059334' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-08T23:06:37.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:37 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/553059334' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-08T23:06:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:37 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/553059334' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-08T23:06:37.971 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-08T23:06:37.971 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 -- 172.21.15.46:0/1284230830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 msgr2=0x7f01a41049c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 --2- 172.21.15.46:0/1284230830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a41049c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0190009900 tx=0x7f019002bc40 comp rx=0 tx=0).stop 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 -- 172.21.15.46:0/1284230830 shutdown_connections 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 --2- 172.21.15.46:0/1284230830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a410b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 --2- 172.21.15.46:0/1284230830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a41049c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 --2- 172.21.15.46:0/1284230830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 -- 172.21.15.46:0/1284230830 >> 172.21.15.46:0/1284230830 conn(0x7f01a40fe8c0 msgr2=0x7f01a4100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 -- 172.21.15.46:0/1284230830 shutdown_connections 2024-03-08T23:06:38.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.202+0000 7f01a3e00640 1 -- 172.21.15.46:0/1284230830 wait complete. 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 Processor -- start 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 -- start start 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a4199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a41a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.203+0000 7f01a3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a410efe0 con 0x7f01a4104540 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a410ee60 con 0x7f01a4103350 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a3e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a410f160 con 0x7f01a4104f00 2024-03-08T23:06:38.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43172/0 (socket says 172.21.15.46:43172) 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 -- 172.21.15.46:0/3358093849 learned_addr learned my addr 172.21.15.46:0/3358093849 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a3400640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a41a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a4199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 -- 172.21.15.46:0/3358093849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 msgr2=0x7f01a41a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a41a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 -- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 msgr2=0x7f01a4199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a4199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a2a00640 1 -- 172.21.15.46:0/3358093849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a41a1250 con 0x7f01a4103350 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.204+0000 7f01a3400640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a41a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f01a2a00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f019400e9c0 tx=0x7f019400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019400cd80 con 0x7f01a4103350 2024-03-08T23:06:38.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a41a1540 con 0x7f01a4103350 2024-03-08T23:06:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01940035f0 con 0x7f01a4103350 2024-03-08T23:06:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0194010690 con 0x7f01a4103350 2024-03-08T23:06:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.205+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a41a1ab0 con 0x7f01a4103350 2024-03-08T23:06:38.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.206+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a410a690 con 0x7f01a4103350 2024-03-08T23:06:38.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.208+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f01940037a0 con 0x7f01a4103350 2024-03-08T23:06:38.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.209+0000 7f019b400640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 0x7f01880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.210+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f01940cf1b0 con 0x7f01a4103350 2024-03-08T23:06:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.210+0000 7f01a2000640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 0x7f01880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.211+0000 7f01a2000640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 0x7f01880a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f019002c7c0 tx=0x7f0190005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:38.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.214+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f019409beb0 con 0x7f01a4103350 2024-03-08T23:06:38.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.422+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01a4061ef0 con 0x7f01a4103350 2024-03-08T23:06:38.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.425+0000 7f019b400640 1 -- 172.21.15.46:0/3358093849 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24545 (secure 0 0 0) 0x7f019409b7b0 con 0x7f01a4103350 2024-03-08T23:06:38.428 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:38.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":91,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:05:40.635177+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:06:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.427+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 msgr2=0x7f01880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.427+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 0x7f01880a4cc0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f019002c7c0 tx=0x7f0190005fd0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.427+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 msgr2=0x7f01a4199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.427+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f019400e9c0 tx=0x7f019400ee80 comp rx=0 tx=0).stop 2024-03-08T23:06:38.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 shutdown_connections 2024-03-08T23:06:38.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f01880a2800 0x7f01880a4cc0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01a4104f00 0x7f01a41a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01a4104540 0x7f01a4199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 --2- 172.21.15.46:0/3358093849 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01a4103350 0x7f01a4199630 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 >> 172.21.15.46:0/3358093849 conn(0x7f01a40fe8c0 msgr2=0x7f01a4107860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:38.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 shutdown_connections 2024-03-08T23:06:38.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.428+0000 7f01a3e00640 1 -- 172.21.15.46:0/3358093849 wait complete. 2024-03-08T23:06:38.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:06:38.453 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:06:38.534 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:38 smithi046 ceph-mon[31859]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:38 smithi192 ceph-mon[36700]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:38.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:38 smithi062 ceph-mon[35761]: pgmap v380: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:38.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 -- 172.21.15.46:0/1711463983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c1023e0 msgr2=0x7f877c102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 --2- 172.21.15.46:0/1711463983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c1023e0 0x7f877c102860 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f876c009900 tx=0x7f876c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 -- 172.21.15.46:0/1711463983 shutdown_connections 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 --2- 172.21.15.46:0/1711463983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c102da0 0x7f877c109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 --2- 172.21.15.46:0/1711463983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c1023e0 0x7f877c102860 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 --2- 172.21.15.46:0/1711463983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.676+0000 7f8783200640 1 -- 172.21.15.46:0/1711463983 >> 172.21.15.46:0/1711463983 conn(0x7f877c0fc760 msgr2=0x7f877c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.677+0000 7f8783200640 1 -- 172.21.15.46:0/1711463983 shutdown_connections 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.677+0000 7f8783200640 1 -- 172.21.15.46:0/1711463983 wait complete. 2024-03-08T23:06:38.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 Processor -- start 2024-03-08T23:06:38.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 -- start start 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c1023e0 0x7f877c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c102da0 0x7f877c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10ced0 con 0x7f877c1023e0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10cd50 con 0x7f877c1011f0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.678+0000 7f8783200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c10d050 con 0x7f877c102da0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c1023e0 0x7f877c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43204/0 (socket says 172.21.15.46:43204) 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 -- 172.21.15.46:0/3841202348 learned_addr learned my addr 172.21.15.46:0/3841202348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8782800640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c102da0 0x7f877c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 -- 172.21.15.46:0/3841202348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c102da0 msgr2=0x7f877c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c102da0 0x7f877c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 -- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c1023e0 msgr2=0x7f877c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c1023e0 0x7f877c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.679+0000 7f8781e00640 1 -- 172.21.15.46:0/3841202348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c0744b0 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8781e00640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f877400eb70 tx=0x7f877400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774003b70 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f877400ee70 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f877c1a1610 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8774010830 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.680+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f877c1a1ae0 con 0x7f877c1011f0 2024-03-08T23:06:38.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.681+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f877c068000 con 0x7f877c1011f0 2024-03-08T23:06:38.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.683+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8774004a40 con 0x7f877c1011f0 2024-03-08T23:06:38.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.684+0000 7f8772a00640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 0x7f87680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:38.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.685+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f87740cf4e0 con 0x7f877c1011f0 2024-03-08T23:06:38.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.685+0000 7f8781400640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 0x7f87680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:38.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.685+0000 7f8781400640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 0x7f87680a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f876c02c240 tx=0x7f876c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:38.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.687+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f877409c1e0 con 0x7f877c1011f0 2024-03-08T23:06:38.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.889+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f877c02d200 con 0x7f877c1011f0 2024-03-08T23:06:38.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.891+0000 7f8772a00640 1 -- 172.21.15.46:0/3841202348 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f877409bae0 con 0x7f877c1011f0 2024-03-08T23:06:38.894 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:06:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.893+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 msgr2=0x7f87680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.893+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 0x7f87680a4cc0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f876c02c240 tx=0x7f876c02df90 comp rx=0 tx=0).stop 2024-03-08T23:06:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.893+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 msgr2=0x7f877c0750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:38.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.893+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f877400eb70 tx=0x7f877400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 shutdown_connections 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f87680a2800 0x7f87680a4cc0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877c102da0 0x7f877c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f877c1023e0 0x7f877c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 --2- 172.21.15.46:0/3841202348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f877c1011f0 0x7f877c0750a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 >> 172.21.15.46:0/3841202348 conn(0x7f877c0fc760 msgr2=0x7f877c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 shutdown_connections 2024-03-08T23:06:38.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:38.894+0000 7f8783200640 1 -- 172.21.15.46:0/3841202348 wait complete. 2024-03-08T23:06:38.916 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:06:39.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 -- 172.21.15.46:0/2095040598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb41011d0 msgr2=0x7f0bb41015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 --2- 172.21.15.46:0/2095040598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb41011d0 0x7f0bb41015f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009be0 tx=0x7f0ba402bef0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 -- 172.21.15.46:0/2095040598 shutdown_connections 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 --2- 172.21.15.46:0/2095040598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb4102d80 0x7f0bb4109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 --2- 172.21.15.46:0/2095040598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 --2- 172.21.15.46:0/2095040598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb41011d0 0x7f0bb41015f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 -- 172.21.15.46:0/2095040598 >> 172.21.15.46:0/2095040598 conn(0x7f0bb40fc760 msgr2=0x7f0bb40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 -- 172.21.15.46:0/2095040598 shutdown_connections 2024-03-08T23:06:39.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.134+0000 7f0bb3200640 1 -- 172.21.15.46:0/2095040598 wait complete. 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.135+0000 7f0bb3200640 1 Processor -- start 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.135+0000 7f0bb3200640 1 -- start start 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb4102d80 0x7f0bb41a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410ceb0 con 0x7f0bb41023c0 2024-03-08T23:06:39.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410cd30 con 0x7f0bb41011d0 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb3200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb410d030 con 0x7f0bb4102d80 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43226/0 (socket says 172.21.15.46:43226) 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0babe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 -- 172.21.15.46:0/1185957382 learned_addr learned my addr 172.21.15.46:0/1185957382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb2800640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb4102d80 0x7f0bb41a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 -- 172.21.15.46:0/1185957382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb4102d80 msgr2=0x7f0bb41a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb4102d80 0x7f0bb41a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 -- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 msgr2=0x7f0bb4199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0bb1e00640 1 -- 172.21.15.46:0/1185957382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb41a1310 con 0x7f0bb41011d0 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.136+0000 7f0babe00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:06:39.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0bb1e00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c500 tx=0x7f0ba4033580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4043020 con 0x7f0bb41011d0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba4003e50 con 0x7f0bb41011d0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4034690 con 0x7f0bb41011d0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb41a15a0 con 0x7f0bb41011d0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.137+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41a1b40 con 0x7f0bb41011d0 2024-03-08T23:06:39.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.138+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb41a1e60 con 0x7f0bb41011d0 2024-03-08T23:06:39.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.140+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ba4046020 con 0x7f0bb41011d0 2024-03-08T23:06:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.142+0000 7f0baaa00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 0x7f0b980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.142+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f0ba40f19c0 con 0x7f0bb41011d0 2024-03-08T23:06:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.142+0000 7f0babe00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 0x7f0b980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.145 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.143+0000 7f0babe00640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 0x7f0b980a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0bb419ac00 tx=0x7f0b9c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:39.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.144+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0ba404d030 con 0x7f0bb41011d0 2024-03-08T23:06:39.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.350+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0bb41a2030 con 0x7f0bb41011d0 2024-03-08T23:06:39.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.351+0000 7f0baaa00640 1 -- 172.21.15.46:0/1185957382 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0ba40be6c0 con 0x7f0bb41011d0 2024-03-08T23:06:39.354 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.353+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 msgr2=0x7f0b980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.353+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 0x7f0b980a4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0bb419ac00 tx=0x7f0b9c009260 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.353+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 msgr2=0x7f0bb41996f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.353+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c500 tx=0x7f0ba4033580 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 shutdown_connections 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0b980a2880 0x7f0b980a4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0bb4102d80 0x7f0bb41a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0bb41023c0 0x7f0bb4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 --2- 172.21.15.46:0/1185957382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0bb41011d0 0x7f0bb41996f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 >> 172.21.15.46:0/1185957382 conn(0x7f0bb40fc760 msgr2=0x7f0bb4073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:39.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 shutdown_connections 2024-03-08T23:06:39.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.354+0000 7f0bb3200640 1 -- 172.21.15.46:0/1185957382 wait complete. 2024-03-08T23:06:39.377 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:06:39.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:39 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3358093849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:39.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:39 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3841202348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:39.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 -- 172.21.15.46:0/1637969168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900fefa0 msgr2=0x7f76900ff420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 --2- 172.21.15.46:0/1637969168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900fefa0 0x7f76900ff420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7678009860 tx=0x7f767802bb70 comp rx=0 tx=0).stop 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 -- 172.21.15.46:0/1637969168 shutdown_connections 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 --2- 172.21.15.46:0/1637969168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900ff960 0x7f76901015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 --2- 172.21.15.46:0/1637969168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900fefa0 0x7f76900ff420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 --2- 172.21.15.46:0/1637969168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 0x7f7690100970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 -- 172.21.15.46:0/1637969168 >> 172.21.15.46:0/1637969168 conn(0x7f76900faab0 msgr2=0x7f76900fcef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 -- 172.21.15.46:0/1637969168 shutdown_connections 2024-03-08T23:06:39.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.606+0000 7f7695600640 1 -- 172.21.15.46:0/1637969168 wait complete. 2024-03-08T23:06:39.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.607+0000 7f7695600640 1 Processor -- start 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 -- start start 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900ff960 0x7f7690199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 0x7f76901a0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769010f170 con 0x7f7690100550 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769010eff0 con 0x7f76900ff960 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.608+0000 7f7695600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769010f2f0 con 0x7f76900fefa0 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:42854/0 (socket says 172.21.15.46:42854) 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 -- 172.21.15.46:0/2521257276 learned_addr learned my addr 172.21.15.46:0/2521257276 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768f400640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 0x7f76901a0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f7687e00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900ff960 0x7f7690199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 -- 172.21.15.46:0/2521257276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900ff960 msgr2=0x7f7690199de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900ff960 0x7f7690199de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 -- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 msgr2=0x7f76901a0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 0x7f76901a0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 -- 172.21.15.46:0/2521257276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76901a14c0 con 0x7f76900fefa0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.609+0000 7f768ea00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f768000e9c0 tx=0x7f768000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.610+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768000cde0 con 0x7f76900fefa0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.610+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76800035f0 con 0x7f76900fefa0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.610+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680010690 con 0x7f76900fefa0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.610+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76901a17b0 con 0x7f76900fefa0 2024-03-08T23:06:39.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.610+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76901a1d20 con 0x7f76900fefa0 2024-03-08T23:06:39.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.611+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769010b890 con 0x7f76900fefa0 2024-03-08T23:06:39.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.614+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f76800037a0 con 0x7f76900fefa0 2024-03-08T23:06:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.615+0000 7f7687400640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 0x7f76740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:39.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.616+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7f76800cf0b0 con 0x7f76900fefa0 2024-03-08T23:06:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.616+0000 7f7687e00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 0x7f76740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:39.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.617+0000 7f7687e00640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 0x7f76740a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7678009830 tx=0x7f7678034ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:39.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.620+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f768009bdb0 con 0x7f76900fefa0 2024-03-08T23:06:39.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:39 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3358093849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:39.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:39 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3841202348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:39.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:39 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3358093849' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:39.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:39 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3841202348' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:39.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.822+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7690061ef0 con 0x7f76900fefa0 2024-03-08T23:06:39.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.823+0000 7f7687400640 1 -- 172.21.15.46:0/2521257276 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f76800bb070 con 0x7f76900fefa0 2024-03-08T23:06:39.826 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 msgr2=0x7f76740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 0x7f76740a4d40 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7678009830 tx=0x7f7678034ad0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 msgr2=0x7f76901998a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f768000e9c0 tx=0x7f768000ee80 comp rx=0 tx=0).stop 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 shutdown_connections 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f76740a2880 0x7f76740a4d40 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7690100550 0x7f76901a0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76900ff960 0x7f7690199de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 --2- 172.21.15.46:0/2521257276 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76900fefa0 0x7f76901998a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:39.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 >> 172.21.15.46:0/2521257276 conn(0x7f76900faab0 msgr2=0x7f76900fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:39.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 shutdown_connections 2024-03-08T23:06:39.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:39.826+0000 7f7695600640 1 -- 172.21.15.46:0/2521257276 wait complete. 2024-03-08T23:06:39.848 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:06:40.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 -- 172.21.15.46:0/651879167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc1024f0 msgr2=0x7fd5cc102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/651879167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc102970 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd5b4009860 tx=0x7fd5b402bb70 comp rx=0 tx=0).stop 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 -- 172.21.15.46:0/651879167 shutdown_connections 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/651879167 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc102eb0 0x7fd5cc109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/651879167 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc102970 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/651879167 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc101300 0x7fd5cc101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 -- 172.21.15.46:0/651879167 >> 172.21.15.46:0/651879167 conn(0x7fd5cc0fc890 msgr2=0x7fd5cc0fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 -- 172.21.15.46:0/651879167 shutdown_connections 2024-03-08T23:06:40.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.067+0000 7fd5d1e00640 1 -- 172.21.15.46:0/651879167 wait complete. 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 Processor -- start 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 -- start start 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 0x7fd5cc197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc102eb0 0x7fd5cc19ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc10cf90 con 0x7fd5cc102eb0 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc10ce10 con 0x7fd5cc1024f0 2024-03-08T23:06:40.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.069+0000 7fd5d1e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc10d110 con 0x7fd5cc101300 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5caa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 0x7fd5cc197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43258/0 (socket says 172.21.15.46:43258) 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 -- 172.21.15.46:0/933705191 learned_addr learned my addr 172.21.15.46:0/933705191 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5cb400640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc102eb0 0x7fd5cc19ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 -- 172.21.15.46:0/933705191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 msgr2=0x7fd5cc197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 0x7fd5cc197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 -- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc102eb0 msgr2=0x7fd5cc19ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc102eb0 0x7fd5cc19ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 -- 172.21.15.46:0/933705191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc19f240 con 0x7fd5cc1024f0 2024-03-08T23:06:40.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5caa00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 0x7fd5cc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.070+0000 7fd5c2000640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40349e0 tx=0x7fd5b40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.071+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4041030 con 0x7fd5cc1024f0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.071+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc19f4d0 con 0x7fd5cc1024f0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.071+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5b40347b0 con 0x7fd5cc1024f0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.071+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4039040 con 0x7fd5cc1024f0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.071+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc19fa10 con 0x7fd5cc1024f0 2024-03-08T23:06:40.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.072+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc1086a0 con 0x7fd5cc1024f0 2024-03-08T23:06:40.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.074+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd5b4044070 con 0x7fd5cc1024f0 2024-03-08T23:06:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.075+0000 7fd5d0a00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.076+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9071+0+0 (secure 0 0 0) 0x7fd5b40f0450 con 0x7fd5cc1024f0 2024-03-08T23:06:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.076+0000 7fd5caa00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.076+0000 7fd5caa00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc102350 tx=0x7fd5bc010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:40.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.080+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd5b40bd150 con 0x7fd5cc1024f0 2024-03-08T23:06:40.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.288+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd5cc061ef0 con 0x7fd5cc1024f0 2024-03-08T23:06:40.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.295+0000 7fd5d0a00640 1 -- 172.21.15.46:0/933705191 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd5b40bca50 con 0x7fd5cc1024f0 2024-03-08T23:06:40.298 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:06:40.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.298+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 msgr2=0x7fd5b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.298+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc102350 tx=0x7fd5bc010040 comp rx=0 tx=0).stop 2024-03-08T23:06:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.299+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 msgr2=0x7fd5cc197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.299+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40349e0 tx=0x7fd5b40042f0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 shutdown_connections 2024-03-08T23:06:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd5b00a2800 0x7fd5b00a4cc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd5cc102eb0 0x7fd5cc19ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd5cc1024f0 0x7fd5cc197b60 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 --2- 172.21.15.46:0/933705191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd5cc101300 0x7fd5cc197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.300+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 >> 172.21.15.46:0/933705191 conn(0x7fd5cc0fc890 msgr2=0x7fd5cc105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.301+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 shutdown_connections 2024-03-08T23:06:40.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.301+0000 7fd5d1e00640 1 -- 172.21.15.46:0/933705191 wait complete. 2024-03-08T23:06:40.322 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:06:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 -- 172.21.15.46:0/2382373750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e01044f0 msgr2=0x7f63e01068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 --2- 172.21.15.46:0/2382373750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e01044f0 0x7f63e01068e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f63cc009900 tx=0x7f63cc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 -- 172.21.15.46:0/2382373750 shutdown_connections 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 --2- 172.21.15.46:0/2382373750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e0106e20 0x7f63e0109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 --2- 172.21.15.46:0/2382373750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e01044f0 0x7f63e01068e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 --2- 172.21.15.46:0/2382373750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0103fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 -- 172.21.15.46:0/2382373750 >> 172.21.15.46:0/2382373750 conn(0x7f63e00fb5d0 msgr2=0x7f63e00fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 -- 172.21.15.46:0/2382373750 shutdown_connections 2024-03-08T23:06:40.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.520+0000 7f63e5600640 1 -- 172.21.15.46:0/2382373750 wait complete. 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 Processor -- start 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 -- start start 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e01044f0 0x7f63e0197980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e0106e20 0x7f63e019e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e010cda0 con 0x7f63e0106e20 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e010cc20 con 0x7f63e01044f0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.521+0000 7f63e5600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e010cf20 con 0x7f63e0101bc0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63e4c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e0106e20 0x7f63e019e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:42894/0 (socket says 172.21.15.46:42894) 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 -- 172.21.15.46:0/2306754916 learned_addr learned my addr 172.21.15.46:0/2306754916 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63df400640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e01044f0 0x7f63e0197980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 -- 172.21.15.46:0/2306754916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e01044f0 msgr2=0x7f63e0197980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e01044f0 0x7f63e0197980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 -- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e0106e20 msgr2=0x7f63e019e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e0106e20 0x7f63e019e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.522+0000 7f63dfe00640 1 -- 172.21.15.46:0/2306754916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e019f060 con 0x7f63e0101bc0 2024-03-08T23:06:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63dfe00640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f63d000e9f0 tx=0x7f63d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d000cd80 con 0x7f63e0101bc0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63d0010910 con 0x7f63e0101bc0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e019f350 con 0x7f63e0101bc0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0018990 con 0x7f63e0101bc0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.523+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e019f8c0 con 0x7f63e0101bc0 2024-03-08T23:06:40.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.524+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63e00fd0c0 con 0x7f63e0101bc0 2024-03-08T23:06:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.526+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f63d0010ac0 con 0x7f63e0101bc0 2024-03-08T23:06:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.527+0000 7f63dcc00640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 0x7f63c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.527+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f63d00cf0f0 con 0x7f63e0101bc0 2024-03-08T23:06:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.528+0000 7f63df400640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 0x7f63c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:40.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.528+0000 7f63df400640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 0x7f63c40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63cc000c00 tx=0x7f63cc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:40.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.531+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f63d009b760 con 0x7f63e0101bc0 2024-03-08T23:06:40.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:40 smithi192 ceph-mon[36700]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:40.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:40 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1185957382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-08T23:06:40.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:40 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-08T23:06:40.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:40 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2521257276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:40 smithi062 ceph-mon[35761]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:40.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:40 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1185957382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-08T23:06:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:40 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-08T23:06:40.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:40 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2521257276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:40 smithi046 ceph-mon[31859]: pgmap v381: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:06:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1185957382' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-08T23:06:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:40 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]: dispatch 2024-03-08T23:06:40.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2521257276' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:40.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.774+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63e002ce70 con 0x7f63e0101bc0 2024-03-08T23:06:40.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.775+0000 7f63dcc00640 1 -- 172.21.15.46:0/2306754916 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f63e002ce70 con 0x7f63e0101bc0 2024-03-08T23:06:40.778 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:40.778 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:06:40.779 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 msgr2=0x7f63c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 0x7f63c40a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f63cc000c00 tx=0x7f63cc005fd0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 msgr2=0x7f63e0197440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f63d000e9f0 tx=0x7f63d000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 shutdown_connections 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f63c40a2880 0x7f63c40a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f63e0106e20 0x7f63e019e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f63e01044f0 0x7f63e0197980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 --2- 172.21.15.46:0/2306754916 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63e0101bc0 0x7f63e0197440 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 >> 172.21.15.46:0/2306754916 conn(0x7f63e00fb5d0 msgr2=0x7f63e01025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:40.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.778+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 shutdown_connections 2024-03-08T23:06:40.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:40.779+0000 7f63e5600640 1 -- 172.21.15.46:0/2306754916 wait complete. 2024-03-08T23:06:40.802 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:06:41.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 -- 172.21.15.46:0/2689636820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4073cf0 msgr2=0x7efcb4074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 --2- 172.21.15.46:0/2689636820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4073cf0 0x7efcb4074150 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009900 tx=0x7efc9c02bcc0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 -- 172.21.15.46:0/2689636820 shutdown_connections 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 --2- 172.21.15.46:0/2689636820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb4108fe0 0x7efcb410b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 --2- 172.21.15.46:0/2689636820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4073cf0 0x7efcb4074150 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 --2- 172.21.15.46:0/2689636820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb40752d0 0x7efcb4073720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 -- 172.21.15.46:0/2689636820 >> 172.21.15.46:0/2689636820 conn(0x7efcb40fc6c0 msgr2=0x7efcb40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 -- 172.21.15.46:0/2689636820 shutdown_connections 2024-03-08T23:06:41.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.020+0000 7efcb9000640 1 -- 172.21.15.46:0/2689636820 wait complete. 2024-03-08T23:06:41.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.021+0000 7efcb9000640 1 Processor -- start 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.021+0000 7efcb9000640 1 -- start start 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb40752d0 0x7efcb4199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4108fe0 0x7efcb41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb410f030 con 0x7efcb4073cf0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb410eeb0 con 0x7efcb4108fe0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb9000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcb410f1b0 con 0x7efcb40752d0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60372/0 (socket says 172.21.15.46:60372) 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb3400640 1 -- 172.21.15.46:0/3547784827 learned_addr learned my addr 172.21.15.46:0/3547784827 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb2a00640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb40752d0 0x7efcb4199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.022+0000 7efcb3e00640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4108fe0 0x7efcb41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 -- 172.21.15.46:0/3547784827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb40752d0 msgr2=0x7efcb4199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb40752d0 0x7efcb4199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 -- 172.21.15.46:0/3547784827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4108fe0 msgr2=0x7efcb41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4108fe0 0x7efcb41a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 -- 172.21.15.46:0/3547784827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb41a1240 con 0x7efcb4073cf0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb3400640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efca400ebc0 tx=0x7efca400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4010630 con 0x7efcb4073cf0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca4010cb0 con 0x7efcb4073cf0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb41a1530 con 0x7efcb4073cf0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.023+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4018bd0 con 0x7efcb4073cf0 2024-03-08T23:06:41.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.024+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb41a1a50 con 0x7efcb4073cf0 2024-03-08T23:06:41.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.025+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcb4107da0 con 0x7efcb4073cf0 2024-03-08T23:06:41.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.027+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efca40107d0 con 0x7efcb4073cf0 2024-03-08T23:06:41.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.029+0000 7efcabe00640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 0x7efc980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.029+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9129+0+0 (secure 0 0 0) 0x7efca40cf420 con 0x7efcb4073cf0 2024-03-08T23:06:41.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.029+0000 7efcb2a00640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 0x7efc980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.030+0000 7efcb2a00640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 0x7efc980a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc9c02c2d0 tx=0x7efc9c02c910 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.034+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efca409ba90 con 0x7efcb4073cf0 2024-03-08T23:06:41.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.239+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efcb4061ef0 con 0x7efcb4073cf0 2024-03-08T23:06:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.242+0000 7efcabe00640 1 -- 172.21.15.46:0/3547784827 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+24696 (secure 0 0 0) 0x7efca409fb30 con 0x7efcb4073cf0 2024-03-08T23:06:41.246 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:41.246 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":92,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:40.285237+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:06:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 msgr2=0x7efc980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 0x7efc980a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efc9c02c2d0 tx=0x7efc9c02c910 comp rx=0 tx=0).stop 2024-03-08T23:06:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 msgr2=0x7efcb4199620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7efca400ebc0 tx=0x7efca400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 shutdown_connections 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efc980a2800 0x7efc980a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efcb4108fe0 0x7efcb41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efcb40752d0 0x7efcb4199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 --2- 172.21.15.46:0/3547784827 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efcb4073cf0 0x7efcb4199620 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 >> 172.21.15.46:0/3547784827 conn(0x7efcb40fc6c0 msgr2=0x7efcb4104f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 shutdown_connections 2024-03-08T23:06:41.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.245+0000 7efcb9000640 1 -- 172.21.15.46:0/3547784827 wait complete. 2024-03-08T23:06:41.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:06:41.268 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 -- 172.21.15.46:0/3061446598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1033f0 msgr2=0x7fd6ac103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3061446598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac103810 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009ca0 tx=0x7fd69c02c100 comp rx=0 tx=0).stop 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 -- 172.21.15.46:0/3061446598 shutdown_connections 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3061446598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 0x7fd6ac10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3061446598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3061446598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac103810 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 -- 172.21.15.46:0/3061446598 >> 172.21.15.46:0/3061446598 conn(0x7fd6ac0fe960 msgr2=0x7fd6ac100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:41.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 -- 172.21.15.46:0/3061446598 shutdown_connections 2024-03-08T23:06:41.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.484+0000 7fd6ab200640 1 -- 172.21.15.46:0/3061446598 wait complete. 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.485+0000 7fd6ab200640 1 Processor -- start 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.485+0000 7fd6ab200640 1 -- start start 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 0x7fd6ac1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac10f080 con 0x7fd6ac1033f0 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac10ef00 con 0x7fd6ac104fa0 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6ab200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ac10f200 con 0x7fd6ac1045e0 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6aa800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 0x7fd6ac1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60392/0 (socket says 172.21.15.46:60392) 2024-03-08T23:06:41.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a9e00640 1 -- 172.21.15.46:0/3239747837 learned_addr learned my addr 172.21.15.46:0/3239747837 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 -- 172.21.15.46:0/3239747837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 msgr2=0x7fd6ac1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 0x7fd6ac1a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 -- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 msgr2=0x7fd6ac1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac1996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.486+0000 7fd6a1400640 1 -- 172.21.15.46:0/3239747837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ac1a12f0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6a1400640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac199c10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd69400e9f0 tx=0x7fd69400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69400ccc0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd694010910 con 0x7fd6ac1045e0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694018960 con 0x7fd6ac1045e0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ac1a15e0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.487+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ac1a1ae0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.488+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ac10a710 con 0x7fd6ac1045e0 2024-03-08T23:06:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.491+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd694010ac0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.493+0000 7fd6a3400640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 0x7fd6900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.493+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fd6940cf150 con 0x7fd6ac1045e0 2024-03-08T23:06:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.493+0000 7fd6a9e00640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 0x7fd6900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.494+0000 7fd6a9e00640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 0x7fd6900a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd69c005e30 tx=0x7fd69c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.497+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd69409b7c0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-08T23:06:41.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-08T23:06:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/933705191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: osdmap e92: 12 total, 12 up, 12 in 2024-03-08T23:06:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2306754916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3547784827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:41 smithi192 ceph-mon[36700]: osdmap e93: 12 total, 12 up, 12 in 2024-03-08T23:06:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-08T23:06:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-08T23:06:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/933705191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:41.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: osdmap e92: 12 total, 12 up, 12 in 2024-03-08T23:06:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2306754916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3547784827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:41 smithi062 ceph-mon[35761]: osdmap e93: 12 total, 12 up, 12 in 2024-03-08T23:06:41.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.702+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd6ac061ef0 con 0x7fd6ac1045e0 2024-03-08T23:06:41.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.703+0000 7fd6a3400640 1 -- 172.21.15.46:0/3239747837 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd69409f860 con 0x7fd6ac1045e0 2024-03-08T23:06:41.706 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 msgr2=0x7fd6900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 0x7fd6900a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd69c005e30 tx=0x7fd69c004a40 comp rx=0 tx=0).stop 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 msgr2=0x7fd6ac199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac199c10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd69400e9f0 tx=0x7fd69400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 shutdown_connections 2024-03-08T23:06:41.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd6900a2880 0x7fd6900a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6ac104fa0 0x7fd6ac1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6ac1045e0 0x7fd6ac199c10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 --2- 172.21.15.46:0/3239747837 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6ac1033f0 0x7fd6ac1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 >> 172.21.15.46:0/3239747837 conn(0x7fd6ac0fe960 msgr2=0x7fd6ac1078e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 shutdown_connections 2024-03-08T23:06:41.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.706+0000 7fd6ab200640 1 -- 172.21.15.46:0/3239747837 wait complete. 2024-03-08T23:06:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-08T23:06:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [8, 11]}]': finished 2024-03-08T23:06:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/933705191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: osdmap e92: 12 total, 12 up, 12 in 2024-03-08T23:06:41.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2306754916' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3547784827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:41.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:41 smithi046 ceph-mon[31859]: osdmap e93: 12 total, 12 up, 12 in 2024-03-08T23:06:41.731 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:06:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 -- 172.21.15.46:0/1768816765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f8107810 msgr2=0x7fa0f8107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 --2- 172.21.15.46:0/1768816765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f8107810 0x7fa0f8107c30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8009ca0 tx=0x7fa0e802c100 comp rx=0 tx=0).stop 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 -- 172.21.15.46:0/1768816765 shutdown_connections 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 --2- 172.21.15.46:0/1768816765 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f81093c0 0x7fa0f810fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 --2- 172.21.15.46:0/1768816765 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8108a00 0x7fa0f8108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 --2- 172.21.15.46:0/1768816765 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f8107810 0x7fa0f8107c30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 -- 172.21.15.46:0/1768816765 >> 172.21.15.46:0/1768816765 conn(0x7fa0f80767c0 msgr2=0x7fa0f8078be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 -- 172.21.15.46:0/1768816765 shutdown_connections 2024-03-08T23:06:41.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.959+0000 7fa0f7200640 1 -- 172.21.15.46:0/1768816765 wait complete. 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.960+0000 7fa0f7200640 1 Processor -- start 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.960+0000 7fa0f7200640 1 -- start start 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 0x7fa0f819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f8108a00 0x7fa0f819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 0x7fa0f81a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f8113460 con 0x7fa0f81093c0 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f81132e0 con 0x7fa0f8108a00 2024-03-08T23:06:41.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f7200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f81135e0 con 0x7fa0f8107810 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 0x7fa0f81a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 0x7fa0f81a50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60404/0 (socket says 172.21.15.46:60404) 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 0x7fa0f819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f6800640 1 -- 172.21.15.46:0/2336826082 learned_addr learned my addr 172.21.15.46:0/2336826082 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5400640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f8108a00 0x7fa0f819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 -- 172.21.15.46:0/2336826082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f8108a00 msgr2=0x7fa0f819e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f8108a00 0x7fa0f819e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 -- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 msgr2=0x7fa0f81a50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 0x7fa0f81a50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.961+0000 7fa0f5e00640 1 -- 172.21.15.46:0/2336826082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f81a5750 con 0x7fa0f8107810 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0f5e00640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 0x7fa0f819db30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0e802c640 tx=0x7fa0e8035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8039040 con 0x7fa0f8107810 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0e8003970 con 0x7fa0f8107810 2024-03-08T23:06:41.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8004310 con 0x7fa0f8107810 2024-03-08T23:06:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f81a59e0 con 0x7fa0f8107810 2024-03-08T23:06:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.962+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f81a5e80 con 0x7fa0f8107810 2024-03-08T23:06:41.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.963+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f810eb30 con 0x7fa0f8107810 2024-03-08T23:06:41.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.965+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa0e8046030 con 0x7fa0f8107810 2024-03-08T23:06:41.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.967+0000 7fa0eea00640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 0x7fa0dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.967+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fa0e80f0eb0 con 0x7fa0f8107810 2024-03-08T23:06:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.967+0000 7fa0f5400640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 0x7fa0dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:41.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.968+0000 7fa0f5400640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 0x7fa0dc0a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0f819f040 tx=0x7fa0e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:41.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:41.969+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa0e804d060 con 0x7fa0f8107810 2024-03-08T23:06:42.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.171+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0f802d000 con 0x7fa0f8107810 2024-03-08T23:06:42.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.172+0000 7fa0eea00640 1 -- 172.21.15.46:0/2336826082 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa0e80bdc00 con 0x7fa0f8107810 2024-03-08T23:06:42.175 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:06:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.174+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 msgr2=0x7fa0dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.174+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 0x7fa0dc0a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa0f819f040 tx=0x7fa0e0005f90 comp rx=0 tx=0).stop 2024-03-08T23:06:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.174+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 msgr2=0x7fa0f819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.174+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 0x7fa0f819db30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0e802c640 tx=0x7fa0e8035d90 comp rx=0 tx=0).stop 2024-03-08T23:06:42.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 shutdown_connections 2024-03-08T23:06:42.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0dc0a2880 0x7fa0dc0a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0f81093c0 0x7fa0f81a50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0f8108a00 0x7fa0f819e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 --2- 172.21.15.46:0/2336826082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0f8107810 0x7fa0f819db30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 >> 172.21.15.46:0/2336826082 conn(0x7fa0f80767c0 msgr2=0x7fa0f810bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:42.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 shutdown_connections 2024-03-08T23:06:42.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.175+0000 7fa0f7200640 1 -- 172.21.15.46:0/2336826082 wait complete. 2024-03-08T23:06:42.198 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 -- 172.21.15.46:0/2768061171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510103510 msgr2=0x7f9510103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 --2- 172.21.15.46:0/2768061171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510103510 0x7f9510103930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9500009900 tx=0x7f950002bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 -- 172.21.15.46:0/2768061171 shutdown_connections 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 --2- 172.21.15.46:0/2768061171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 0x7f951010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 --2- 172.21.15.46:0/2768061171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510104700 0x7f9510104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 --2- 172.21.15.46:0/2768061171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510103510 0x7f9510103930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 -- 172.21.15.46:0/2768061171 >> 172.21.15.46:0/2768061171 conn(0x7f95100feaa0 msgr2=0x7f9510100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:42.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 -- 172.21.15.46:0/2768061171 shutdown_connections 2024-03-08T23:06:42.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.428+0000 7f9514c00640 1 -- 172.21.15.46:0/2768061171 wait complete. 2024-03-08T23:06:42.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 Processor -- start 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 -- start start 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 0x7f9510199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 0x7f95101a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951010f1f0 con 0x7f9510104700 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951010f070 con 0x7f95101050c0 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.429+0000 7f9514c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f951010f370 con 0x7f9510103510 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 0x7f9510199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:42994/0 (socket says 172.21.15.46:42994) 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 -- 172.21.15.46:0/3130613101 learned_addr learned my addr 172.21.15.46:0/3130613101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:42.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950e800640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 0x7f95101a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 -- 172.21.15.46:0/3130613101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 msgr2=0x7f95101a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 0x7f95101a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 -- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 msgr2=0x7f9510199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 0x7f9510199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 -- 172.21.15.46:0/3130613101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95101a1410 con 0x7f9510103510 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950d400640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 0x7f9510199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.430+0000 7f950de00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9500000c00 tx=0x7f950002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.431+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500043020 con 0x7f9510103510 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.431+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95000054e0 con 0x7f9510103510 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.431+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500003c40 con 0x7f9510103510 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.431+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95101a16a0 con 0x7f9510103510 2024-03-08T23:06:42.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.431+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a1ba0 con 0x7f9510103510 2024-03-08T23:06:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.432+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951010a830 con 0x7f9510103510 2024-03-08T23:06:42.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.434+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9500046020 con 0x7f9510103510 2024-03-08T23:06:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.435+0000 7f9506a00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 0x7f94f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.435+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f95000f0da0 con 0x7f9510103510 2024-03-08T23:06:42.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.436+0000 7f950d400640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 0x7f94f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.436+0000 7f950d400640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 0x7f94f40a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f951019ad00 tx=0x7f94fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:42.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.440+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f950004d020 con 0x7f9510103510 2024-03-08T23:06:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:42 smithi046 ceph-mon[31859]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:06:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:42 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3239747837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:42 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2336826082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:42 smithi192 ceph-mon[36700]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:06:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:42 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3239747837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:42 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2336826082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:42 smithi062 ceph-mon[35761]: pgmap v383: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:06:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:42 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3239747837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:42 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2336826082' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:06:42.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.640+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f951002d0b0 con 0x7f9510103510 2024-03-08T23:06:42.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.641+0000 7f9506a00640 1 -- 172.21.15.46:0/3130613101 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95000bda60 con 0x7f9510103510 2024-03-08T23:06:42.644 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 msgr2=0x7f94f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 0x7f94f40a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f951019ad00 tx=0x7f94fc005f90 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 msgr2=0x7f95101997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f9500000c00 tx=0x7f950002c650 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 shutdown_connections 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94f40a2880 0x7f94f40a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95101050c0 0x7f95101a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9510104700 0x7f9510199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 --2- 172.21.15.46:0/3130613101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9510103510 0x7f95101997f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 >> 172.21.15.46:0/3130613101 conn(0x7f95100feaa0 msgr2=0x7f9510107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:42.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 shutdown_connections 2024-03-08T23:06:42.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.644+0000 7f9514c00640 1 -- 172.21.15.46:0/3130613101 wait complete. 2024-03-08T23:06:42.665 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 -- 172.21.15.46:0/237679539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d41093c0 msgr2=0x7f03d410fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 --2- 172.21.15.46:0/237679539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d41093c0 0x7f03d410fc50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f03cc009900 tx=0x7f03cc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 -- 172.21.15.46:0/237679539 shutdown_connections 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 --2- 172.21.15.46:0/237679539 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d41093c0 0x7f03d410fc50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 --2- 172.21.15.46:0/237679539 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4108a00 0x7f03d4108e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 --2- 172.21.15.46:0/237679539 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d4107810 0x7f03d4107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 -- 172.21.15.46:0/237679539 >> 172.21.15.46:0/237679539 conn(0x7f03d40767c0 msgr2=0x7f03d4078be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 -- 172.21.15.46:0/237679539 shutdown_connections 2024-03-08T23:06:42.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.882+0000 7f03d8a00640 1 -- 172.21.15.46:0/237679539 wait complete. 2024-03-08T23:06:42.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 Processor -- start 2024-03-08T23:06:42.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 -- start start 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4107810 0x7f03d419da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 0x7f03d41a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4113490 con 0x7f03d41093c0 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4113310 con 0x7f03d4108a00 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.883+0000 7f03d8a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d4113610 con 0x7f03d4107810 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03d3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 0x7f03d41a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49466/0 (socket says 172.21.15.46:49466) 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 -- 172.21.15.46:0/143424976 learned_addr learned my addr 172.21.15.46:0/143424976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03d3400640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4107810 0x7f03d419da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 -- 172.21.15.46:0/143424976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4107810 msgr2=0x7f03d419da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4107810 0x7f03d419da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 -- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 msgr2=0x7f03d41a5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 0x7f03d41a5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03caa00640 1 -- 172.21.15.46:0/143424976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d41a5690 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.884+0000 7f03d3e00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 0x7f03d41a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03caa00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00e9f0 tx=0x7f03bc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc00cd80 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03bc005500 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d41a5980 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03bc010690 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.885+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d41a5ed0 con 0x7f03d4108a00 2024-03-08T23:06:42.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.886+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d410eb30 con 0x7f03d4108a00 2024-03-08T23:06:42.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.888+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f03bc0046c0 con 0x7f03d4108a00 2024-03-08T23:06:42.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.889+0000 7f03d0c00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 0x7f03b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:42.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.889+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f03bc0cf1a0 con 0x7f03d4108a00 2024-03-08T23:06:42.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.890+0000 7f03d3400640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 0x7f03b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:42.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.890+0000 7f03d3400640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 0x7f03b80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03d4108860 tx=0x7f03c4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:42.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:42.893+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f03bc09be60 con 0x7f03d4108a00 2024-03-08T23:06:43.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.087+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f03d4061ef0 con 0x7f03d4108a00 2024-03-08T23:06:43.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.088+0000 7f03d0c00640 1 -- 172.21.15.46:0/143424976 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f03bc09b760 con 0x7f03d4108a00 2024-03-08T23:06:43.091 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 msgr2=0x7f03b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 0x7f03b80a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f03d4108860 tx=0x7f03c4009500 comp rx=0 tx=0).stop 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 msgr2=0x7f03d419dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f03bc00e9f0 tx=0x7f03bc00edf0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 shutdown_connections 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f03b80a2880 0x7f03b80a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f03d41093c0 0x7f03d41a5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f03d4108a00 0x7f03d419dfb0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 --2- 172.21.15.46:0/143424976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03d4107810 0x7f03d419da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 >> 172.21.15.46:0/143424976 conn(0x7f03d40767c0 msgr2=0x7f03d410bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 shutdown_connections 2024-03-08T23:06:43.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.091+0000 7f03d8a00640 1 -- 172.21.15.46:0/143424976 wait complete. 2024-03-08T23:06:43.113 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:06:43.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 -- 172.21.15.46:0/3483191212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 msgr2=0x7f94641049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 --2- 172.21.15.46:0/3483191212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f94641049b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f944c009900 tx=0x7f944c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 -- 172.21.15.46:0/3483191212 shutdown_connections 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 --2- 172.21.15.46:0/3483191212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f946410b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 --2- 172.21.15.46:0/3483191212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f94641049b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 --2- 172.21.15.46:0/3483191212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f9464103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 -- 172.21.15.46:0/3483191212 >> 172.21.15.46:0/3483191212 conn(0x7f94640fe8d0 msgr2=0x7f9464100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.329+0000 7f9463200640 1 -- 172.21.15.46:0/3483191212 shutdown_connections 2024-03-08T23:06:43.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.330+0000 7f9463200640 1 -- 172.21.15.46:0/3483191212 wait complete. 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 Processor -- start 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 -- start start 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f94641995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f9464199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946410f020 con 0x7f9464104ef0 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946410eea0 con 0x7f9464103340 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9463200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946410f1a0 con 0x7f9464104530 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9462800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.331+0000 7f9462800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50286/0 (socket says 172.21.15.46:50286) 2024-03-08T23:06:43.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9459400640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f9464199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 -- 172.21.15.46:0/3498416075 learned_addr learned my addr 172.21.15.46:0/3498416075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9461e00640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f94641995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 -- 172.21.15.46:0/3498416075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 msgr2=0x7f9464199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f9464199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 -- 172.21.15.46:0/3498416075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 msgr2=0x7f94641995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f94641995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 -- 172.21.15.46:0/3498416075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94641a1210 con 0x7f9464104ef0 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9461e00640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f94641995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:06:43.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f9462800640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e970 tx=0x7f945c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.332+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00ccc0 con 0x7f9464104ef0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.333+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f945c00ce60 con 0x7f9464104ef0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.333+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c010490 con 0x7f9464104ef0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.333+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94641a1500 con 0x7f9464104ef0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.333+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94641a19d0 con 0x7f9464104ef0 2024-03-08T23:06:43.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.334+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946410a600 con 0x7f9464104ef0 2024-03-08T23:06:43.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.336+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f945c005020 con 0x7f9464104ef0 2024-03-08T23:06:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.337+0000 7f945b400640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.337+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f945c0cf370 con 0x7f9464104ef0 2024-03-08T23:06:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.337+0000 7f9461e00640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.338+0000 7f9461e00640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 0x7f94480a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9464104390 tx=0x7f9454009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:43.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.341+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f945c09c030 con 0x7f9464104ef0 2024-03-08T23:06:43.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:43 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3130613101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:43 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/143424976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.581+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f946402ce70 con 0x7f9464104ef0 2024-03-08T23:06:43.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.583+0000 7f945b400640 1 -- 172.21.15.46:0/3498416075 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f945c09b930 con 0x7f9464104ef0 2024-03-08T23:06:43.587 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:43.587 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:06:43.587 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 msgr2=0x7f94480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 0x7f94480a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9464104390 tx=0x7f9454009500 comp rx=0 tx=0).stop 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 msgr2=0x7f94641a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f945c00e970 tx=0x7f945c00ed70 comp rx=0 tx=0).stop 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 shutdown_connections 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f94480a2800 0x7f94480a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9464104ef0 0x7f94641a0ba0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9464104530 0x7f9464199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 --2- 172.21.15.46:0/3498416075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9464103340 0x7f94641995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.587+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 >> 172.21.15.46:0/3498416075 conn(0x7f94640fe8d0 msgr2=0x7f94641077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:43.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.588+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 shutdown_connections 2024-03-08T23:06:43.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.588+0000 7f9463200640 1 -- 172.21.15.46:0/3498416075 wait complete. 2024-03-08T23:06:43.609 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:06:43.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:43 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3130613101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:43 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/143424976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:43 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3130613101' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:43 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/143424976' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 -- 172.21.15.46:0/2806175813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c104ef0 msgr2=0x7fd28c10b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 --2- 172.21.15.46:0/2806175813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c104ef0 0x7fd28c10b780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd284009be0 tx=0x7fd28402bef0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 -- 172.21.15.46:0/2806175813 shutdown_connections 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 --2- 172.21.15.46:0/2806175813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c104ef0 0x7fd28c10b780 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 --2- 172.21.15.46:0/2806175813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104530 0x7fd28c1049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 --2- 172.21.15.46:0/2806175813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c103340 0x7fd28c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 -- 172.21.15.46:0/2806175813 >> 172.21.15.46:0/2806175813 conn(0x7fd28c0fe8d0 msgr2=0x7fd28c100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 -- 172.21.15.46:0/2806175813 shutdown_connections 2024-03-08T23:06:43.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.821+0000 7fd28be00640 1 -- 172.21.15.46:0/2806175813 wait complete. 2024-03-08T23:06:43.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.822+0000 7fd28be00640 1 Processor -- start 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.822+0000 7fd28be00640 1 -- start start 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.822+0000 7fd28be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c103340 0x7fd28c1995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.822+0000 7fd28be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 0x7fd28c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c10efa0 con 0x7fd28c104ef0 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c10ee20 con 0x7fd28c103340 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28be00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c10f120 con 0x7fd28c104530 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 0x7fd28c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33386/0 (socket says 172.21.15.46:33386) 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 -- 172.21.15.46:0/109432225 learned_addr learned my addr 172.21.15.46:0/109432225 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28aa00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c103340 0x7fd28c1995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 -- 172.21.15.46:0/109432225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c103340 msgr2=0x7fd28c1995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c103340 0x7fd28c1995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 -- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 msgr2=0x7fd28c1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 0x7fd28c1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28a000640 1 -- 172.21.15.46:0/109432225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c1a11d0 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.823+0000 7fd28b400640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 0x7fd28c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd28a000640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd27400e9c0 tx=0x7fd27400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27400cde0 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd2740035f0 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd274010690 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c1a14c0 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.824+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c1a1a10 con 0x7fd28c104530 2024-03-08T23:06:43.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.825+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c10a600 con 0x7fd28c104530 2024-03-08T23:06:43.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.827+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd2740037a0 con 0x7fd28c104530 2024-03-08T23:06:43.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.828+0000 7fd283400640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 0x7fd2700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:43.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.829+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fd2740cf240 con 0x7fd28c104530 2024-03-08T23:06:43.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.829+0000 7fd28aa00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 0x7fd2700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:43.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.829+0000 7fd28aa00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 0x7fd2700a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd28c104390 tx=0x7fd27c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:43.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:43.831+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd27409bf90 con 0x7fd28c104530 2024-03-08T23:06:44.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.026+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd28c02d040 con 0x7fd28c104530 2024-03-08T23:06:44.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.028+0000 7fd283400640 1 -- 172.21.15.46:0/109432225 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7fd274084e20 con 0x7fd28c104530 2024-03-08T23:06:44.031 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:44.032 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.030+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 msgr2=0x7fd2700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.030+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 0x7fd2700a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd28c104390 tx=0x7fd27c005f90 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.030+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 msgr2=0x7fd28c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.030+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd27400e9c0 tx=0x7fd27400ee80 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 shutdown_connections 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd2700a2800 0x7fd2700a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd28c104ef0 0x7fd28c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd28c104530 0x7fd28c199af0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 --2- 172.21.15.46:0/109432225 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd28c103340 0x7fd28c1995b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 >> 172.21.15.46:0/109432225 conn(0x7fd28c0fe8d0 msgr2=0x7fd28c1077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 shutdown_connections 2024-03-08T23:06:44.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.031+0000 7fd28be00640 1 -- 172.21.15.46:0/109432225 wait complete. 2024-03-08T23:06:44.052 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:06:44.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 -- 172.21.15.46:0/1700775155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc101bc0 msgr2=0x7f6bdc103fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 --2- 172.21.15.46:0/1700775155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc103fb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009860 tx=0x7f6bc402bb70 comp rx=0 tx=0).stop 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 -- 172.21.15.46:0/1700775155 shutdown_connections 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 --2- 172.21.15.46:0/1700775155 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc106e20 0x7f6bdc109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 --2- 172.21.15.46:0/1700775155 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc1044f0 0x7f6bdc1068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 --2- 172.21.15.46:0/1700775155 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc103fb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 -- 172.21.15.46:0/1700775155 >> 172.21.15.46:0/1700775155 conn(0x7f6bdc0fb5d0 msgr2=0x7f6bdc0fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.268+0000 7f6be1400640 1 -- 172.21.15.46:0/1700775155 shutdown_connections 2024-03-08T23:06:44.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.269+0000 7f6be1400640 1 -- 172.21.15.46:0/1700775155 wait complete. 2024-03-08T23:06:44.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 Processor -- start 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 -- start start 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc1044f0 0x7f6bdc1979e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc106e20 0x7f6bdc19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc10cdd0 con 0x7f6bdc1044f0 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc10cc50 con 0x7f6bdc106e20 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.270+0000 7f6be1400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bdc10cf50 con 0x7f6bdc101bc0 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc1044f0 0x7f6bdc1979e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33398/0 (socket says 172.21.15.46:33398) 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 -- 172.21.15.46:0/899858691 learned_addr learned my addr 172.21.15.46:0/899858691 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6be0a00640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc106e20 0x7f6bdc19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 -- 172.21.15.46:0/899858691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc106e20 msgr2=0x7f6bdc19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc106e20 0x7f6bdc19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 -- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc1044f0 msgr2=0x7f6bdc1979e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc1044f0 0x7f6bdc1979e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 -- 172.21.15.46:0/899858691 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc19f0c0 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.271+0000 7f6bdbe00640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40349e0 tx=0x7f6bc4032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.272+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4041030 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.272+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6bc402c510 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.272+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bdc19f350 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.272+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc4039040 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.272+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bdc19f890 con 0x7f6bdc101bc0 2024-03-08T23:06:44.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.273+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bdc0fd0c0 con 0x7f6bdc101bc0 2024-03-08T23:06:44.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.275+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6bc4044070 con 0x7f6bdc101bc0 2024-03-08T23:06:44.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.276+0000 7f6bd8c00640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 0x7f6bc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.276+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f6bc40f0ac0 con 0x7f6bdc101bc0 2024-03-08T23:06:44.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.277+0000 7f6bdb400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 0x7f6bc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.277+0000 7f6bdb400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 0x7f6bc00a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc1989b0 tx=0x7f6bcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.279+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6bc40bd800 con 0x7f6bdc101bc0 2024-03-08T23:06:44.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.525+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6bdc02ce70 con 0x7f6bdc101bc0 2024-03-08T23:06:44.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.527+0000 7f6bd8c00640 1 -- 172.21.15.46:0/899858691 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f6bc40bd100 con 0x7f6bdc101bc0 2024-03-08T23:06:44.530 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:06:44.530 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:06:44.530 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:06:44.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 msgr2=0x7f6bc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 0x7f6bc00a4d40 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc1989b0 tx=0x7f6bcc009260 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 msgr2=0x7f6bdc1974a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6bc40349e0 tx=0x7f6bc4032450 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 shutdown_connections 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6bc00a2880 0x7f6bc00a4d40 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6bdc106e20 0x7f6bdc19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6bdc1044f0 0x7f6bdc1979e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 --2- 172.21.15.46:0/899858691 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6bdc101bc0 0x7f6bdc1974a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 >> 172.21.15.46:0/899858691 conn(0x7f6bdc0fb5d0 msgr2=0x7f6bdc102520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.529+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 shutdown_connections 2024-03-08T23:06:44.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.530+0000 7f6be1400640 1 -- 172.21.15.46:0/899858691 wait complete. 2024-03-08T23:06:44.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-08T23:04:38.798478+0000', 'modified': '2024-03-08T23:05:06.080225+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24578, 'mds_1': 24560, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/3258548049', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 3258548049}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 3258548049}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.62:6837/2774760539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6836', 'nonce': 2774760539}, {'type': 'v1', 'addr': '172.21.15.62:6837', 'nonce': 2774760539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6837/3408320280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 3408320280}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 3408320280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.192:6839/441315746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 441315746}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 441315746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24560': {'gid': 24560, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.46:6841/2979513578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6840', 'nonce': 2979513578}, {'type': 'v1', 'addr': '172.21.15.46:6841', 'nonce': 2979513578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24578': {'gid': 24578, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6839/1938633866', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 1938633866}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 1938633866}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-08T23:06:44.552 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-08T23:06:44.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-08T23:06:44.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:44 smithi046 ceph-mon[31859]: pgmap v385: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:44.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:44 smithi046 ceph-mon[31859]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:06:44.630 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:44 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3498416075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.631 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:44 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/109432225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:44 smithi062 ceph-mon[35761]: pgmap v385: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:44 smithi062 ceph-mon[35761]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:06:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:44 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3498416075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:44 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/109432225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 -- 172.21.15.46:0/1002626795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0101c50 msgr2=0x7f67c0104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 --2- 172.21.15.46:0/1002626795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0101c50 0x7f67c0104040 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f67b0009900 tx=0x7f67b002bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:44.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 -- 172.21.15.46:0/1002626795 shutdown_connections 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 --2- 172.21.15.46:0/1002626795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0106eb0 0x7f67c0109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 --2- 172.21.15.46:0/1002626795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0104580 0x7f67c0106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 --2- 172.21.15.46:0/1002626795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0101c50 0x7f67c0104040 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 -- 172.21.15.46:0/1002626795 >> 172.21.15.46:0/1002626795 conn(0x7f67c00fb660 msgr2=0x7f67c00fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 -- 172.21.15.46:0/1002626795 shutdown_connections 2024-03-08T23:06:44.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.766+0000 7f67c4a00640 1 -- 172.21.15.46:0/1002626795 wait complete. 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.767+0000 7f67c4a00640 1 Processor -- start 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 -- start start 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0101c50 0x7f67c01015c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 0x7f67c00ffc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 0x7f67c0100140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010ceb0 con 0x7f67c0106eb0 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010cd30 con 0x7f67c0101c50 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67c4a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c010d030 con 0x7f67c0104580 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67bf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 0x7f67c0100140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67bde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 0x7f67c00ffc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67bde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 0x7f67c00ffc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33420/0 (socket says 172.21.15.46:33420) 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67bde00640 1 -- 172.21.15.46:0/4125950261 learned_addr learned my addr 172.21.15.46:0/4125950261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.768+0000 7f67be800640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0101c50 0x7f67c01015c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 -- 172.21.15.46:0/4125950261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 msgr2=0x7f67c00ffc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 0x7f67c00ffc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 -- 172.21.15.46:0/4125950261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0101c50 msgr2=0x7f67c01015c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0101c50 0x7f67c01015c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 -- 172.21.15.46:0/4125950261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c01009d0 con 0x7f67c0106eb0 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67bf200640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 0x7f67c0100140 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009fd0 tx=0x7f67b800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67b7400640 1 -- 172.21.15.46:0/4125950261 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b800ce20 con 0x7f67c0106eb0 2024-03-08T23:06:44.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c01a15f0 con 0x7f67c0106eb0 2024-03-08T23:06:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.769+0000 7f67b7400640 1 -- 172.21.15.46:0/4125950261 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67b8005020 con 0x7f67c0106eb0 2024-03-08T23:06:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.770+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c01a1b90 con 0x7f67c0106eb0 2024-03-08T23:06:44.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.770+0000 7f67b7400640 1 -- 172.21.15.46:0/4125950261 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b80106c0 con 0x7f67c0106eb0 2024-03-08T23:06:44.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.773+0000 7f67c81ca740 1 Processor -- start 2024-03-08T23:06:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67b7400640 1 -- 172.21.15.46:0/4125950261 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f67b8005540 con 0x7f67c0106eb0 2024-03-08T23:06:44.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 -- start start 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb53863c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb53866750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb537c36f0 con 0x55bb5385e760 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb5376c9e0 con 0x55bb538642c0 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.774+0000 7f67c81ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb537bf790 con 0x55bb5385f150 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50368/0 (socket says 172.21.15.46:50368) 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 -- 172.21.15.46:0/1275499700 learned_addr learned my addr 172.21.15.46:0/1275499700 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67be800640 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb53863c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb53866750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 -- 172.21.15.46:0/1275499700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 msgr2=0x55bb53863c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb53863c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 -- 172.21.15.46:0/1275499700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 msgr2=0x55bb53866750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb53866750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67bde00640 1 -- 172.21.15.46:0/1275499700 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb53760720 con 0x55bb5385e760 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.775+0000 7f67b7400640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 0x7f67a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67bde00640 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0136a0 tx=0x7f67ac009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61a24df846fcf16 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67b7400640 1 -- 172.21.15.46:0/4125950261 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f67b80cf500 con 0x7f67c0106eb0 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67be800640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 0x7f67a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67ab400640 1 -- 172.21.15.46:0/1275499700 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac004050 con 0x55bb5385e760 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67ab400640 1 -- 172.21.15.46:0/1275499700 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac0041f0 con 0x55bb5385e760 2024-03-08T23:06:44.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.776+0000 7f67ab400640 1 -- 172.21.15.46:0/1275499700 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac00f810 con 0x55bb5385e760 2024-03-08T23:06:44.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- 172.21.15.46:0/1275499700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 msgr2=0x55bb5385eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0136a0 tx=0x7f67ac009910 comp rx=0 tx=0).stop 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67be800640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 0x7f67a40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f67b002c240 tx=0x7f67b002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- 172.21.15.46:0/1275499700 shutdown_connections 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb53866750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb53863c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1275499700 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5385eb80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- 172.21.15.46:0/1275499700 >> 172.21.15.46:0/1275499700 conn(0x55bb53859cd0 msgr2=0x55bb5385a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- 172.21.15.46:0/1275499700 shutdown_connections 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- 172.21.15.46:0/1275499700 wait complete. 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.777+0000 7f67c81ca740 1 -- start start 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 Processor -- start 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb53868a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb5386ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53867270 con 0x55bb5385f150 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb5377fe20 con 0x55bb538642c0 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb538673f0 con 0x55bb5385e760 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67c81ca740 10 client.0 _ll_register_callbacks cb 0x55bb537ba8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67bde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67bde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50380/0 (socket says 172.21.15.46:50380) 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67bde00640 1 -- 172.21.15.46:0/2624471981 learned_addr learned my addr 172.21.15.46:0/2624471981 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.778+0000 7f67be800640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb53868a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bf200640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb5386ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 -- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 msgr2=0x55bb53868a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb53868a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 -- 172.21.15.46:0/2624471981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 msgr2=0x55bb5386ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb5386ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 -- 172.21.15.46:0/2624471981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb534d82d0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67be800640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb53868a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bde00640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005dd0 tx=0x7f67ac009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0b0547bce3d0b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f67bf200640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb5386ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f679fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.779+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac003f50 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f67ac0043a0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67ac0041f0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67ac00fb00 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 10 client.15045 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bb535a26b0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67ac0041f0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f67ac010ce0 con 0x55bb5385f150 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 10 client.15045 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:44.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 10 client.15045 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.780+0000 7f67c81ca740 20 client.15045 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.781+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb53875290 0x55bb538744a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.781+0000 7f67c81ca740 4 client.15045 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.781+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb537d0c30 con 0x55bb53875290 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.781+0000 7f67be800640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb53875290 0x55bb538744a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.782+0000 7f67be800640 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb53875290 0x55bb538744a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=190b5ad10cc18ec1 server_cookie=ed860eaac245428f in_seq=0 out_seq=0 2024-03-08T23:06:44.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.782+0000 7f679fe00640 10 client.15045 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.782+0000 7f679fe00640 1 -- 172.21.15.46:0/2624471981 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bb537d0c30 con 0x55bb53875290 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.782+0000 7f679fe00640 10 client.15045 handle_command_reply: tid=0 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.782+0000 7f67c81ca740 1 client.15045 shutdown 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 msgr2=0x55bb53868f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67ac005dd0 tx=0x7f67ac009d10 comp rx=0 tx=0).stop 2024-03-08T23:06:44.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 shutdown_connections 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb53875290 0x55bb538744a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb538642c0 0x55bb5386ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb53868f70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f679fe00640 0 client.15045 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2624471981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb53868a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 >> 172.21.15.46:0/2624471981 conn(0x55bb53859cd0 msgr2=0x55bb53858ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 shutdown_connections 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 1 -- 172.21.15.46:0/2624471981 wait complete. 2024-03-08T23:06:44.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.783+0000 7f67c81ca740 20 client.15045 trim_cache size 0 max 16384 2024-03-08T23:06:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.809+0000 7f67c81ca740 1 Processor -- start 2024-03-08T23:06:44.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.809+0000 7f67c81ca740 1 -- start start 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb53868020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb5386ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb538666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53874ab0 con 0x55bb5385f150 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53875110 con 0x55bb53859cd0 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67c81ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53875770 con 0x55bb5385e760 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67bf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb538666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67bf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb538666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50382/0 (socket says 172.21.15.46:50382) 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67bf200640 1 -- 172.21.15.46:0/1970270285 learned_addr learned my addr 172.21.15.46:0/1970270285 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67bde00640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb5386ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.810+0000 7f67be800640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb53868020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 -- 172.21.15.46:0/1970270285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 msgr2=0x55bb5386ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb5386ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 -- 172.21.15.46:0/1970270285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 msgr2=0x55bb538666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb538666b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 -- 172.21.15.46:0/1970270285 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb53874810 con 0x55bb53859cd0 2024-03-08T23:06:44.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67bde00640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb5386ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:44.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:44 smithi192 ceph-mon[36700]: pgmap v385: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:06:44.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:44 smithi192 ceph-mon[36700]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-08T23:06:44.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:44 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3498416075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:44 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/109432225' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67be800640 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb53868020 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f67b0036040 tx=0x7f67b0042000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc9a2dda0a0b3160 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67aaa00640 1 -- 172.21.15.46:0/1970270285 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b005c030 con 0x55bb53859cd0 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67aaa00640 1 -- 172.21.15.46:0/1970270285 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67b00046c0 con 0x55bb53859cd0 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.811+0000 7f67aaa00640 1 -- 172.21.15.46:0/1970270285 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b0046070 con 0x55bb53859cd0 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 -- 172.21.15.46:0/1970270285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 msgr2=0x55bb53868020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb53868020 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f67b0036040 tx=0x7f67b0042000 comp rx=0 tx=0).stop 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 -- 172.21.15.46:0/1970270285 shutdown_connections 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385f150 0x55bb538666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385e760 0x55bb5386ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 --2- 172.21.15.46:0/1970270285 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb53868020 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 -- 172.21.15.46:0/1970270285 >> 172.21.15.46:0/1970270285 conn(0x55bb53863dc0 msgr2=0x55bb53868640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 -- 172.21.15.46:0/1970270285 shutdown_connections 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.812+0000 7f67c81ca740 1 -- 172.21.15.46:0/1970270285 wait complete. 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 -- start start 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 Processor -- start 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb5386fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5386c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53876430 con 0x55bb5385e760 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53875dd0 con 0x55bb53859cd0 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb53876a90 con 0x55bb5385f150 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.813+0000 7f67c81ca740 10 client.0 _ll_register_callbacks cb 0x7f67ac009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:44.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5386c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33474/0 (socket says 172.21.15.46:33474) 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 -- 172.21.15.46:0/2045846954 learned_addr learned my addr 172.21.15.46:0/2045846954 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67be800640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb5386fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 -- 172.21.15.46:0/2045846954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 msgr2=0x55bb5386fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb5386fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 -- 172.21.15.46:0/2045846954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 msgr2=0x55bb5386c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5386c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 -- 172.21.15.46:0/2045846954 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb5386d360 con 0x55bb5385f150 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67be800640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb5386fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bde00640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5386c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67bf200640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f67b8098440 tx=0x7f67b809c290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbef608d519e25f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:44.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f679fe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b80b8030 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67b00046c0 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.814+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f67b8099dc0 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67b8082050 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 10 client.25174 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bb538d4730 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67b00046c0 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f67b80b4b80 con 0x55bb5385f150 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 10 client.25174 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 10 client.25174 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 20 client.25174 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb538642c0 0x55bb538d4a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 4 client.25174 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:06:44.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb537caf80 con 0x55bb538642c0 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.815+0000 7f67be800640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb538642c0 0x55bb538d4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67be800640 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb538642c0 0x55bb538d4a60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca84f12091d4b51e server_cookie=9bca149870f668ed in_seq=0 out_seq=0 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f679fe00640 10 client.25174 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f679fe00640 1 -- 172.21.15.46:0/2045846954 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55bb537caf80 con 0x55bb538642c0 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f679fe00640 10 client.25174 handle_command_reply: tid=0 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 client.25174 shutdown 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 msgr2=0x55bb5386cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f67b8098440 tx=0x7f67b809c290 comp rx=0 tx=0).stop 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 shutdown_connections 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb538642c0 0x55bb538d4a60 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb5385f150 0x55bb5386cd20 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb5385e760 0x55bb5386c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 --2- 172.21.15.46:0/2045846954 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb53859cd0 0x55bb5386fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 >> 172.21.15.46:0/2045846954 conn(0x55bb53863dc0 msgr2=0x55bb5385cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f679fe00640 0 client.25174 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.816+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 shutdown_connections 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c81ca740 1 -- 172.21.15.46:0/2045846954 wait complete. 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c81ca740 20 client.25174 trim_cache size 0 max 16384 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24553, 2024-03-08T23:06:44.825 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 99.744816424999996, 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 144.12916668599999 2024-03-08T23:06:44.826 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 msgr2=0x7f67a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 0x7f67a40a4cc0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f67b002c240 tx=0x7f67b002df90 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 msgr2=0x7f67c0100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 0x7f67c0100140 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f67b8009fd0 tx=0x7f67b800ef10 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 shutdown_connections 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f67a40a2800 0x7f67a40a4cc0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f67c0106eb0 0x7f67c0100140 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f67c0104580 0x7f67c00ffc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 --2- 172.21.15.46:0/4125950261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f67c0101c50 0x7f67c01015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.817+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 >> 172.21.15.46:0/4125950261 conn(0x7f67c00fb660 msgr2=0x7f67c01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.818+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 shutdown_connections 2024-03-08T23:06:44.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:44.818+0000 7f67c4a00640 1 -- 172.21.15.46:0/4125950261 wait complete. 2024-03-08T23:06:44.838 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-08T23:06:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 -- 172.21.15.46:0/4192348086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 msgr2=0x7f8ac4073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 --2- 172.21.15.46:0/4192348086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac4073780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009930 tx=0x7f8ab402bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 -- 172.21.15.46:0/4192348086 shutdown_connections 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 --2- 172.21.15.46:0/4192348086 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac410fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 --2- 172.21.15.46:0/4192348086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac40741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 --2- 172.21.15.46:0/4192348086 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac4073780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 -- 172.21.15.46:0/4192348086 >> 172.21.15.46:0/4192348086 conn(0x7f8ac40fc890 msgr2=0x7f8ac40fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 -- 172.21.15.46:0/4192348086 shutdown_connections 2024-03-08T23:06:45.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.026+0000 7f8acb600640 1 -- 172.21.15.46:0/4192348086 wait complete. 2024-03-08T23:06:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.027+0000 7f8acb600640 1 Processor -- start 2024-03-08T23:06:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.027+0000 7f8acb600640 1 -- start start 2024-03-08T23:06:45.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac419dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac419e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac4113570 con 0x7f8ac410d8d0 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41133f0 con 0x7f8ac4075330 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8acb600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac41136f0 con 0x7f8ac4073d50 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac419e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50404/0 (socket says 172.21.15.46:50404) 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 -- 172.21.15.46:0/966611422 learned_addr learned my addr 172.21.15.46:0/966611422 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac419dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 -- 172.21.15.46:0/966611422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 msgr2=0x7f8ac419dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac419dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 -- 172.21.15.46:0/966611422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 msgr2=0x7f8ac419e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac419e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac9600640 1 -- 172.21.15.46:0/966611422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac41a58b0 con 0x7f8ac410d8d0 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac419dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.028+0000 7f8abbe00640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8ac9600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00eb10 tx=0x7f8abc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8ac2000640 1 -- 172.21.15.46:0/966611422 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc004b20 con 0x7f8ac410d8d0 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac41a5ba0 con 0x7f8ac410d8d0 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8ac2000640 1 -- 172.21.15.46:0/966611422 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8abc004cc0 con 0x7f8ac410d8d0 2024-03-08T23:06:45.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8ac2000640 1 -- 172.21.15.46:0/966611422 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8abc010700 con 0x7f8ac410d8d0 2024-03-08T23:06:45.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.029+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac41a60c0 con 0x7f8ac410d8d0 2024-03-08T23:06:45.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.033+0000 7f8ace3c3740 1 Processor -- start 2024-03-08T23:06:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ac2000640 1 -- 172.21.15.46:0/966611422 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8abc005000 con 0x7f8ac410d8d0 2024-03-08T23:06:45.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 -- start start 2024-03-08T23:06:45.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c78580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3bd56f0 con 0x5611c3c70f80 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3b7e9e0 con 0x5611c3c70590 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3bd1790 con 0x5611c3c760f0 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.034+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c78580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50418/0 (socket says 172.21.15.46:50418) 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 -- 172.21.15.46:0/2100153936 learned_addr learned my addr 172.21.15.46:0/2100153936 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8abbe00640 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 -- 172.21.15.46:0/2100153936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 msgr2=0x5611c3c78580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c78580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 -- 172.21.15.46:0/2100153936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 msgr2=0x5611c3c709b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c709b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 -- 172.21.15.46:0/2100153936 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611c3b72720 con 0x5611c3c70f80 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac2000640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 0x7f8aa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.035+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009900 tx=0x7f8ab402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78e9fae42f0737d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.036+0000 7f8ac2000640 1 -- 172.21.15.46:0/966611422 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f8abc0cf540 con 0x7f8ac410d8d0 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.036+0000 7f8abbe00640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 0x7f8aa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.036+0000 7f8ab9c00640 1 -- 172.21.15.46:0/2100153936 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4005d50 con 0x5611c3c70f80 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.036+0000 7f8ab9c00640 1 -- 172.21.15.46:0/2100153936 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4005520 con 0x5611c3c70f80 2024-03-08T23:06:45.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.036+0000 7f8ab9c00640 1 -- 172.21.15.46:0/2100153936 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4004c60 con 0x5611c3c70f80 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8abbe00640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 0x7f8aa80a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5611c3c717e0 tx=0x7f8aac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 -- 172.21.15.46:0/2100153936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 msgr2=0x5611c3c75a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009900 tx=0x7f8ab402cd30 comp rx=0 tx=0).stop 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 -- 172.21.15.46:0/2100153936 shutdown_connections 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c78580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c75a80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/2100153936 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c709b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 -- 172.21.15.46:0/2100153936 >> 172.21.15.46:0/2100153936 conn(0x5611c3c6bb00 msgr2=0x5611c3c6bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 -- 172.21.15.46:0/2100153936 shutdown_connections 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.037+0000 7f8ace3c3740 1 -- 172.21.15.46:0/2100153936 wait complete. 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.038+0000 7f8ace3c3740 1 -- start start 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.038+0000 7f8ace3c3740 1 Processor -- start 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 0x5611c3c7a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70f80 0x5611c3c7ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c79040 con 0x5611c3c760f0 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3b91e20 con 0x5611c3c70f80 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c791c0 con 0x5611c3c70590 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ace3c3740 10 client.0 _ll_register_callbacks cb 0x5611c3bcc8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac9600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac9600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50430/0 (socket says 172.21.15.46:50430) 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 0x5611c3c7a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac9600640 1 -- 172.21.15.46:0/292846675 learned_addr learned my addr 172.21.15.46:0/292846675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8abbe00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70f80 0x5611c3c7ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 -- 172.21.15.46:0/292846675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70f80 msgr2=0x5611c3c7ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70f80 0x5611c3c7ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 -- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 msgr2=0x5611c3c81c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac8c00640 1 -- 172.21.15.46:0/292846675 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611c38ea2d0 con 0x5611c3c70590 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.039+0000 7f8ac9600640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 0x5611c3c7a710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5611c3c6ca70 tx=0x7f8ab4003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=365862fc70348d88 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ab2000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4005d50 con 0x5611c3c70590 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f8ab4003e20 con 0x5611c3c70590 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ab4005520 con 0x5611c3c70590 2024-03-08T23:06:45.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ab40044d0 con 0x5611c3c70590 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ace3c3740 10 client.25186 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.040+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5611c39b46b0 con 0x5611c3c70590 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4005520 con 0x5611c3c70590 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ab404e450 con 0x5611c3c70590 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 10 client.25186 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 10 client.25186 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 20 client.25186 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3c86180 0x5611c3c8b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 4 client.25186 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611c3be2c30 con 0x5611c3c86180 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.041+0000 7f8abbe00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3c86180 0x5611c3c8b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.042+0000 7f8abbe00640 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3c86180 0x5611c3c8b1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=99673f092d1c7e25 server_cookie=ac3df2617cf2b28d in_seq=0 out_seq=0 2024-03-08T23:06:45.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.042+0000 7f8ab2000640 10 client.25186 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ab2000640 1 -- 172.21.15.46:0/292846675 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5611c3be2c30 con 0x5611c3c86180 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ab2000640 10 client.25186 handle_command_reply: tid=0 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 client.25186 shutdown 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 msgr2=0x5611c3c7a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 0x5611c3c7a710 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x5611c3c6ca70 tx=0x7f8ab4003820 comp rx=0 tx=0).stop 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 shutdown_connections 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c760f0 0x5611c3c81c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70f80 0x5611c3c7ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3c86180 0x5611c3c8b1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/292846675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70590 0x5611c3c7a710 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 >> 172.21.15.46:0/292846675 conn(0x5611c3c6bb00 msgr2=0x5611c3c6ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ab2000640 0 client.25186 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 shutdown_connections 2024-03-08T23:06:45.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.043+0000 7f8ace3c3740 1 -- 172.21.15.46:0/292846675 wait complete. 2024-03-08T23:06:45.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.044+0000 7f8ace3c3740 20 client.25186 trim_cache size 0 max 16384 2024-03-08T23:06:45.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 Processor -- start 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 -- start start 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70f80 0x5611c3c81bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c760f0 0x5611c3c8f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c86570 con 0x5611c3c70590 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c86bd0 con 0x5611c3c760f0 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.070+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c87230 con 0x5611c3c70f80 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50438/0 (socket says 172.21.15.46:50438) 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 -- 172.21.15.46:0/3663410692 learned_addr learned my addr 172.21.15.46:0/3663410692 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac9600640 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c760f0 0x5611c3c8f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8abbe00640 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70f80 0x5611c3c81bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 -- 172.21.15.46:0/3663410692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70f80 msgr2=0x5611c3c81bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70f80 0x5611c3c81bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 -- 172.21.15.46:0/3663410692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c760f0 msgr2=0x5611c3c8f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c760f0 0x5611c3c8f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 -- 172.21.15.46:0/3663410692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611c3c6c870 con 0x5611c3c70590 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.071+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ab402c170 tx=0x7f8ab402cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13073b4c33a7863 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.072+0000 7f8ab9200640 1 -- 172.21.15.46:0/3663410692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab40044d0 con 0x5611c3c70590 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.072+0000 7f8ab9200640 1 -- 172.21.15.46:0/3663410692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4009520 con 0x5611c3c70590 2024-03-08T23:06:45.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.072+0000 7f8ab9200640 1 -- 172.21.15.46:0/3663410692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab4003e40 con 0x5611c3c70590 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.072+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3663410692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 msgr2=0x5611c3c668b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.072+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8ab402c170 tx=0x7f8ab402cd30 comp rx=0 tx=0).stop 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3663410692 shutdown_connections 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c760f0 0x5611c3c8f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c70f80 0x5611c3c81bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3663410692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70590 0x5611c3c668b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3663410692 >> 172.21.15.46:0/3663410692 conn(0x5611c3c6bb00 msgr2=0x5611c3c6ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3663410692 shutdown_connections 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3663410692 wait complete. 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 -- start start 2024-03-08T23:06:45.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.073+0000 7f8ace3c3740 1 Processor -- start 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c7e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c87ef0 con 0x5611c3c70f80 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c87890 con 0x5611c3c70590 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611c3c88550 con 0x5611c3c760f0 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ace3c3740 10 client.0 _ll_register_callbacks cb 0x7f8aac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac8c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49644/0 (socket says 172.21.15.46:49644) 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50444/0 (socket says 172.21.15.46:50444) 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac8c00640 1 -- 172.21.15.46:0/3762335926 learned_addr learned my addr 172.21.15.46:0/3762335926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac9600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c7e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 -- 172.21.15.46:0/3762335926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 msgr2=0x5611c3c7e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c7e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 -- 172.21.15.46:0/3762335926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 msgr2=0x5611c3c817e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8abbe00640 1 -- 172.21.15.46:0/3762335926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611c3c79390 con 0x5611c3c70f80 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac9600640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c7e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.074+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8abbe00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5611c3c88da0 tx=0x7f8aac005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a24b46916eea04a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ab2000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aac033070 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ab4009520 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f8aac005020 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8aac03f050 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ace3c3740 10 client.15069 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.075+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5611c3cddb90 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ab4009520 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8aac02f050 con 0x5611c3c70f80 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 10 client.15069 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 10 client.15069 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 20 client.15069 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:45.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3ce2220 0x5611c3cdde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 4 client.15069 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5611c3bdcf80 con 0x5611c3ce2220 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.076+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3ce2220 0x5611c3cdde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ac8c00640 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3ce2220 0x5611c3cdde80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ac909a6e60d5eda4 server_cookie=76ecf574a21b5368 in_seq=0 out_seq=0 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ab2000640 10 client.15069 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ab2000640 1 -- 172.21.15.46:0/3762335926 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5611c3bdcf80 con 0x5611c3ce2220 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ab2000640 10 client.15069 handle_command_reply: tid=0 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 client.15069 shutdown 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 msgr2=0x5611c3c81d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x5611c3c88da0 tx=0x7f8aac005850 comp rx=0 tx=0).stop 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 shutdown_connections 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5611c3ce2220 0x5611c3cdde80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5611c3c760f0 0x5611c3c7e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5611c3c70f80 0x5611c3c81d20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 --2- 172.21.15.46:0/3762335926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5611c3c70590 0x5611c3c817e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ab2000640 0 client.15069 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.077+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 >> 172.21.15.46:0/3762335926 conn(0x5611c3c6bb00 msgr2=0x5611c3c6ac40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 shutdown_connections 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8ace3c3740 1 -- 172.21.15.46:0/3762335926 wait complete. 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8ace3c3740 20 client.15069 trim_cache size 0 max 16384 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:06:45.085 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24560, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 101.01566323500001, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 136.26052578400001 2024-03-08T23:06:45.086 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 msgr2=0x7f8aa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 0x7f8aa80a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x5611c3c717e0 tx=0x7f8aac005f90 comp rx=0 tx=0).stop 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 msgr2=0x7f8ac41a5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8abc00eb10 tx=0x7f8abc00c2d0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 shutdown_connections 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8aa80a2800 0x7f8aa80a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8ac410d8d0 0x7f8ac41a5210 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8ac4075330 0x7f8ac419e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 --2- 172.21.15.46:0/966611422 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8ac4073d50 0x7f8ac419dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.078+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 >> 172.21.15.46:0/966611422 conn(0x7f8ac40fc890 msgr2=0x7f8ac4100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.079+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 shutdown_connections 2024-03-08T23:06:45.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.079+0000 7f8acb600640 1 -- 172.21.15.46:0/966611422 wait complete. 2024-03-08T23:06:45.098 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 -- 172.21.15.46:0/1784042452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc103530 msgr2=0x7f38bc103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 --2- 172.21.15.46:0/1784042452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc103530 0x7f38bc103950 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009900 tx=0x7f38ac02bc30 comp rx=0 tx=0).stop 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 -- 172.21.15.46:0/1784042452 shutdown_connections 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 --2- 172.21.15.46:0/1784042452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc1050e0 0x7f38bc10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 --2- 172.21.15.46:0/1784042452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc104720 0x7f38bc104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 --2- 172.21.15.46:0/1784042452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc103530 0x7f38bc103950 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.300+0000 7f38c2200640 1 -- 172.21.15.46:0/1784042452 >> 172.21.15.46:0/1784042452 conn(0x7f38bc0feaa0 msgr2=0x7f38bc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.301+0000 7f38c2200640 1 -- 172.21.15.46:0/1784042452 shutdown_connections 2024-03-08T23:06:45.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.301+0000 7f38c2200640 1 -- 172.21.15.46:0/1784042452 wait complete. 2024-03-08T23:06:45.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 Processor -- start 2024-03-08T23:06:45.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 -- start start 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc103530 0x7f38bc199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc1050e0 0x7f38bc1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc10f210 con 0x7f38bc1050e0 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc10f090 con 0x7f38bc103530 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38c2200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38bc10f390 con 0x7f38bc104720 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38baa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38baa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33546/0 (socket says 172.21.15.46:33546) 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38baa00640 1 -- 172.21.15.46:0/996088489 learned_addr learned my addr 172.21.15.46:0/996088489 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.302+0000 7f38bbe00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc1050e0 0x7f38bc1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38bb400640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc103530 0x7f38bc199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 -- 172.21.15.46:0/996088489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc103530 msgr2=0x7f38bc199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc103530 0x7f38bc199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 -- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc1050e0 msgr2=0x7f38bc1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc1050e0 0x7f38bc1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 -- 172.21.15.46:0/996088489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc1a1480 con 0x7f38bc104720 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.303+0000 7f38baa00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.304+0000 7f38b3e00640 1 -- 172.21.15.46:0/996088489 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cd80 con 0x7f38bc104720 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.304+0000 7f38b3e00640 1 -- 172.21.15.46:0/996088489 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38a8010910 con 0x7f38bc104720 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.304+0000 7f38b3e00640 1 -- 172.21.15.46:0/996088489 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8018990 con 0x7f38bc104720 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.304+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38bc1a1770 con 0x7f38bc104720 2024-03-08T23:06:45.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.304+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38bc1a1d10 con 0x7f38bc104720 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 Processor -- start 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 -- start start 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 0x564ef5ad29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 0x564ef5ada580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5a376f0 con 0x564ef5ad80f0 2024-03-08T23:06:45.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef59e09e0 con 0x564ef5ad2f80 2024-03-08T23:06:45.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.307+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5a33790 con 0x564ef5ad2590 2024-03-08T23:06:45.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 0x564ef5ad29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38b3e00640 1 -- 172.21.15.46:0/996088489 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f38a8010ac0 con 0x7f38bc104720 2024-03-08T23:06:45.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49680/0 (socket says 172.21.15.46:49680) 2024-03-08T23:06:45.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 0x564ef5ad29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33550/0 (socket says 172.21.15.46:33550) 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 -- 172.21.15.46:0/516327178 learned_addr learned my addr 172.21.15.46:0/516327178 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 -- 172.21.15.46:0/516327178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 msgr2=0x564ef5ad29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 0x564ef5ad29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38baa00640 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 0x564ef5ada580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 -- 172.21.15.46:0/516327178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 msgr2=0x564ef5ada580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 0x564ef5ada580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38bb400640 1 -- 172.21.15.46:0/516327178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ef59d4720 con 0x564ef5ad2f80 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.308+0000 7f38baa00640 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 0x564ef5ada580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38bb400640 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f38ac000c00 tx=0x7f38ac009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1807b31ce3199d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38a7e00640 1 -- 172.21.15.46:0/516327178 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac02c980 con 0x564ef5ad2f80 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38a7e00640 1 -- 172.21.15.46:0/516327178 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac02cb20 con 0x564ef5ad2f80 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38a7e00640 1 -- 172.21.15.46:0/516327178 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac004920 con 0x564ef5ad2f80 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38b3e00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 0x7f38a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.309+0000 7f38b3e00640 1 -- 172.21.15.46:0/996088489 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f38a80cf090 con 0x7f38bc104720 2024-03-08T23:06:45.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38bbe00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 0x7f38a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 -- 172.21.15.46:0/516327178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 msgr2=0x564ef5ad7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f38ac000c00 tx=0x7f38ac009560 comp rx=0 tx=0).stop 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 -- 172.21.15.46:0/516327178 shutdown_connections 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad80f0 0x564ef5ada580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5ad7a80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/516327178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad2590 0x564ef5ad29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 -- 172.21.15.46:0/516327178 >> 172.21.15.46:0/516327178 conn(0x564ef5acdb00 msgr2=0x564ef5acdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 -- 172.21.15.46:0/516327178 shutdown_connections 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38c4fc4740 1 -- 172.21.15.46:0/516327178 wait complete. 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.310+0000 7f38bbe00640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 0x7f38a00a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x564ef5ad37e0 tx=0x7f38b4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 -- start start 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 Processor -- start 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5adcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5adb040 con 0x564ef5ad2590 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef59f3e20 con 0x564ef5ad2f80 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.311+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5adb1c0 con 0x564ef5ad80f0 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38c4fc4740 10 client.0 _ll_register_callbacks cb 0x564ef5a2e8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50494/0 (socket says 172.21.15.46:50494) 2024-03-08T23:06:45.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 -- 172.21.15.46:0/2730512654 learned_addr learned my addr 172.21.15.46:0/2730512654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38baa00640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5adcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bbe00640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 -- 172.21.15.46:0/2730512654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 msgr2=0x564ef5ae3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 -- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 msgr2=0x564ef5adcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5adcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.312+0000 7f38bb400640 1 -- 172.21.15.46:0/2730512654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ef574c2d0 con 0x564ef5ad2590 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38bb400640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f38ac02c240 tx=0x7f38ac0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa766c02df837de9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac04d070 con 0x564ef5ad2590 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f38ac003a00 con 0x564ef5ad2590 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38ac02cb20 con 0x564ef5ad2590 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38ac059050 con 0x564ef5ad2590 2024-03-08T23:06:45.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38c4fc4740 10 client.15078 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564ef58166b0 con 0x564ef5ad2590 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.313+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac02cb20 con 0x564ef5ad2590 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f38ac03fbf0 con 0x564ef5ad2590 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 10 client.15078 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 10 client.15078 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 20 client.15078 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5ae80c0 0x564ef5aed100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 4 client.15078 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ef5a44c30 con 0x564ef5ae80c0 2024-03-08T23:06:45.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.314+0000 7f38baa00640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5ae80c0 0x564ef5aed100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.315+0000 7f38baa00640 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5ae80c0 0x564ef5aed100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1942c349f73bacf4 server_cookie=dcb26c1c98e98a41 in_seq=0 out_seq=0 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.315+0000 7f38a4c00640 10 client.15078 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.315+0000 7f38a4c00640 1 -- 172.21.15.46:0/2730512654 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564ef5a44c30 con 0x564ef5ae80c0 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.315+0000 7f38a4c00640 10 client.15078 handle_command_reply: tid=0 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 client.15078 shutdown 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 msgr2=0x564ef5adc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f38ac02c240 tx=0x7f38ac0045e0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 shutdown_connections 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5ae80c0 0x564ef5aed100 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2f80 0x564ef5adcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/2730512654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2590 0x564ef5adc750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 >> 172.21.15.46:0/2730512654 conn(0x564ef5acdb00 msgr2=0x564ef5accc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38a4c00640 0 client.15078 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 shutdown_connections 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 1 -- 172.21.15.46:0/2730512654 wait complete. 2024-03-08T23:06:45.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.316+0000 7f38c4fc4740 20 client.15078 trim_cache size 0 max 16384 2024-03-08T23:06:45.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 Processor -- start 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 -- start start 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5acdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ad3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5ae84a0 con 0x564ef5ad2f80 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5ae8b00 con 0x564ef5ad2590 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5ae9160 con 0x564ef5ad80f0 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38baa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38bb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5acdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38baa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50500/0 (socket says 172.21.15.46:50500) 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.335+0000 7f38baa00640 1 -- 172.21.15.46:0/3827113746 learned_addr learned my addr 172.21.15.46:0/3827113746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bbe00640 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ad3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 -- 172.21.15.46:0/3827113746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 msgr2=0x564ef5ad3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ad3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 -- 172.21.15.46:0/3827113746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 msgr2=0x564ef5ae3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 -- 172.21.15.46:0/3827113746 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ef5ad3360 con 0x564ef5ad2590 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38baa00640 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38bb400640 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5acdee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38ac004650 tx=0x7f38ac005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5ff63fd4c0067a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38a7400640 1 -- 172.21.15.46:0/3827113746 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac04d070 con 0x564ef5ad2590 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38a7400640 1 -- 172.21.15.46:0/3827113746 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac02cb20 con 0x564ef5ad2590 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.336+0000 7f38a7400640 1 -- 172.21.15.46:0/3827113746 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac005030 con 0x564ef5ad2590 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 -- 172.21.15.46:0/3827113746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 msgr2=0x564ef5acdee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5acdee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38ac004650 tx=0x7f38ac005fd0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 -- 172.21.15.46:0/3827113746 shutdown_connections 2024-03-08T23:06:45.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ad3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/3827113746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5acdee0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 -- 172.21.15.46:0/3827113746 >> 172.21.15.46:0/3827113746 conn(0x564ef5acdb00 msgr2=0x564ef5ac8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 -- 172.21.15.46:0/3827113746 shutdown_connections 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.337+0000 7f38c4fc4740 1 -- 172.21.15.46:0/3827113746 wait complete. 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 -- start start 2024-03-08T23:06:45.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 Processor -- start 2024-03-08T23:06:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5ae3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5ae9e20 con 0x564ef5ad2f80 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5ae97c0 con 0x564ef5ad2590 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ef5aea480 con 0x564ef5ad80f0 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.338+0000 7f38c4fc4740 10 client.0 _ll_register_callbacks cb 0x7f38b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50516/0 (socket says 172.21.15.46:50516) 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 -- 172.21.15.46:0/216786911 learned_addr learned my addr 172.21.15.46:0/216786911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38bb400640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5ae3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38bbe00640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 -- 172.21.15.46:0/216786911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 msgr2=0x564ef5ae0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 -- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 msgr2=0x564ef5ae3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5ae3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 -- 172.21.15.46:0/216786911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ef5ae0b30 con 0x564ef5ad2f80 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38baa00640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c6a0 tx=0x7f38a80d1560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6342341992021a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.339+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a80cc890 con 0x564ef5ad2f80 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f38a80b4510 con 0x564ef5ad2f80 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38ac02cb20 con 0x564ef5ad2f80 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38a8097dd0 con 0x564ef5ad2f80 2024-03-08T23:06:45.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 10 client.15084 fetch_fsmap learned FSMap version 23 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564ef5b3fc50 con 0x564ef5ad2f80 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38ac02cb20 con 0x564ef5ad2f80 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f38a8018b30 con 0x564ef5ad2f80 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 10 client.15084 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 10 client.15084 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 20 client.15084 populate_metadata read hostname 'smithi046' 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5b44310 0x564ef5b3ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 4 client.15084 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.340+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564ef5a3ef80 con 0x564ef5b44310 2024-03-08T23:06:45.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.341+0000 7f38bb400640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5b44310 0x564ef5b3ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.341+0000 7f38bb400640 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5b44310 0x564ef5b3ff40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8f3bebc91135913 server_cookie=66f503062360a396 in_seq=0 out_seq=0 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.341+0000 7f38a4c00640 10 client.15084 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.341+0000 7f38a4c00640 1 -- 172.21.15.46:0/216786911 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x564ef5a3ef80 con 0x564ef5b44310 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.341+0000 7f38a4c00640 10 client.15084 handle_command_reply: tid=0 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 client.15084 shutdown 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 msgr2=0x564ef5ae3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f38a800c6a0 tx=0x7f38a80d1560 comp rx=0 tx=0).stop 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 shutdown_connections 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ef5b44310 0x564ef5b3ff40 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ef5ad80f0 0x564ef5ae0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ef5ad2f80 0x564ef5ae3ca0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38a4c00640 0 client.15084 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 --2- 172.21.15.46:0/216786911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ef5ad2590 0x564ef5ae3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 >> 172.21.15.46:0/216786911 conn(0x564ef5acdb00 msgr2=0x564ef5acdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 shutdown_connections 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 1 -- 172.21.15.46:0/216786911 wait complete. 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.342+0000 7f38c4fc4740 20 client.15084 trim_cache size 0 max 16384 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24578, 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:06:45.353 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 125.98620817699999, 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 133.540741567 2024-03-08T23:06:45.354 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 msgr2=0x7f38a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 0x7f38a00a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x564ef5ad37e0 tx=0x7f38b4009500 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 msgr2=0x7f38bc199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e9f0 tx=0x7f38a800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 shutdown_connections 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f38a00a2880 0x7f38a00a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f38bc1050e0 0x7f38bc1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38bc104720 0x7f38bc199da0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 --2- 172.21.15.46:0/996088489 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38bc103530 0x7f38bc199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 >> 172.21.15.46:0/996088489 conn(0x7f38bc0feaa0 msgr2=0x7f38bc107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 shutdown_connections 2024-03-08T23:06:45.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:45.343+0000 7f38c2200640 1 -- 172.21.15.46:0/996088489 wait complete. 2024-03-08T23:06:45.363 INFO:tasks.cephadm:Setup complete, yielding 2024-03-08T23:06:45.363 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-08T23:06:45.374 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:06:45.375 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-08T23:06:45.444 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:45 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/899858691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:45.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:45 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/899858691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:45.804 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:45.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:45 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/899858691' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:06:46.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:46 smithi062 ceph-mon[35761]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:06:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:46 smithi046 ceph-mon[31859]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:06:46.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:46 smithi192 ceph-mon[36700]: pgmap v386: 129 pgs: 2 peering, 127 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:06:47.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:47 smithi046 ceph-mon[31859]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-08T23:06:47.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:47 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:06:47.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:47 smithi062 ceph-mon[35761]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-08T23:06:47.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:47 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:06:47.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:47 smithi192 ceph-mon[36700]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-08T23:06:47.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:47 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:06:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 -- 172.21.15.46:0/1389167322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc075db0 msgr2=0x7fc7bc074210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/1389167322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc075db0 0x7fc7bc074210 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac009990 tx=0x7fc7ac009ca0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 -- 172.21.15.46:0/1389167322 shutdown_connections 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/1389167322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc109c10 0x7fc7bc10c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/1389167322 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc075db0 0x7fc7bc074210 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/1389167322 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc074c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 -- 172.21.15.46:0/1389167322 >> 172.21.15.46:0/1389167322 conn(0x7fc7bc0fd2f0 msgr2=0x7fc7bc0ff710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 -- 172.21.15.46:0/1389167322 shutdown_connections 2024-03-08T23:06:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.066+0000 7fc7c0a00700 1 -- 172.21.15.46:0/1389167322 wait complete. 2024-03-08T23:06:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 Processor -- start 2024-03-08T23:06:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 -- start start 2024-03-08T23:06:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc075db0 0x7fc7bc19a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc109c10 0x7fc7bc1a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc10fad0 con 0x7fc7bc109c10 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc10f950 con 0x7fc7bc075db0 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.068+0000 7fc7c0a00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7bc10fc50 con 0x7fc7bc0747e0 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33594/0 (socket says 172.21.15.46:33594) 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 -- 172.21.15.46:0/513093494 learned_addr learned my addr 172.21.15.46:0/513093494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7bb200700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc109c10 0x7fc7bc1a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7b9e00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc075db0 0x7fc7bc19a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 -- 172.21.15.46:0/513093494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc075db0 msgr2=0x7fc7bc19a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc075db0 0x7fc7bc19a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 -- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc109c10 msgr2=0x7fc7bc1a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc109c10 0x7fc7bc1a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.069+0000 7fc7ba800700 1 -- 172.21.15.46:0/513093494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7bc1a1d20 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7ba800700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00bbb0 tx=0x7fc7ac00bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac026070 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7bc1a1fb0 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7ac00f460 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7ac021650 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.070+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7bc1a2350 con 0x7fc7bc0747e0 2024-03-08T23:06:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.071+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7bc068aa0 con 0x7fc7bc0747e0 2024-03-08T23:06:48.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.073+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc7ac029070 con 0x7fc7bc0747e0 2024-03-08T23:06:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.075+0000 7fc7b3400700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 0x7fc7a00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.075+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fc7ac0cffc0 con 0x7fc7bc0747e0 2024-03-08T23:06:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.075+0000 7fc7b9e00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 0x7fc7a00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.076+0000 7fc7b9e00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 0x7fc7a00a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009c60 tx=0x7fc7a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.077+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc7ac030020 con 0x7fc7bc0747e0 2024-03-08T23:06:48.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.325+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7bc062220 con 0x7fc7a00a2890 2024-03-08T23:06:48.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.328+0000 7fc7b3400700 1 -- 172.21.15.46:0/513093494 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc7bc062220 con 0x7fc7a00a2890 2024-03-08T23:06:48.331 INFO:teuthology.orchestra.run.smithi046.stdout:Backend: cephadm 2024-03-08T23:06:48.331 INFO:teuthology.orchestra.run.smithi046.stdout:Available: Yes 2024-03-08T23:06:48.331 INFO:teuthology.orchestra.run.smithi046.stdout:Paused: No 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 msgr2=0x7fc7a00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 0x7fc7a00a4d50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8009c60 tx=0x7fc7a8009260 comp rx=0 tx=0).stop 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 msgr2=0x7fc7bc19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc7ac00bbb0 tx=0x7fc7ac00bbe0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 shutdown_connections 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc7a00a2890 0x7fc7a00a4d50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7bc109c10 0x7fc7bc1a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7bc0747e0 0x7fc7bc19a0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 --2- 172.21.15.46:0/513093494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7bc075db0 0x7fc7bc19a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:48.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 >> 172.21.15.46:0/513093494 conn(0x7fc7bc0fd2f0 msgr2=0x7fc7bc105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:48.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 shutdown_connections 2024-03-08T23:06:48.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:48.331+0000 7fc7c0a00700 1 -- 172.21.15.46:0/513093494 wait complete. 2024-03-08T23:06:48.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:48 smithi062 ceph-mon[35761]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:48.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:48 smithi046 ceph-mon[31859]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:48 smithi192 ceph-mon[36700]: pgmap v387: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:49.228 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-08T23:06:49.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:49 smithi046 ceph-mon[31859]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:49.715 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:49 smithi192 ceph-mon[36700]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:49 smithi062 ceph-mon[35761]: from='client.25216 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:50.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:50 smithi046 ceph-mon[31859]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:50.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:50 smithi192 ceph-mon[36700]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:50.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:50 smithi062 ceph-mon[35761]: pgmap v388: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:51.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 -- 172.21.15.46:0/2741363935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8109cf0 msgr2=0x7f59e810c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:51.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 --2- 172.21.15.46:0/2741363935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8109cf0 0x7f59e810c180 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f59e000b0a0 tx=0x7f59e000b3b0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 -- 172.21.15.46:0/2741363935 shutdown_connections 2024-03-08T23:06:51.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 --2- 172.21.15.46:0/2741363935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8109cf0 0x7f59e810c180 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 --2- 172.21.15.46:0/2741363935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e8074270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 --2- 172.21.15.46:0/2741363935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8074840 0x7f59e8074cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 -- 172.21.15.46:0/2741363935 >> 172.21.15.46:0/2741363935 conn(0x7f59e80fd380 msgr2=0x7f59e80ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 -- 172.21.15.46:0/2741363935 shutdown_connections 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.938+0000 7f59ef000700 1 -- 172.21.15.46:0/2741363935 wait complete. 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.939+0000 7f59ef000700 1 Processor -- start 2024-03-08T23:06:51.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.939+0000 7f59ef000700 1 -- start start 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8074840 0x7f59e819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 0x7f59e81a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e810fc50 con 0x7f59e8109cf0 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e810fad0 con 0x7f59e8075e10 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ef000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e810fdd0 con 0x7f59e8074840 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59ed000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 0x7f59e81a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59e7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59e7400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49726/0 (socket says 172.21.15.46:49726) 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59e7400700 1 -- 172.21.15.46:0/2346562176 learned_addr learned my addr 172.21.15.46:0/2346562176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.940+0000 7f59e7e00700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8074840 0x7f59e819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 -- 172.21.15.46:0/2346562176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8074840 msgr2=0x7f59e819a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:51.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8074840 0x7f59e819a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 -- 172.21.15.46:0/2346562176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 msgr2=0x7f59e819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 -- 172.21.15.46:0/2346562176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e81a1e60 con 0x7f59e8109cf0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59e7400700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ed000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 0x7f59e81a17c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59e0007fa0 tx=0x7f59e0004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0021070 con 0x7f59e8109cf0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e81a20f0 con 0x7f59e8109cf0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59e0004e50 con 0x7f59e8109cf0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.941+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0004780 con 0x7f59e8109cf0 2024-03-08T23:06:51.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.942+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e81a25a0 con 0x7f59e8109cf0 2024-03-08T23:06:51.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.943+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e8068aa0 con 0x7f59e8109cf0 2024-03-08T23:06:51.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.945+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f59e0003a00 con 0x7f59e8109cf0 2024-03-08T23:06:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.946+0000 7f59e4c00700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 0x7f59cc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.946+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f59e00cfd20 con 0x7f59e8109cf0 2024-03-08T23:06:51.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.946+0000 7f59e7e00700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 0x7f59cc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:51.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.947+0000 7f59e7e00700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 0x7f59cc0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f59e80761f0 tx=0x7f59d8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:51.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:51.951+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f59e009ca70 con 0x7f59e8109cf0 2024-03-08T23:06:52.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.190+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f59e8062220 con 0x7f59cc0a2890 2024-03-08T23:06:52.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.239+0000 7f59e4c00700 1 -- 172.21.15.46:0/2346562176 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7f59e8062220 con 0x7f59cc0a2890 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.a smithi046 running (2m) 31s ago 2m 15.9M - 19.0.0-1965-g7d63482a a272146cb810 ea547086676b 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.b smithi062 running (2m) 71s ago 2m 25.1M - 19.0.0-1965-g7d63482a a272146cb810 b0ed8db6071d 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.c smithi192 running (2m) 71s ago 2m 16.0M - 19.0.0-1965-g7d63482a a272146cb810 a231115212da 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.d smithi046 running (2m) 31s ago 2m 12.4M - 19.0.0-1965-g7d63482a a272146cb810 4272141bb326 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.e smithi062 running (2m) 71s ago 2m 13.7M - 19.0.0-1965-g7d63482a a272146cb810 cfa48cce9437 2024-03-08T23:06:52.243 INFO:teuthology.orchestra.run.smithi046.stdout:mds.f smithi192 running (2m) 71s ago 2m 18.2M - 19.0.0-1965-g7d63482a a272146cb810 3f68d9560838 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.g smithi046 running (2m) 31s ago 2m 12.4M - 19.0.0-1965-g7d63482a a272146cb810 710f98c9daf4 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.h smithi062 running (2m) 71s ago 2m 14.9M - 19.0.0-1965-g7d63482a a272146cb810 198957b123a1 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.i smithi192 running (2m) 71s ago 2m 16.6M - 19.0.0-1965-g7d63482a a272146cb810 eceea483525b 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.j smithi046 running (2m) 31s ago 2m 12.4M - 19.0.0-1965-g7d63482a a272146cb810 ae76c76d253b 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.k smithi062 running (2m) 71s ago 2m 15.2M - 19.0.0-1965-g7d63482a a272146cb810 242497e8c070 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mds.l smithi192 running (2m) 71s ago 2m 16.2M - 19.0.0-1965-g7d63482a a272146cb810 3c9dc191f9bc 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.x smithi046 *:9283,8765 running (11m) 31s ago 11m 514M - 19.0.0-1965-g7d63482a a272146cb810 924ab432f56e 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.y smithi062 *:8443,8765 running (8m) 71s ago 8m 453M - 19.0.0-1965-g7d63482a a272146cb810 4bbe139a01e3 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mgr.z smithi192 *:8443,8765 running (8m) 71s ago 8m 453M - 19.0.0-1965-g7d63482a a272146cb810 588a77c10263 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mon.a smithi046 running (11m) 31s ago 11m 49.0M 2048M 19.0.0-1965-g7d63482a a272146cb810 874a63e74f7d 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mon.b smithi062 running (9m) 71s ago 9m 41.2M 2048M 19.0.0-1965-g7d63482a a272146cb810 600f2b51dbe4 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:mon.c smithi192 running (9m) 71s ago 9m 40.4M 2048M 19.0.0-1965-g7d63482a a272146cb810 287209fb75c3 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 smithi046 running (8m) 31s ago 8m 56.5M 4273M 19.0.0-1965-g7d63482a a272146cb810 22d01cd6bc3f 2024-03-08T23:06:52.244 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 smithi062 running (7m) 71s ago 7m 46.4M 4270M 19.0.0-1965-g7d63482a a272146cb810 d397f1567baa 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 smithi192 running (7m) 71s ago 7m 56.6M 4270M 19.0.0-1965-g7d63482a a272146cb810 7ff87dc3ac46 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 smithi046 running (7m) 31s ago 7m 43.2M 4273M 19.0.0-1965-g7d63482a a272146cb810 d92eab0fe036 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 smithi062 running (6m) 71s ago 6m 54.2M 4270M 19.0.0-1965-g7d63482a a272146cb810 7ab24ed681a1 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 smithi192 running (6m) 71s ago 6m 45.2M 4270M 19.0.0-1965-g7d63482a a272146cb810 74c41ef27c47 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 smithi046 running (5m) 31s ago 5m 53.2M 4273M 19.0.0-1965-g7d63482a a272146cb810 8bc949816d6f 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 smithi062 running (5m) 71s ago 5m 45.0M 4270M 19.0.0-1965-g7d63482a a272146cb810 46ff3805b2c9 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 smithi192 running (4m) 71s ago 4m 52.9M 4270M 19.0.0-1965-g7d63482a a272146cb810 489577fda7bd 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 smithi046 running (4m) 31s ago 4m 51.5M 4273M 19.0.0-1965-g7d63482a a272146cb810 7b1f6682db06 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 smithi062 running (3m) 71s ago 3m 43.4M 4270M 19.0.0-1965-g7d63482a a272146cb810 c1fc44f2aee0 2024-03-08T23:06:52.245 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 smithi192 running (3m) 71s ago 3m 43.4M 4270M 19.0.0-1965-g7d63482a a272146cb810 e9e1a9429edb 2024-03-08T23:06:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.242+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 msgr2=0x7f59cc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.242+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 0x7f59cc0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f59e80761f0 tx=0x7f59d8009260 comp rx=0 tx=0).stop 2024-03-08T23:06:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.242+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 msgr2=0x7f59e81a17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.242+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 0x7f59e81a17c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f59e0007fa0 tx=0x7f59e0004c60 comp rx=0 tx=0).stop 2024-03-08T23:06:52.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 shutdown_connections 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f59cc0a2890 0x7f59cc0a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f59e8109cf0 0x7f59e81a17c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f59e8074840 0x7f59e819a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 --2- 172.21.15.46:0/2346562176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59e8075e10 0x7f59e819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 >> 172.21.15.46:0/2346562176 conn(0x7f59e80fd380 msgr2=0x7f59e8105bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 shutdown_connections 2024-03-08T23:06:52.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:52.243+0000 7f59ef000700 1 -- 172.21.15.46:0/2346562176 wait complete. 2024-03-08T23:06:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:52 smithi046 ceph-mon[31859]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:52.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:52 smithi192 ceph-mon[36700]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:52.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:52 smithi062 ceph-mon[35761]: pgmap v389: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:53.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-08T23:06:53.551 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:53.704 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:53 smithi046 ceph-mon[31859]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:53.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:53 smithi192 ceph-mon[36700]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:53.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:53 smithi062 ceph-mon[35761]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:54.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:54 smithi046 ceph-mon[31859]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:54.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:54 smithi192 ceph-mon[36700]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:54.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:54 smithi062 ceph-mon[35761]: pgmap v390: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:55.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 -- 172.21.15.46:0/1175522895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 msgr2=0x7ff218102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:55.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 --2- 172.21.15.46:0/1175522895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff218102370 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff204009920 tx=0x7ff204009c30 comp rx=0 tx=0).stop 2024-03-08T23:06:55.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 -- 172.21.15.46:0/1175522895 shutdown_connections 2024-03-08T23:06:55.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 --2- 172.21.15.46:0/1175522895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff218109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 --2- 172.21.15.46:0/1175522895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff218101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 --2- 172.21.15.46:0/1175522895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff218102370 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.713+0000 7ff220200700 1 -- 172.21.15.46:0/1175522895 >> 172.21.15.46:0/1175522895 conn(0x7ff2180fc2b0 msgr2=0x7ff2180fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.714+0000 7ff220200700 1 -- 172.21.15.46:0/1175522895 shutdown_connections 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.714+0000 7ff220200700 1 -- 172.21.15.46:0/1175522895 wait complete. 2024-03-08T23:06:55.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.716+0000 7ff220200700 1 Processor -- start 2024-03-08T23:06:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.716+0000 7ff220200700 1 -- start start 2024-03-08T23:06:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.716+0000 7ff220200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff220200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff21819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:55.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff220200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff21819f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:55.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff220200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810c920 con 0x7ff2181028b0 2024-03-08T23:06:55.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff220200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810c7a0 con 0x7ff218101f10 2024-03-08T23:06:55.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff220200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21810caa0 con 0x7ff218100d10 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52874/0 (socket says 172.21.15.46:52874) 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 -- 172.21.15.46:0/2723138103 learned_addr learned my addr 172.21.15.46:0/2723138103 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21ce00700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff21819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21e200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff21819f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:55.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 -- 172.21.15.46:0/2723138103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 msgr2=0x7ff21819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff21819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 -- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 msgr2=0x7ff21819f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff21819f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21d800700 1 -- 172.21.15.46:0/2723138103 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21819fde0 con 0x7ff218100d10 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff21ce00700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff21819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.717+0000 7ff21e200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff21819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff21d800700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00ebf0 tx=0x7ff20c00ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00cd40 con 0x7ff218100d10 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c010910 con 0x7ff218100d10 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0189c0 con 0x7ff218100d10 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2181a00d0 con 0x7ff218100d10 2024-03-08T23:06:55.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.718+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2181a0660 con 0x7ff218100d10 2024-03-08T23:06:55.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.719+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff218068aa0 con 0x7ff218100d10 2024-03-08T23:06:55.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.721+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff20c01f020 con 0x7ff218100d10 2024-03-08T23:06:55.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.723+0000 7ff216000700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 0x7ff2000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:55.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.723+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7ff20c0cefe0 con 0x7ff218100d10 2024-03-08T23:06:55.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.723+0000 7ff21ce00700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 0x7ff2000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:55.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.724+0000 7ff21ce00700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 0x7ff2000a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff20400fca0 tx=0x7ff204019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:55.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.727+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff20c053260 con 0x7ff218100d10 2024-03-08T23:06:55.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.967+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff218062220 con 0x7ff2000a2910 2024-03-08T23:06:55.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.975+0000 7ff216000700 1 -- 172.21.15.46:0/2723138103 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7ff218062220 con 0x7ff2000a2910 2024-03-08T23:06:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-08T23:06:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:mds.all 12/12 75s ago 3m smithi046=a;smithi046=d;smithi046=g;smithi046=j;smithi062=b;smithi062=e;smithi062=h;smithi062=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-03-08T23:06:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:mgr 3/3 75s ago 9m smithi046=x;smithi062=y;smithi192=z;count:3 2024-03-08T23:06:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:mon 3/3 75s ago 9m smithi046:172.21.15.46=a;smithi062:172.21.15.62=b;smithi192:172.21.15.192=c;count:3 2024-03-08T23:06:55.979 INFO:teuthology.orchestra.run.smithi046.stdout:osd 12 75s ago - 2024-03-08T23:06:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.978+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 msgr2=0x7ff2000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:55.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.978+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 0x7ff2000a4dd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff20400fca0 tx=0x7ff204019040 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.978+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 msgr2=0x7ff21806a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.978+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00ebf0 tx=0x7ff20c00ef00 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 shutdown_connections 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ff2000a2910 0x7ff2000a4dd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2181028b0 0x7ff21819f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff218100d10 0x7ff21806a650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 --2- 172.21.15.46:0/2723138103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff218101f10 0x7ff21819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 >> 172.21.15.46:0/2723138103 conn(0x7ff2180fc2b0 msgr2=0x7ff218105380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 shutdown_connections 2024-03-08T23:06:55.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:55.979+0000 7ff220200700 1 -- 172.21.15.46:0/2723138103 wait complete. 2024-03-08T23:06:56.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:56 smithi046 ceph-mon[31859]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:56.655 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:56 smithi046 ceph-mon[31859]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:56.729 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-08T23:06:56.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:56 smithi192 ceph-mon[36700]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:56.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:56 smithi192 ceph-mon[36700]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:56.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:56 smithi062 ceph-mon[35761]: pgmap v391: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:56.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:56 smithi062 ceph-mon[35761]: from='client.25228 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:06:57.203 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:06:58.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:06:58 smithi046 ceph-mon[31859]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:58.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:06:58 smithi192 ceph-mon[36700]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:58.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:06:58 smithi062 ceph-mon[35761]: pgmap v392: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 0 B/s, 0 objects/s recovering 2024-03-08T23:06:59.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 -- 172.21.15.46:0/3704756757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa604109660 msgr2=0x7fa604109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 --2- 172.21.15.46:0/3704756757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa604109660 0x7fa604109ac0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00099c0 tx=0x7fa5f0009cd0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 -- 172.21.15.46:0/3704756757 shutdown_connections 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 --2- 172.21.15.46:0/3704756757 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa60410a000 0x7fa6041108b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 --2- 172.21.15.46:0/3704756757 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa604108880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 --2- 172.21.15.46:0/3704756757 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa604109660 0x7fa604109ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 -- 172.21.15.46:0/3704756757 >> 172.21.15.46:0/3704756757 conn(0x7fa604077430 msgr2=0x7fa604079870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 -- 172.21.15.46:0/3704756757 shutdown_connections 2024-03-08T23:06:59.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.315+0000 7fa60b000700 1 -- 172.21.15.46:0/3704756757 wait complete. 2024-03-08T23:06:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 Processor -- start 2024-03-08T23:06:59.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 -- start start 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 0x7fa60419eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60410a000 0x7fa6041a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604114030 con 0x7fa60410a000 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa604113eb0 con 0x7fa604109660 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.317+0000 7fa60b000700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6041141b0 con 0x7fa604108460 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa609000700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60410a000 0x7fa6041a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603400700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 0x7fa60419eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52880/0 (socket says 172.21.15.46:52880) 2024-03-08T23:06:59.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 -- 172.21.15.46:0/10286735 learned_addr learned my addr 172.21.15.46:0/10286735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:06:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 -- 172.21.15.46:0/10286735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 msgr2=0x7fa60419eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:59.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 0x7fa60419eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 -- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60410a000 msgr2=0x7fa6041a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60410a000 0x7fa6041a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603e00700 1 -- 172.21.15.46:0/10286735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6041a6220 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.318+0000 7fa603400700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 0x7fa60419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa603e00700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400e9a0 tx=0x7fa5f400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4004220 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5f40043c0 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6041a6510 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4003df0 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.319+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6041a6a50 con 0x7fa604108460 2024-03-08T23:06:59.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.320+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa604068aa0 con 0x7fa604108460 2024-03-08T23:06:59.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.323+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa5f400c810 con 0x7fa604108460 2024-03-08T23:06:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.324+0000 7fa600c00700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 0x7fa5e80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:06:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.324+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fa5f40cefe0 con 0x7fa604108460 2024-03-08T23:06:59.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.324+0000 7fa603400700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 0x7fa5e80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:06:59.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.325+0000 7fa603400700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 0x7fa5e80a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0000c00 tx=0x7fa5f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:06:59.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.328+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa5f409bce0 con 0x7fa604108460 2024-03-08T23:06:59.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.568+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa604062220 con 0x7fa5e80a2890 2024-03-08T23:06:59.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.572+0000 7fa600c00700 1 -- 172.21.15.46:0/10286735 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fa604062220 con 0x7fa5e80a2890 2024-03-08T23:06:59.575 INFO:teuthology.orchestra.run.smithi046.stdout:HOST ADDR LABELS STATUS 2024-03-08T23:06:59.575 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 172.21.15.46 2024-03-08T23:06:59.576 INFO:teuthology.orchestra.run.smithi046.stdout:smithi062 172.21.15.62 2024-03-08T23:06:59.576 INFO:teuthology.orchestra.run.smithi046.stdout:smithi192 172.21.15.192 2024-03-08T23:06:59.576 INFO:teuthology.orchestra.run.smithi046.stdout:3 hosts in cluster 2024-03-08T23:06:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 msgr2=0x7fa5e80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 0x7fa5e80a4d50 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0000c00 tx=0x7fa5f0005fb0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 msgr2=0x7fa60419e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:06:59.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa5f400e9a0 tx=0x7fa5f400ecb0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 shutdown_connections 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa5e80a2890 0x7fa5e80a4d50 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa60410a000 0x7fa6041a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa604108460 0x7fa60419e5c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 --2- 172.21.15.46:0/10286735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa604109660 0x7fa60419eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.575+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 >> 172.21.15.46:0/10286735 conn(0x7fa604077430 msgr2=0x7fa60410c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.576+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 shutdown_connections 2024-03-08T23:06:59.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:06:59.576+0000 7fa60b000700 1 -- 172.21.15.46:0/10286735 wait complete. 2024-03-08T23:07:00.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-08T23:07:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:00 smithi046 ceph-mon[31859]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:00 smithi046 ceph-mon[31859]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:00.798 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:07:00.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:00 smithi192 ceph-mon[36700]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:00.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:00 smithi192 ceph-mon[36700]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:00.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:00 smithi062 ceph-mon[35761]: pgmap v393: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:00.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:00 smithi062 ceph-mon[35761]: from='client.25234 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:02.693 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:02 smithi062 ceph-mon[35761]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:02.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:02 smithi046 ceph-mon[31859]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:02.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:02 smithi192 ceph-mon[36700]: pgmap v394: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 -- 172.21.15.46:0/1111303545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 msgr2=0x7f65f410a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 --2- 172.21.15.46:0/1111303545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f410a010 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f65ec00f530 tx=0x7f65ec00f840 comp rx=0 tx=0).stop 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 -- 172.21.15.46:0/1111303545 shutdown_connections 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 --2- 172.21.15.46:0/1111303545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f410a010 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 --2- 172.21.15.46:0/1111303545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 --2- 172.21.15.46:0/1111303545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 0x7f65f4107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.916+0000 7f65f9e00700 1 -- 172.21.15.46:0/1111303545 >> 172.21.15.46:0/1111303545 conn(0x7f65f40fc370 msgr2=0x7f65f40fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.917+0000 7f65f9e00700 1 -- 172.21.15.46:0/1111303545 shutdown_connections 2024-03-08T23:07:02.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.917+0000 7f65f9e00700 1 -- 172.21.15.46:0/1111303545 wait complete. 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 Processor -- start 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 -- start start 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 0x7f65f4196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f419d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f410b890 con 0x7f65f4107ba0 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f410b710 con 0x7f65f4102940 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.918+0000 7f65f9e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f410ba10 con 0x7f65f4105270 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f3400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f419d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:02.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2000700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 0x7f65f4196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:58836/0 (socket says 172.21.15.46:58836) 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 -- 172.21.15.46:0/758788480 learned_addr learned my addr 172.21.15.46:0/758788480 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 -- 172.21.15.46:0/758788480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 msgr2=0x7f65f4196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 0x7f65f4196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 -- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 msgr2=0x7f65f419d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f419d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:02.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 -- 172.21.15.46:0/758788480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f419da90 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.919+0000 7f65f2a00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65e400e9a0 tx=0x7f65e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.920+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e400ca10 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.920+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65e400cbb0 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.920+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f419dd80 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.920+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4018710 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.920+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f419e230 con 0x7f65f4102940 2024-03-08T23:07:02.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.921+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f402d830 con 0x7f65f4102940 2024-03-08T23:07:02.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.923+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f65e40188b0 con 0x7f65f4102940 2024-03-08T23:07:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.924+0000 7f65f8a00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 0x7f65d80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.924+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f65e40cef00 con 0x7f65f4102940 2024-03-08T23:07:02.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.925+0000 7f65f2000700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 0x7f65d80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:02.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.925+0000 7f65f2000700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 0x7f65d80a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009c60 tx=0x7f65e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:02.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:02.929+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f65e409bbc0 con 0x7f65f4102940 2024-03-08T23:07:03.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.174+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f65f4062220 con 0x7f65d80a2910 2024-03-08T23:07:03.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.178+0000 7f65f8a00700 1 -- 172.21.15.46:0/758788480 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f65f4062220 con 0x7f65d80a2910 2024-03-08T23:07:03.182 INFO:teuthology.orchestra.run.smithi046.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-08T23:07:03.182 INFO:teuthology.orchestra.run.smithi046.stdout:smithi046 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009A480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-03-08T23:07:03.182 INFO:teuthology.orchestra.run.smithi046.stdout:smithi062 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440094400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-08T23:07:03.182 INFO:teuthology.orchestra.run.smithi046.stdout:smithi192 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500NN400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-08T23:07:03.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 msgr2=0x7f65d80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 0x7f65d80a4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009c60 tx=0x7f65e0009260 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 msgr2=0x7f65f4195e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f65e400e9a0 tx=0x7f65e400ecb0 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 shutdown_connections 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f65d80a2910 0x7f65d80a4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65f4107ba0 0x7f65f419d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f65f4102940 0x7f65f4195e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 --2- 172.21.15.46:0/758788480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65f4105270 0x7f65f4196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.182+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 >> 172.21.15.46:0/758788480 conn(0x7f65f40fc370 msgr2=0x7f65f41031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.183+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 shutdown_connections 2024-03-08T23:07:03.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:03.183+0000 7f65f9e00700 1 -- 172.21.15.46:0/758788480 wait complete. 2024-03-08T23:07:03.964 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-08T23:07:03.973 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:07:03.973 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-08T23:07:04.459 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:07:04.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:04 smithi046 ceph-mon[31859]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:04.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:04 smithi046 ceph-mon[31859]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:04.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:04 smithi192 ceph-mon[36700]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:04.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:04 smithi192 ceph-mon[36700]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:04.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:04 smithi062 ceph-mon[35761]: from='client.25220 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:04.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:04 smithi062 ceph-mon[35761]: pgmap v395: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.621+0000 7fea11200700 1 -- 172.21.15.46:0/3117709584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c104020 msgr2=0x7fea0c104440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.621+0000 7fea11200700 1 --2- 172.21.15.46:0/3117709584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c104020 0x7fea0c104440 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0099c0 tx=0x7fe9fc009cd0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 -- 172.21.15.46:0/3117709584 shutdown_connections 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 --2- 172.21.15.46:0/3117709584 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c105bc0 0x7fea0c10c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 --2- 172.21.15.46:0/3117709584 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c104020 0x7fea0c104440 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 --2- 172.21.15.46:0/3117709584 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105220 0x7fea0c105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 -- 172.21.15.46:0/3117709584 >> 172.21.15.46:0/3117709584 conn(0x7fea0c0ff5a0 msgr2=0x7fea0c101a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 -- 172.21.15.46:0/3117709584 shutdown_connections 2024-03-08T23:07:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.622+0000 7fea11200700 1 -- 172.21.15.46:0/3117709584 wait complete. 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 Processor -- start 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 -- start start 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c104020 0x7fea0c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105bc0 0x7fea0c1a1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c10fc30 con 0x7fea0c105220 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c10fab0 con 0x7fea0c104020 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.623+0000 7fea11200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea0c10fdb0 con 0x7fea0c105bc0 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33556/0 (socket says 172.21.15.46:33556) 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 -- 172.21.15.46:0/3091572769 learned_addr learned my addr 172.21.15.46:0/3091572769 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea0a800700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105bc0 0x7fea0c1a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09e00700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c104020 0x7fea0c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 -- 172.21.15.46:0/3091572769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105bc0 msgr2=0x7fea0c1a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105bc0 0x7fea0c1a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 -- 172.21.15.46:0/3091572769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c104020 msgr2=0x7fea0c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c104020 0x7fea0c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 -- 172.21.15.46:0/3091572769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0c1a1eb0 con 0x7fea0c105220 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.624+0000 7fea09400700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800e9d0 tx=0x7fe9f800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.625+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f800cbd0 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.625+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea0c1a21a0 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.625+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9f800ee70 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.625+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8018660 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.625+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea0c1a26e0 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.626+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea0c068aa0 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.628+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe9f8010430 con 0x7fea0c105220 2024-03-08T23:07:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.629+0000 7fea02a00700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 0x7fe9f00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.630+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fe9f80cf060 con 0x7fea0c105220 2024-03-08T23:07:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.630+0000 7fea09e00700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 0x7fe9f00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:06.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.631+0000 7fea09e00700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 0x7fe9f00a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc00b580 tx=0x7fe9fc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:06.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.634+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe9f809bd20 con 0x7fea0c105220 2024-03-08T23:07:06.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:06 smithi046 ceph-mon[31859]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:06.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:06 smithi192 ceph-mon[36700]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:06.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:06 smithi062 ceph-mon[35761]: pgmap v396: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.913+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fea0c0629f0 con 0x7fea0c105220 2024-03-08T23:07:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.914+0000 7fea02a00700 1 -- 172.21.15.46:0/3091572769 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2937 (secure 0 0 0) 0x7fe9f800cd70 con 0x7fea0c105220 2024-03-08T23:07:06.917 INFO:teuthology.orchestra.run.smithi046.stdout:e23 2024-03-08T23:07:06.917 INFO:teuthology.orchestra.run.smithi046.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:legacy client fscid: 1 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:Filesystem 'cephfs' (1) 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:fs_name cephfs 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 23 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-03-08T23:04:38.798478+0000 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-03-08T23:05:06.080225+0000 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:tableserver 0 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:root 0 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:session_timeout 300 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:session_autoclose 300 2024-03-08T23:07:06.918 INFO:teuthology.orchestra.run.smithi046.stdout:max_file_size 1099511627776 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:max_xattr_size 65536 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:required_client_features {} 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure 0 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:last_failure_osd_epoch 0 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:max_mds 3 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:in 0,1,2 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:up {0=24578,1=24560,2=24553} 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:failed 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:damaged 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:stopped 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:data_pools [3,4] 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:metadata_pool 2 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:inline_data disabled 2024-03-08T23:07:06.919 INFO:teuthology.orchestra.run.smithi046.stdout:balancer 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:bal_rank_mask -1 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:standby_count_wanted 0 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.b{0:24578} state up:active seq 3 addr [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.c{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.192:6836/3408320280,v1:172.21.15.192:6837/3408320280] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.a{1:24560} state up:active seq 10 addr [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.h{1:24524} state up:standby-replay seq 1 addr [v2:172.21.15.62:6836/2774760539,v1:172.21.15.62:6837/2774760539] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.i{2:24553} state up:active seq 13 addr [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.f{2:24520} state up:standby-replay seq 1 addr [v2:172.21.15.192:6834/3258548049,v1:172.21.15.192:6835/3258548049] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:Standby daemons: 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.j{-1:14382} state up:standby seq 1 addr [v2:172.21.15.46:6834/2033972073,v1:172.21.15.46:6835/2033972073] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.g{-1:14448} state up:standby seq 1 addr [v2:172.21.15.46:6838/2215569681,v1:172.21.15.46:6839/2215569681] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.920 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.l{-1:24457} state up:standby seq 1 addr [v2:172.21.15.192:6832/400907664,v1:172.21.15.192:6833/400907664] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.921 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.k{-1:24458} state up:standby seq 1 addr [v2:172.21.15.62:6832/3990698134,v1:172.21.15.62:6833/3990698134] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.921 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.d{-1:24488} state up:standby seq 1 addr [v2:172.21.15.46:6836/950462730,v1:172.21.15.46:6837/950462730] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.921 INFO:teuthology.orchestra.run.smithi046.stdout:[mds.e{-1:24500} state up:standby seq 1 addr [v2:172.21.15.62:6834/213914904,v1:172.21.15.62:6835/213914904] compat {c=[1],r=[1],i=[fff]}] 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.916+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 msgr2=0x7fe9f00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.916+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 0x7fe9f00a4d50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc00b580 tx=0x7fe9fc005fb0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.916+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 msgr2=0x7fea0c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.916+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe9f800e9d0 tx=0x7fe9f800ece0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 shutdown_connections 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fe9f00a2890 0x7fe9f00a4d50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea0c105bc0 0x7fea0c1a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fea0c104020 0x7fea0c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 --2- 172.21.15.46:0/3091572769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fea0c105220 0x7fea0c19a790 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 >> 172.21.15.46:0/3091572769 conn(0x7fea0c0ff5a0 msgr2=0x7fea0c1084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 shutdown_connections 2024-03-08T23:07:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:06.917+0000 7fea11200700 1 -- 172.21.15.46:0/3091572769 wait complete. 2024-03-08T23:07:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:07:07.660 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:07 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3091572769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-08T23:07:07.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-08T23:07:07.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:07 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3091572769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-08T23:07:07.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:07 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3091572769' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-08T23:07:08.193 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:07:08.639 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:08 smithi046 ceph-mon[31859]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:08.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:08 smithi192 ceph-mon[36700]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:08.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:08 smithi062 ceph-mon[35761]: pgmap v397: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:10.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 -- 172.21.15.46:0/1766091410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c109ef0 msgr2=0x7f689c1107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:10.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 --2- 172.21.15.46:0/1766091410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c109ef0 0x7f689c1107a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f689400b0d0 tx=0x7f689400b3e0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 -- 172.21.15.46:0/1766091410 shutdown_connections 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 --2- 172.21.15.46:0/1766091410 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c109ef0 0x7f689c1107a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 --2- 172.21.15.46:0/1766091410 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c108350 0x7f689c108770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 --2- 172.21.15.46:0/1766091410 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109550 0x7f689c1099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 -- 172.21.15.46:0/1766091410 >> 172.21.15.46:0/1766091410 conn(0x7f689c077320 msgr2=0x7f689c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 -- 172.21.15.46:0/1766091410 shutdown_connections 2024-03-08T23:07:10.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.326+0000 7f68a1200700 1 -- 172.21.15.46:0/1766091410 wait complete. 2024-03-08T23:07:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.327+0000 7f68a1200700 1 Processor -- start 2024-03-08T23:07:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 -- start start 2024-03-08T23:07:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:10.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 0x7f689c19e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109ef0 0x7f689c1a5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c113f70 con 0x7f689c108350 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c113df0 con 0x7f689c109ef0 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f68a1200700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c1140f0 con 0x7f689c109550 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f689b400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f689aa00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 0x7f689c19e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f689b400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33572/0 (socket says 172.21.15.46:33572) 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f689be00700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109ef0 0x7f689c1a5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:10.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.328+0000 7f689b400700 1 -- 172.21.15.46:0/3461989667 learned_addr learned my addr 172.21.15.46:0/3461989667 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:10.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 -- 172.21.15.46:0/3461989667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109ef0 msgr2=0x7f689c1a5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109ef0 0x7f689c1a5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 -- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 msgr2=0x7f689c19e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 -- 172.21.15.46:0/3461989667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c1a60f0 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689b400700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f689aa00700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 0x7f689c19e9d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f688800ea20 tx=0x7f688800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.329+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688800cbd0 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.330+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c1a63e0 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.330+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f688800ee60 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.330+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888018680 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.330+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689c1a6970 con 0x7f689c109550 2024-03-08T23:07:10.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.331+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689c068aa0 con 0x7f689c109550 2024-03-08T23:07:10.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.333+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6888018820 con 0x7f689c109550 2024-03-08T23:07:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.334+0000 7f6893e00700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 0x7f68800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.335+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f68880cf130 con 0x7f689c109550 2024-03-08T23:07:10.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.335+0000 7f689b400700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 0x7f68800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:10.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.335+0000 7f689b400700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 0x7f68800a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f688c005950 tx=0x7f688c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:10.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.336+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f688809bdf0 con 0x7f689c109550 2024-03-08T23:07:10.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.577+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f689c02d830 con 0x7f689c109550 2024-03-08T23:07:10.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.578+0000 7f6893e00700 1 -- 172.21.15.46:0/3461989667 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5624 (secure 0 0 0) 0x7f688809b6f0 con 0x7f689c109550 2024-03-08T23:07:10.583 INFO:teuthology.orchestra.run.smithi046.stdout:epoch 93 2024-03-08T23:07:10.583 INFO:teuthology.orchestra.run.smithi046.stdout:fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:07:10.583 INFO:teuthology.orchestra.run.smithi046.stdout:created 2024-03-08T22:55:32.140469+0000 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:modified 2024-03-08T23:06:41.288000+0000 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:crush_version 27 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:full_ratio 0.95 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:backfillfull_ratio 0.9 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:nearfull_ratio 0.85 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:require_min_compat_client luminous 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:min_compat_client luminous 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:require_osd_release squid 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:stretch_mode_enabled false 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-08T23:07:10.584 INFO:teuthology.orchestra.run.smithi046.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:max_osd 12 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6802/3833351406,v1:172.21.15.46:6803/3833351406] [v2:172.21.15.46:6804/3833351406,v1:172.21.15.46:6805/3833351406] exists,up a6108879-28e8-496c-9422-46c94c3d7ca1 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6800/1681427025,v1:172.21.15.62:6801/1681427025] [v2:172.21.15.62:6802/1681427025,v1:172.21.15.62:6803/1681427025] exists,up 5b19e310-9857-4d83-97f8-eab41f626164 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/2982587479,v1:172.21.15.192:6801/2982587479] [v2:172.21.15.192:6802/2982587479,v1:172.21.15.192:6803/2982587479] exists,up aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6810/2986867677,v1:172.21.15.46:6811/2986867677] [v2:172.21.15.46:6812/2986867677,v1:172.21.15.46:6813/2986867677] exists,up 0f714570-8e8a-416b-a6f3-b0a42f737b81 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6808/170069406,v1:172.21.15.62:6809/170069406] [v2:172.21.15.62:6810/170069406,v1:172.21.15.62:6811/170069406] exists,up c5a9bc30-1ba9-4a47-8568-bb143164fc44 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/2546200396,v1:172.21.15.192:6809/2546200396] [v2:172.21.15.192:6810/2546200396,v1:172.21.15.192:6811/2546200396] exists,up 0d68a52e-f5cd-4087-8d3d-0289330f4a9d 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6818/2295220560,v1:172.21.15.46:6819/2295220560] [v2:172.21.15.46:6820/2295220560,v1:172.21.15.46:6821/2295220560] exists,up 2b897988-ec0f-4984-bc95-ec8014984db8 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6816/1007444458,v1:172.21.15.62:6817/1007444458] [v2:172.21.15.62:6818/1007444458,v1:172.21.15.62:6819/1007444458] exists,up da72fad7-cbc8-48e4-8941-79da443bfc2a 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/3623660745,v1:172.21.15.192:6817/3623660745] [v2:172.21.15.192:6818/3623660745,v1:172.21.15.192:6819/3623660745] exists,up a65ba870-b01f-4b9a-847c-d0aee0440027 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.46:6826/1699784429,v1:172.21.15.46:6827/1699784429] [v2:172.21.15.46:6828/1699784429,v1:172.21.15.46:6829/1699784429] exists,up f73f96e3-f0af-44cf-9faa-76a4a469b378 2024-03-08T23:07:10.585 INFO:teuthology.orchestra.run.smithi046.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.62:6824/3230310440,v1:172.21.15.62:6825/3230310440] [v2:172.21.15.62:6826/3230310440,v1:172.21.15.62:6827/3230310440] exists,up dce7acee-a4ac-4a8c-952e-a210943385e5 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/3216803010,v1:172.21.15.192:6825/3216803010] [v2:172.21.15.192:6826/3216803010,v1:172.21.15.192:6827/3216803010] exists,up 8a7095bc-14f0-4fe3-b8d4-717280a11bd2 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.a [2,11] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.e [2,4] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 2.1c [8,11] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.c [3,9] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.e [4,6] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.21 [4,8,5,9] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 3.24 [4,9] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.f [9,8] 2024-03-08T23:07:10.586 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.12 [9,10] 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.13 [9,8] 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:pg_upmap_items 4.15 [5,10] 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/3556325343 expires 2024-03-09T22:56:39.034402+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/2941246107 expires 2024-03-09T22:56:05.453593+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/4209254516 expires 2024-03-09T22:56:39.034402+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/1288083028 expires 2024-03-09T22:56:39.034402+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/1166845583 expires 2024-03-09T22:56:05.453593+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6801/2941246107 expires 2024-03-09T22:56:05.453593+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/2084846526 expires 2024-03-09T22:56:05.453593+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/300257767 expires 2024-03-09T22:56:05.453593+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:6800/4209254516 expires 2024-03-09T22:56:39.034402+0000 2024-03-08T23:07:10.587 INFO:teuthology.orchestra.run.smithi046.stdout:blocklist 172.21.15.46:0/256680715 expires 2024-03-09T22:56:39.034402+0000 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 msgr2=0x7f68800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 0x7f68800a4d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f688c005950 tx=0x7f688c009260 comp rx=0 tx=0).stop 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 msgr2=0x7f689c19e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 0x7f689c19e9d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f688800ea20 tx=0x7f688800ed30 comp rx=0 tx=0).stop 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 shutdown_connections 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f68800a2890 0x7f68800a4d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f689c109ef0 0x7f689c1a5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f689c108350 0x7f689c19e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 --2- 172.21.15.46:0/3461989667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f689c109550 0x7f689c19e9d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.581+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 >> 172.21.15.46:0/3461989667 conn(0x7f689c077320 msgr2=0x7f689c10c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.582+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 shutdown_connections 2024-03-08T23:07:10.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:10.582+0000 7f68a1200700 1 -- 172.21.15.46:0/3461989667 wait complete. 2024-03-08T23:07:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:10 smithi046 ceph-mon[31859]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:10 smithi192 ceph-mon[36700]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:10.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:10 smithi062 ceph-mon[35761]: pgmap v398: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:11.373 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-08T23:07:11.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 -- 172.21.15.46:0/221351213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88075cb0 msgr2=0x7f2a880760d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 --2- 172.21.15.46:0/221351213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88075cb0 0x7f2a880760d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009900 tx=0x7f2a7c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 -- 172.21.15.46:0/221351213 shutdown_connections 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 --2- 172.21.15.46:0/221351213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88077890 0x7f2a88078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 --2- 172.21.15.46:0/221351213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a88077350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 --2- 172.21.15.46:0/221351213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88075cb0 0x7f2a880760d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.614+0000 7f2a8e600640 1 -- 172.21.15.46:0/221351213 >> 172.21.15.46:0/221351213 conn(0x7f2a880fe8d0 msgr2=0x7f2a88100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.615+0000 7f2a8e600640 1 -- 172.21.15.46:0/221351213 shutdown_connections 2024-03-08T23:07:11.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.615+0000 7f2a8e600640 1 -- 172.21.15.46:0/221351213 wait complete. 2024-03-08T23:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 Processor -- start 2024-03-08T23:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 -- start start 2024-03-08T23:07:11.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88075cb0 0x7f2a8819da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a8819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a881133d0 con 0x7f2a88077890 2024-03-08T23:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88113250 con 0x7f2a88075cb0 2024-03-08T23:07:11.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8e600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88113550 con 0x7f2a88076ef0 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.616+0000 7f2a8dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a87e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a8819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33586/0 (socket says 172.21.15.46:33586) 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88075cb0 0x7f2a8819da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 -- 172.21.15.46:0/1850102499 learned_addr learned my addr 172.21.15.46:0/1850102499 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 -- 172.21.15.46:0/1850102499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 msgr2=0x7f2a8819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a8819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 -- 172.21.15.46:0/1850102499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88075cb0 msgr2=0x7f2a8819da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88075cb0 0x7f2a8819da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 -- 172.21.15.46:0/1850102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a881a5650 con 0x7f2a88077890 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a87e00640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a8819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.617+0000 7f2a8dc00640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000eb10 tx=0x7f2a8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.618+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80004b20 con 0x7f2a88077890 2024-03-08T23:07:11.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.618+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a80004cc0 con 0x7f2a88077890 2024-03-08T23:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.618+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a881a5940 con 0x7f2a88077890 2024-03-08T23:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.618+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a80010760 con 0x7f2a88077890 2024-03-08T23:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.618+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a881a5e90 con 0x7f2a88077890 2024-03-08T23:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.619+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8810fe60 con 0x7f2a88077890 2024-03-08T23:07:11.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.621+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2a80005020 con 0x7f2a88077890 2024-03-08T23:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.622+0000 7f2a85600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 0x7f2a700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.622+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f2a800cf320 con 0x7f2a88077890 2024-03-08T23:07:11.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.623+0000 7f2a8d200640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 0x7f2a700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:11.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.623+0000 7f2a8d200640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 0x7f2a700a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c02c7b0 tx=0x7f2a7c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:11.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.627+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2a8009bfe0 con 0x7f2a88077890 2024-03-08T23:07:11.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3461989667' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-08T23:07:11.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3461989667' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-08T23:07:11.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.834+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a88061ef0 con 0x7f2a88077890 2024-03-08T23:07:11.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.836+0000 7f2a85600640 1 -- 172.21.15.46:0/1850102499 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7f2a8009b8e0 con 0x7f2a88077890 2024-03-08T23:07:11.839 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:11.839 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.838+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 msgr2=0x7f2a700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.838+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 0x7f2a700a4d40 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c02c7b0 tx=0x7f2a7c005fd0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 msgr2=0x7f2a881a4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:11.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2a8000eb10 tx=0x7f2a8000c2d0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 shutdown_connections 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f2a700a2880 0x7f2a700a4d40 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2a88077890 0x7f2a881a4fe0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a88076ef0 0x7f2a8819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 --2- 172.21.15.46:0/1850102499 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2a88075cb0 0x7f2a8819da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 >> 172.21.15.46:0/1850102499 conn(0x7f2a880fe8d0 msgr2=0x7f2a880fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 shutdown_connections 2024-03-08T23:07:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:11.839+0000 7f2a8e600640 1 -- 172.21.15.46:0/1850102499 wait complete. 2024-03-08T23:07:11.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:07:11.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:07:11.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3461989667' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 -- 172.21.15.46:0/3455821032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc104580 msgr2=0x7f50dc106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3455821032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc104580 0x7f50dc106970 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f50c4009900 tx=0x7f50c402bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 -- 172.21.15.46:0/3455821032 shutdown_connections 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3455821032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 0x7f50dc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3455821032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc104580 0x7f50dc106970 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3455821032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc101c50 0x7f50dc104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 -- 172.21.15.46:0/3455821032 >> 172.21.15.46:0/3455821032 conn(0x7f50dc0fb660 msgr2=0x7f50dc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 -- 172.21.15.46:0/3455821032 shutdown_connections 2024-03-08T23:07:12.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.091+0000 7f50e0a00640 1 -- 172.21.15.46:0/3455821032 wait complete. 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.092+0000 7f50e0a00640 1 Processor -- start 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.092+0000 7f50e0a00640 1 -- start start 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.092+0000 7f50e0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.092+0000 7f50e0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 0x7f50dc199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50e0a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 0x7f50dc1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50e0a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc10ce60 con 0x7f50dc104580 2024-03-08T23:07:12.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50e0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc10cce0 con 0x7f50dc101c50 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50e0a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc10cfe0 con 0x7f50dc106eb0 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50da800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 0x7f50dc199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50da800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:58918/0 (socket says 172.21.15.46:58918) 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50da800640 1 -- 172.21.15.46:0/3911687830 learned_addr learned my addr 172.21.15.46:0/3911687830 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50db200640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 0x7f50dc1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 -- 172.21.15.46:0/3911687830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 msgr2=0x7f50dc1a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 0x7f50dc1a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 -- 172.21.15.46:0/3911687830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 msgr2=0x7f50dc1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50d1e00640 1 -- 172.21.15.46:0/3911687830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc1a12f0 con 0x7f50dc104580 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.093+0000 7f50da800640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:12.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50d1e00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 0x7f50dc199c10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50c402c170 tx=0x7f50c402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:12.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4041070 con 0x7f50dc104580 2024-03-08T23:07:12.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50c4004450 con 0x7f50dc104580 2024-03-08T23:07:12.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4003e50 con 0x7f50dc104580 2024-03-08T23:07:12.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc1a1580 con 0x7f50dc104580 2024-03-08T23:07:12.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.094+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc1a19f0 con 0x7f50dc104580 2024-03-08T23:07:12.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.095+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc108a00 con 0x7f50dc104580 2024-03-08T23:07:12.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.097+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f50c4044070 con 0x7f50dc104580 2024-03-08T23:07:12.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.099+0000 7f50d3e00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 0x7f50c00a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.099+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f50c40f0830 con 0x7f50dc104580 2024-03-08T23:07:12.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.099+0000 7f50da800640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 0x7f50c00a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.100+0000 7f50da800640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 0x7f50c00a4ee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f50cc011550 tx=0x7f50cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:12.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.103+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f50c40bcea0 con 0x7f50dc104580 2024-03-08T23:07:12.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.305+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f50dc061ef0 con 0x7f50dc104580 2024-03-08T23:07:12.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.306+0000 7f50d3e00640 1 -- 172.21.15.46:0/3911687830 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f50c40c0f40 con 0x7f50dc104580 2024-03-08T23:07:12.309 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.308+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 msgr2=0x7f50c00a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 0x7f50c00a4ee0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f50cc011550 tx=0x7f50cc005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 msgr2=0x7f50dc199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 0x7f50dc199c10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50c402c170 tx=0x7f50c402c650 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 shutdown_connections 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f50c00a2a20 0x7f50c00a4ee0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50dc106eb0 0x7f50dc1a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50dc104580 0x7f50dc199c10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 --2- 172.21.15.46:0/3911687830 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f50dc101c50 0x7f50dc1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 >> 172.21.15.46:0/3911687830 conn(0x7f50dc0fb660 msgr2=0x7f50dc101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 shutdown_connections 2024-03-08T23:07:12.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.309+0000 7f50e0a00640 1 -- 172.21.15.46:0/3911687830 wait complete. 2024-03-08T23:07:12.330 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:07:12.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.550+0000 7fb405400640 1 -- 172.21.15.46:0/3517142394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 msgr2=0x7fb400106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.550+0000 7fb405400640 1 --2- 172.21.15.46:0/3517142394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400106970 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb3e8009900 tx=0x7fb3e802bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.550+0000 7fb405400640 1 -- 172.21.15.46:0/3517142394 shutdown_connections 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 --2- 172.21.15.46:0/3517142394 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb400109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 --2- 172.21.15.46:0/3517142394 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400106970 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 --2- 172.21.15.46:0/3517142394 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb400104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 -- 172.21.15.46:0/3517142394 >> 172.21.15.46:0/3517142394 conn(0x7fb4000fb660 msgr2=0x7fb4000fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 -- 172.21.15.46:0/3517142394 shutdown_connections 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.551+0000 7fb405400640 1 -- 172.21.15.46:0/3517142394 wait complete. 2024-03-08T23:07:12.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.552+0000 7fb405400640 1 Processor -- start 2024-03-08T23:07:12.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.552+0000 7fb405400640 1 -- start start 2024-03-08T23:07:12.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb4001974e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb40019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40010ce60 con 0x7fb400104580 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40010cce0 con 0x7fb400101c50 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb405400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40010cfe0 con 0x7fb400106eb0 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb4001974e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb4001974e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:58934/0 (socket says 172.21.15.46:58934) 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ffe00640 1 -- 172.21.15.46:0/4243779038 learned_addr learned my addr 172.21.15.46:0/4243779038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb404a00640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb40019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 -- 172.21.15.46:0/4243779038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 msgr2=0x7fb40019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb40019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 -- 172.21.15.46:0/4243779038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 msgr2=0x7fb4001974e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb4001974e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.553+0000 7fb3ff400640 1 -- 172.21.15.46:0/4243779038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40019f100 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb404a00640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb40019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb3ff400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400197a20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3e802c170 tx=0x7fb3e802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8041070 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3e8004450 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e8003e50 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40019f390 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.554+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40019f850 con 0x7fb400104580 2024-03-08T23:07:12.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.556+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4000fd150 con 0x7fb400104580 2024-03-08T23:07:12.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.558+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb3e8044070 con 0x7fb400104580 2024-03-08T23:07:12.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.559+0000 7fb3fcc00640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 0x7fb3e40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:12.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.559+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fb3e80f08e0 con 0x7fb400104580 2024-03-08T23:07:12.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.559+0000 7fb3ffe00640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 0x7fb3e40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:12.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.560+0000 7fb3ffe00640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 0x7fb3e40a4e60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00095f0 tx=0x7fb3f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:12.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.566+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb3e80bcf50 con 0x7fb400104580 2024-03-08T23:07:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:12 smithi046 ceph-mon[31859]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:12 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1850102499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:12 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3911687830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:12.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.769+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb400061ef0 con 0x7fb400104580 2024-03-08T23:07:12.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.770+0000 7fb3fcc00640 1 -- 172.21.15.46:0/4243779038 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb3e80c0ff0 con 0x7fb400104580 2024-03-08T23:07:12.773 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.772+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 msgr2=0x7fb3e40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.772+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 0x7fb3e40a4e60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00095f0 tx=0x7fb3f0005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.772+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 msgr2=0x7fb400197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.772+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400197a20 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3e802c170 tx=0x7fb3e802c650 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 shutdown_connections 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb3e40a29a0 0x7fb3e40a4e60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb400106eb0 0x7fb40019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb400104580 0x7fb400197a20 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 --2- 172.21.15.46:0/4243779038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb400101c50 0x7fb4001974e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 >> 172.21.15.46:0/4243779038 conn(0x7fb4000fb660 msgr2=0x7fb400102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 shutdown_connections 2024-03-08T23:07:12.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:12.773+0000 7fb405400640 1 -- 172.21.15.46:0/4243779038 wait complete. 2024-03-08T23:07:12.796 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:07:12.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:12 smithi192 ceph-mon[36700]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:12.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:12 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1850102499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:12.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:12 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3911687830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:12.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:12 smithi062 ceph-mon[35761]: pgmap v399: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:12.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:12 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1850102499' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:12.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:12 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3911687830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 -- 172.21.15.46:0/4067409710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34103530 msgr2=0x7eff34103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 --2- 172.21.15.46:0/4067409710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34103530 0x7eff34103950 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff28009900 tx=0x7eff2802bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 -- 172.21.15.46:0/4067409710 shutdown_connections 2024-03-08T23:07:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 --2- 172.21.15.46:0/4067409710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff341050e0 0x7eff3410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 --2- 172.21.15.46:0/4067409710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff34104720 0x7eff34104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 --2- 172.21.15.46:0/4067409710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34103530 0x7eff34103950 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 -- 172.21.15.46:0/4067409710 >> 172.21.15.46:0/4067409710 conn(0x7eff340feaa0 msgr2=0x7eff34100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:13.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 -- 172.21.15.46:0/4067409710 shutdown_connections 2024-03-08T23:07:13.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.023+0000 7eff3c200640 1 -- 172.21.15.46:0/4067409710 wait complete. 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.024+0000 7eff3c200640 1 Processor -- start 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 -- start start 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff34103530 0x7eff34199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34104720 0x7eff34199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410f210 con 0x7eff34103530 2024-03-08T23:07:13.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410f090 con 0x7eff341050e0 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3c200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3410f390 con 0x7eff34104720 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff39800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff34103530 0x7eff34199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34768/0 (socket says 172.21.15.46:34768) 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff3a200640 1 -- 172.21.15.46:0/765562463 learned_addr learned my addr 172.21.15.46:0/765562463 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:13.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.025+0000 7eff38e00640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34104720 0x7eff34199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 -- 172.21.15.46:0/765562463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34104720 msgr2=0x7eff34199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34104720 0x7eff34199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 -- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff34103530 msgr2=0x7eff34199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff34103530 0x7eff34199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 -- 172.21.15.46:0/765562463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff341a1460 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff3a200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00eb10 tx=0x7eff2c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c004b20 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.026+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff2c004cc0 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.027+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c010760 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.027+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff341a1750 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.027+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff341a1ca0 con 0x7eff341050e0 2024-03-08T23:07:13.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.028+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3410a850 con 0x7eff341050e0 2024-03-08T23:07:13.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.030+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7eff2c005020 con 0x7eff341050e0 2024-03-08T23:07:13.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.031+0000 7eff32000640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 0x7eff1c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.031+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7eff2c0cf320 con 0x7eff341050e0 2024-03-08T23:07:13.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.031+0000 7eff39800640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 0x7eff1c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.032+0000 7eff39800640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 0x7eff1c0a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7eff2802c170 tx=0x7eff2802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.035+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7eff2c09bfe0 con 0x7eff341050e0 2024-03-08T23:07:13.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.237+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7eff34061ef0 con 0x7eff341050e0 2024-03-08T23:07:13.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.238+0000 7eff32000640 1 -- 172.21.15.46:0/765562463 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7eff2c09b8e0 con 0x7eff341050e0 2024-03-08T23:07:13.241 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 msgr2=0x7eff1c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 0x7eff1c0a4d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7eff2802c170 tx=0x7eff2802df90 comp rx=0 tx=0).stop 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 msgr2=0x7eff341a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7eff2c00eb10 tx=0x7eff2c00c2d0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 shutdown_connections 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7eff1c0a2880 0x7eff1c0a4d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff341050e0 0x7eff341a0df0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff34104720 0x7eff34199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 --2- 172.21.15.46:0/765562463 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff34103530 0x7eff34199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 >> 172.21.15.46:0/765562463 conn(0x7eff340feaa0 msgr2=0x7eff34107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 shutdown_connections 2024-03-08T23:07:13.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.241+0000 7eff3c200640 1 -- 172.21.15.46:0/765562463 wait complete. 2024-03-08T23:07:13.262 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 -- 172.21.15.46:0/1325846610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54075330 msgr2=0x7fcd54073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 --2- 172.21.15.46:0/1325846610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54075330 0x7fcd54073780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcd44009900 tx=0x7fcd4402bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 -- 172.21.15.46:0/1325846610 shutdown_connections 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 --2- 172.21.15.46:0/1325846610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd541090b0 0x7fcd5410b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 --2- 172.21.15.46:0/1325846610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd54073d50 0x7fcd540741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 --2- 172.21.15.46:0/1325846610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54075330 0x7fcd54073780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 -- 172.21.15.46:0/1325846610 >> 172.21.15.46:0/1325846610 conn(0x7fcd540fc760 msgr2=0x7fcd540feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.482+0000 7fcd58c00640 1 -- 172.21.15.46:0/1325846610 shutdown_connections 2024-03-08T23:07:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.483+0000 7fcd58c00640 1 -- 172.21.15.46:0/1325846610 wait complete. 2024-03-08T23:07:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.483+0000 7fcd58c00640 1 Processor -- start 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 -- start start 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54073d50 0x7fcd541996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541090b0 0x7fcd541a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5410f080 con 0x7fcd541090b0 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5410ef00 con 0x7fcd54075330 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd58c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5410f200 con 0x7fcd54073d50 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd4aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd53e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541090b0 0x7fcd541a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd4aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34784/0 (socket says 172.21.15.46:34784) 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd4aa00640 1 -- 172.21.15.46:0/674991403 learned_addr learned my addr 172.21.15.46:0/674991403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.484+0000 7fcd53400640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54073d50 0x7fcd541996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 -- 172.21.15.46:0/674991403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54073d50 msgr2=0x7fcd541996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54073d50 0x7fcd541996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 -- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541090b0 msgr2=0x7fcd541a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541090b0 0x7fcd541a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 -- 172.21.15.46:0/674991403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd541a12f0 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd4aa00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00ea60 tx=0x7fcd3c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00cd20 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd3c0035f0 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd541a15e0 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.485+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c010690 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.486+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd541a1b80 con 0x7fcd54075330 2024-03-08T23:07:13.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.487+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd54107de0 con 0x7fcd54075330 2024-03-08T23:07:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.488+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd3c0037a0 con 0x7fcd54075330 2024-03-08T23:07:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.490+0000 7fcd50c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 0x7fcd340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.490+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fcd3c0cf0c0 con 0x7fcd54075330 2024-03-08T23:07:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.490+0000 7fcd53400640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 0x7fcd340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.491+0000 7fcd53400640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 0x7fcd340a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcd4402c250 tx=0x7fcd4402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.494+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcd3c09bd80 con 0x7fcd54075330 2024-03-08T23:07:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.693+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fcd54061ef0 con 0x7fcd54075330 2024-03-08T23:07:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.694+0000 7fcd50c00640 1 -- 172.21.15.46:0/674991403 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fcd3c09b680 con 0x7fcd54075330 2024-03-08T23:07:13.697 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 msgr2=0x7fcd340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 0x7fcd340a4d40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fcd4402c250 tx=0x7fcd4402cf90 comp rx=0 tx=0).stop 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 msgr2=0x7fcd54199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00ea60 tx=0x7fcd3c00ee60 comp rx=0 tx=0).stop 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 shutdown_connections 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd340a2880 0x7fcd340a4d40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcd541090b0 0x7fcd541a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcd54075330 0x7fcd54199c10 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 --2- 172.21.15.46:0/674991403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcd54073d50 0x7fcd541996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 >> 172.21.15.46:0/674991403 conn(0x7fcd540fc760 msgr2=0x7fcd54104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 shutdown_connections 2024-03-08T23:07:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.697+0000 7fcd58c00640 1 -- 172.21.15.46:0/674991403 wait complete. 2024-03-08T23:07:13.717 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:13 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4243779038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.718 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:13 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/765562463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.721 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:07:13.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:13 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4243779038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:13 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/765562463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:13 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4243779038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:13 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/765562463' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 -- 172.21.15.46:0/2122964533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 msgr2=0x7f9ca0102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 --2- 172.21.15.46:0/2122964533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0102860 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009930 tx=0x7f9c8c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 -- 172.21.15.46:0/2122964533 shutdown_connections 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 --2- 172.21.15.46:0/2122964533 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca0109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 --2- 172.21.15.46:0/2122964533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0102860 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 --2- 172.21.15.46:0/2122964533 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca00ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 -- 172.21.15.46:0/2122964533 >> 172.21.15.46:0/2122964533 conn(0x7f9ca00fad40 msgr2=0x7f9ca00fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 -- 172.21.15.46:0/2122964533 shutdown_connections 2024-03-08T23:07:13.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.937+0000 7f9ca5200640 1 -- 172.21.15.46:0/2122964533 wait complete. 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 Processor -- start 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 -- start start 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca0199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca010ce00 con 0x7f9ca0102da0 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca010cc80 con 0x7f9ca00ff410 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.938+0000 7f9ca5200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca010cf80 con 0x7f9ca01023e0 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39124/0 (socket says 172.21.15.46:39124) 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 -- 172.21.15.46:0/3601020839 learned_addr learned my addr 172.21.15.46:0/3601020839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:13.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9f400640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca0199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9ea00640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 -- 172.21.15.46:0/3601020839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 msgr2=0x7f9ca0199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 -- 172.21.15.46:0/3601020839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 msgr2=0x7f9ca0199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca0199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9fe00640 1 -- 172.21.15.46:0/3601020839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca01a12d0 con 0x7f9ca0102da0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.939+0000 7f9c9f400640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca0199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9c9fe00640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800eb10 tx=0x7f9c9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98004b20 con 0x7f9ca0102da0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca01a15c0 con 0x7f9ca0102da0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c98004cc0 con 0x7f9ca0102da0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98010760 con 0x7f9ca0102da0 2024-03-08T23:07:13.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.940+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca01a1ae0 con 0x7f9ca0102da0 2024-03-08T23:07:13.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.941+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca01a1da0 con 0x7f9ca0102da0 2024-03-08T23:07:13.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.943+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9c98005020 con 0x7f9ca0102da0 2024-03-08T23:07:13.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.945+0000 7f9c97e00640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 0x7f9c840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:13.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.945+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f9c980cf3b0 con 0x7f9ca0102da0 2024-03-08T23:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.945+0000 7f9c9f400640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 0x7f9c840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:13.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.946+0000 7f9c9f400640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 0x7f9c840a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00696c0 tx=0x7f9c90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:13.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:13.949+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9c9809c070 con 0x7f9ca0102da0 2024-03-08T23:07:14.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.189+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ca01a1f70 con 0x7f9ca0102da0 2024-03-08T23:07:14.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.191+0000 7f9c97e00640 1 -- 172.21.15.46:0/3601020839 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f9c9809b970 con 0x7f9ca0102da0 2024-03-08T23:07:14.194 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:14.194 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:07:14.194 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:07:14.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.193+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 msgr2=0x7f9c840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 0x7f9c840a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00696c0 tx=0x7f9c90005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:14.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 msgr2=0x7f9ca01a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800eb10 tx=0x7f9c9800c2d0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 shutdown_connections 2024-03-08T23:07:14.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9c840a2800 0x7f9c840a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9ca0102da0 0x7f9ca01a0c60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9ca01023e0 0x7f9ca0199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 --2- 172.21.15.46:0/3601020839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9ca00ff410 0x7f9ca0199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 >> 172.21.15.46:0/3601020839 conn(0x7f9ca00fad40 msgr2=0x7f9ca01001d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 shutdown_connections 2024-03-08T23:07:14.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.194+0000 7f9ca5200640 1 -- 172.21.15.46:0/3601020839 wait complete. 2024-03-08T23:07:14.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.428+0000 7f6998e00640 1 -- 172.21.15.46:0/277022616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 msgr2=0x7f699410b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.428+0000 7f6998e00640 1 --2- 172.21.15.46:0/277022616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f699410b780 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f698c009ca0 tx=0x7f698c02c100 comp rx=0 tx=0).stop 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 -- 172.21.15.46:0/277022616 shutdown_connections 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 --2- 172.21.15.46:0/277022616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f699410b780 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 --2- 172.21.15.46:0/277022616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994104530 0x7f69941049b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 --2- 172.21.15.46:0/277022616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994103340 0x7f6994103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 -- 172.21.15.46:0/277022616 >> 172.21.15.46:0/277022616 conn(0x7f69940fe8d0 msgr2=0x7f6994100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 -- 172.21.15.46:0/277022616 shutdown_connections 2024-03-08T23:07:14.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.429+0000 7f6998e00640 1 -- 172.21.15.46:0/277022616 wait complete. 2024-03-08T23:07:14.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 Processor -- start 2024-03-08T23:07:14.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 -- start start 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 0x7f6994199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f69941a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699410efd0 con 0x7f6994104530 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699410ee50 con 0x7f6994103340 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6998e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699410f150 con 0x7f6994104ef0 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6992a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6993400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 0x7f6994199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6992a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39142/0 (socket says 172.21.15.46:39142) 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6992a00640 1 -- 172.21.15.46:0/68949381 learned_addr learned my addr 172.21.15.46:0/68949381 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.430+0000 7f6993e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f69941a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 -- 172.21.15.46:0/68949381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 msgr2=0x7f69941a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f69941a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 -- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 msgr2=0x7f6994199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 -- 172.21.15.46:0/68949381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69941a1230 con 0x7f6994103340 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6992a00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f69941a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f6993400640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 0x7f6994199610 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f698400e9c0 tx=0x7f698400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.431+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400ccc0 con 0x7f6994103340 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.432+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69941a1520 con 0x7f6994103340 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.432+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69840035f0 con 0x7f6994103340 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.432+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400ce60 con 0x7f6994103340 2024-03-08T23:07:14.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.432+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69941a1ac0 con 0x7f6994103340 2024-03-08T23:07:14.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.433+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699410a600 con 0x7f6994103340 2024-03-08T23:07:14.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.435+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6984003b30 con 0x7f6994103340 2024-03-08T23:07:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.436+0000 7f698be00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 0x7f69780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.436+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f69840cf250 con 0x7f6994103340 2024-03-08T23:07:14.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.437+0000 7f6992a00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 0x7f69780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.437+0000 7f6992a00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 0x7f69780a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f699419ab20 tx=0x7f6980005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:14.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.439+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f698409bfa0 con 0x7f6994103340 2024-03-08T23:07:14.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.639+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f699402d060 con 0x7f6994103340 2024-03-08T23:07:14.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.641+0000 7f698be00640 1 -- 172.21.15.46:0/68949381 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7f698409b8a0 con 0x7f6994103340 2024-03-08T23:07:14.645 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:14.645 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 msgr2=0x7f69780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 0x7f69780a4d40 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f699419ab20 tx=0x7f6980005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 msgr2=0x7f6994199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 0x7f6994199610 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f698400e9c0 tx=0x7f698400ee80 comp rx=0 tx=0).stop 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 shutdown_connections 2024-03-08T23:07:14.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f69780a2880 0x7f69780a4d40 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6994104ef0 0x7f69941a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6994104530 0x7f6994199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 --2- 172.21.15.46:0/68949381 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6994103340 0x7f6994199610 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 >> 172.21.15.46:0/68949381 conn(0x7f69940fe8d0 msgr2=0x7f69941077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 shutdown_connections 2024-03-08T23:07:14.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.644+0000 7f6998e00640 1 -- 172.21.15.46:0/68949381 wait complete. 2024-03-08T23:07:14.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:07:14.668 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:07:14.743 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:14 smithi046 ceph-mon[31859]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:14.744 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/674991403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:14.744 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3601020839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:14 smithi192 ceph-mon[36700]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/674991403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3601020839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:14.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 -- 172.21.15.46:0/3686234251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41023e0 msgr2=0x7fa1f4102860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 --2- 172.21.15.46:0/3686234251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41023e0 0x7fa1f4102860 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009860 tx=0x7fa1e002bb70 comp rx=0 tx=0).stop 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 -- 172.21.15.46:0/3686234251 shutdown_connections 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 --2- 172.21.15.46:0/3686234251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f4102da0 0x7fa1f4109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 --2- 172.21.15.46:0/3686234251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f41023e0 0x7fa1f4102860 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 --2- 172.21.15.46:0/3686234251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f4101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 -- 172.21.15.46:0/3686234251 >> 172.21.15.46:0/3686234251 conn(0x7fa1f40fc760 msgr2=0x7fa1f40feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 -- 172.21.15.46:0/3686234251 shutdown_connections 2024-03-08T23:07:14.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.879+0000 7fa1fa600640 1 -- 172.21.15.46:0/3686234251 wait complete. 2024-03-08T23:07:14.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.880+0000 7fa1fa600640 1 Processor -- start 2024-03-08T23:07:14.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.880+0000 7fa1fa600640 1 -- start start 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f41023e0 0x7fa1f40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f410ce80 con 0x7fa1f41011f0 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f410cd00 con 0x7fa1f4102da0 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1fa600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f410d000 con 0x7fa1f41023e0 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34856/0 (socket says 172.21.15.46:34856) 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9c00640 1 -- 172.21.15.46:0/2053776257 learned_addr learned my addr 172.21.15.46:0/2053776257 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39160/0 (socket says 172.21.15.46:39160) 2024-03-08T23:07:14.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f3e00640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f41023e0 0x7fa1f40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 -- 172.21.15.46:0/2053776257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f41023e0 msgr2=0x7fa1f40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f41023e0 0x7fa1f40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 -- 172.21.15.46:0/2053776257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 msgr2=0x7fa1f4073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9200640 1 -- 172.21.15.46:0/2053776257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4074160 con 0x7fa1f41011f0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.881+0000 7fa1f9c00640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1f9200640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4102240 tx=0x7fa1e800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8004440 con 0x7fa1f41011f0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f4074450 con 0x7fa1f41011f0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e800cda0 con 0x7fa1f41011f0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80196e0 con 0x7fa1f41011f0 2024-03-08T23:07:14.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.882+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f41a5f80 con 0x7fa1f41011f0 2024-03-08T23:07:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.883+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f41084b0 con 0x7fa1f41011f0 2024-03-08T23:07:14.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.885+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa1e8003780 con 0x7fa1f41011f0 2024-03-08T23:07:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.887+0000 7fa1f1600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 0x7fa1dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.888+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fa1e80cf270 con 0x7fa1f41011f0 2024-03-08T23:07:14.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.888+0000 7fa1f3e00640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 0x7fa1dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:14 smithi062 ceph-mon[35761]: pgmap v400: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/674991403' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:14.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3601020839' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:14.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.889+0000 7fa1f3e00640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 0x7fa1dc0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009830 tx=0x7fa1e0034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:14.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:14.891+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa1e8003e20 con 0x7fa1f41011f0 2024-03-08T23:07:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.091+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa1f402ce70 con 0x7fa1f41011f0 2024-03-08T23:07:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.092+0000 7fa1f1600640 1 -- 172.21.15.46:0/2053776257 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa1e8009510 con 0x7fa1f41011f0 2024-03-08T23:07:15.095 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 msgr2=0x7fa1dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 0x7fa1dc0a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009830 tx=0x7fa1e0034ab0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 msgr2=0x7fa1f40750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4102240 tx=0x7fa1e800ee50 comp rx=0 tx=0).stop 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 shutdown_connections 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa1dc0a2800 0x7fa1dc0a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1f4102da0 0x7fa1f4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1f41023e0 0x7fa1f40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 --2- 172.21.15.46:0/2053776257 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1f41011f0 0x7fa1f40750a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 >> 172.21.15.46:0/2053776257 conn(0x7fa1f40fc760 msgr2=0x7fa1f4105680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 shutdown_connections 2024-03-08T23:07:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.095+0000 7fa1fa600640 1 -- 172.21.15.46:0/2053776257 wait complete. 2024-03-08T23:07:15.118 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:07:15.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 -- 172.21.15.46:0/476581824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 msgr2=0x7f911c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 --2- 172.21.15.46:0/476581824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c106aa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9108009900 tx=0x7f910802bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:15.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 -- 172.21.15.46:0/476581824 shutdown_connections 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 --2- 172.21.15.46:0/476581824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 0x7f911c109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 --2- 172.21.15.46:0/476581824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c106aa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 --2- 172.21.15.46:0/476581824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 0x7f911c104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 -- 172.21.15.46:0/476581824 >> 172.21.15.46:0/476581824 conn(0x7f911c0fb790 msgr2=0x7f911c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 -- 172.21.15.46:0/476581824 shutdown_connections 2024-03-08T23:07:15.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.314+0000 7f9123a00640 1 -- 172.21.15.46:0/476581824 wait complete. 2024-03-08T23:07:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.315+0000 7f9123a00640 1 Processor -- start 2024-03-08T23:07:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.315+0000 7f9123a00640 1 -- start start 2024-03-08T23:07:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 0x7f911c1016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 0x7f911c100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c10cf90 con 0x7f911c1046b0 2024-03-08T23:07:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c10ce10 con 0x7f911c106fe0 2024-03-08T23:07:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9123a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911c10d110 con 0x7f911c101d80 2024-03-08T23:07:15.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39174/0 (socket says 172.21.15.46:39174) 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 -- 172.21.15.46:0/1976138714 learned_addr learned my addr 172.21.15.46:0/1976138714 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9121000640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 0x7f911c1016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f9121a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 0x7f911c100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 -- 172.21.15.46:0/1976138714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 msgr2=0x7f911c1016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 0x7f911c1016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 -- 172.21.15.46:0/1976138714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 msgr2=0x7f911c100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 0x7f911c100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.316+0000 7f911be00640 1 -- 172.21.15.46:0/1976138714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c100b00 con 0x7f911c1046b0 2024-03-08T23:07:15.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f911be00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9108000c00 tx=0x7f910802c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108041070 con 0x7f911c1046b0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c1a1740 con 0x7f911c1046b0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9108004450 con 0x7f911c1046b0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9108034440 con 0x7f911c1046b0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.317+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c1a1ce0 con 0x7f911c1046b0 2024-03-08T23:07:15.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.319+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc0052c0 con 0x7f911c1046b0 2024-03-08T23:07:15.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.321+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9108044070 con 0x7f911c1046b0 2024-03-08T23:07:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.322+0000 7f9119600640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 0x7f91040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.323+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f91080f09c0 con 0x7f911c1046b0 2024-03-08T23:07:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.323+0000 7f9121000640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 0x7f91040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.324+0000 7f9121000640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 0x7f91040a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9110009bd0 tx=0x7f9110005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:15.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.325+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f91080bd0c0 con 0x7f911c1046b0 2024-03-08T23:07:15.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.532+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f90fc005880 con 0x7f911c1046b0 2024-03-08T23:07:15.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.533+0000 7f9119600640 1 -- 172.21.15.46:0/1976138714 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f91080c1160 con 0x7f911c1046b0 2024-03-08T23:07:15.536 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:15.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 msgr2=0x7f91040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 0x7f91040a4d40 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9110009bd0 tx=0x7f9110005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:15.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 msgr2=0x7f911c0ffd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9108000c00 tx=0x7f910802c640 comp rx=0 tx=0).stop 2024-03-08T23:07:15.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 shutdown_connections 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f91040a2880 0x7f91040a4d40 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f911c106fe0 0x7f911c100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f911c1046b0 0x7f911c0ffd30 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 --2- 172.21.15.46:0/1976138714 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f911c101d80 0x7f911c1016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 >> 172.21.15.46:0/1976138714 conn(0x7f911c0fb790 msgr2=0x7f911c1028a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 shutdown_connections 2024-03-08T23:07:15.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.536+0000 7f9123a00640 1 -- 172.21.15.46:0/1976138714 wait complete. 2024-03-08T23:07:15.558 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:07:15.636 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:15 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/68949381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:15.636 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:15 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2053776257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 -- 172.21.15.46:0/3622311955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1011f0 msgr2=0x7f992c101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 --2- 172.21.15.46:0/3622311955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1011f0 0x7f992c101610 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f991c009900 tx=0x7f991c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 -- 172.21.15.46:0/3622311955 shutdown_connections 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 --2- 172.21.15.46:0/3622311955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 0x7f992c109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 --2- 172.21.15.46:0/3622311955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1023e0 0x7f992c102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 --2- 172.21.15.46:0/3622311955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1011f0 0x7f992c101610 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.770+0000 7f992b200640 1 -- 172.21.15.46:0/3622311955 >> 172.21.15.46:0/3622311955 conn(0x7f992c0fc760 msgr2=0x7f992c0feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:15.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.771+0000 7f992b200640 1 -- 172.21.15.46:0/3622311955 shutdown_connections 2024-03-08T23:07:15.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.771+0000 7f992b200640 1 -- 172.21.15.46:0/3622311955 wait complete. 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 Processor -- start 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 -- start start 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1023e0 0x7f992c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 0x7f992c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c10ced0 con 0x7f992c102da0 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c10cd50 con 0x7f992c1011f0 2024-03-08T23:07:15.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.772+0000 7f992b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c10d050 con 0x7f992c1023e0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9923e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1023e0 0x7f992c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34910/0 (socket says 172.21.15.46:34910) 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 -- 172.21.15.46:0/724743646 learned_addr learned my addr 172.21.15.46:0/724743646 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f992a800640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 0x7f992c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 -- 172.21.15.46:0/724743646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1023e0 msgr2=0x7f992c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1023e0 0x7f992c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 -- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 msgr2=0x7f992c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 0x7f992c073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 -- 172.21.15.46:0/724743646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c0744b0 con 0x7f992c1011f0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.773+0000 7f9929e00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:15.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.774+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c043020 con 0x7f992c1011f0 2024-03-08T23:07:15.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.774+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f991c0054e0 con 0x7f992c1011f0 2024-03-08T23:07:15.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.774+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c1a1640 con 0x7f992c1011f0 2024-03-08T23:07:15.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.774+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c003bd0 con 0x7f992c1011f0 2024-03-08T23:07:15.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.774+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c1a1b90 con 0x7f992c1011f0 2024-03-08T23:07:15.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.775+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c108510 con 0x7f992c1011f0 2024-03-08T23:07:15.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.777+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f991c046020 con 0x7f992c1011f0 2024-03-08T23:07:15.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.778+0000 7f9922a00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:15.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.779+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f991c0f13a0 con 0x7f992c1011f0 2024-03-08T23:07:15.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.779+0000 7f9923e00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:15.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.779+0000 7f9923e00640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 0x7f99100a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f992c074e10 tx=0x7f9914005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:15.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.781+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f991c04d020 con 0x7f992c1011f0 2024-03-08T23:07:15.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:15 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/68949381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:15.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:15 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2053776257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:15.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:15 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/68949381' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:15.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:15 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2053776257' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.979+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f992c02d000 con 0x7f992c1011f0 2024-03-08T23:07:15.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.979+0000 7f9922a00640 1 -- 172.21.15.46:0/724743646 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f991c0be060 con 0x7f992c1011f0 2024-03-08T23:07:15.983 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 msgr2=0x7f99100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 0x7f99100a4cc0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f992c074e10 tx=0x7f9914005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 msgr2=0x7f992c0750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f991c000c00 tx=0x7f991c02c650 comp rx=0 tx=0).stop 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 shutdown_connections 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f99100a2800 0x7f99100a4cc0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f992c102da0 0x7f992c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f992c1023e0 0x7f992c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 --2- 172.21.15.46:0/724743646 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f992c1011f0 0x7f992c0750a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 >> 172.21.15.46:0/724743646 conn(0x7f992c0fc760 msgr2=0x7f992c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 shutdown_connections 2024-03-08T23:07:15.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:15.982+0000 7f992b200640 1 -- 172.21.15.46:0/724743646 wait complete. 2024-03-08T23:07:16.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:07:16.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 -- 172.21.15.46:0/977413220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c28075330 msgr2=0x7f5c28073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 --2- 172.21.15.46:0/977413220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c28075330 0x7f5c28073780 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c009900 tx=0x7f5c1c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:16.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 -- 172.21.15.46:0/977413220 shutdown_connections 2024-03-08T23:07:16.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 --2- 172.21.15.46:0/977413220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c281090b0 0x7f5c2810b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 --2- 172.21.15.46:0/977413220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28073d50 0x7f5c280741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 --2- 172.21.15.46:0/977413220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c28075330 0x7f5c28073780 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 -- 172.21.15.46:0/977413220 >> 172.21.15.46:0/977413220 conn(0x7f5c280fc760 msgr2=0x7f5c280feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:16.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 -- 172.21.15.46:0/977413220 shutdown_connections 2024-03-08T23:07:16.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.214+0000 7f5c2e000640 1 -- 172.21.15.46:0/977413220 wait complete. 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 Processor -- start 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 -- start start 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c28073d50 0x7f5c281996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 0x7f5c28199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2810f030 con 0x7f5c281090b0 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2810eeb0 con 0x7f5c28075330 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.215+0000 7f5c2e000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c2810f1b0 con 0x7f5c28073d50 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c2d600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c2d600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39204/0 (socket says 172.21.15.46:39204) 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c2d600640 1 -- 172.21.15.46:0/3611434127 learned_addr learned my addr 172.21.15.46:0/3611434127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c2cc00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c28073d50 0x7f5c281996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 0x7f5c28199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 -- 172.21.15.46:0/3611434127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c28073d50 msgr2=0x7f5c281996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c28073d50 0x7f5c281996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 -- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 msgr2=0x7f5c281a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c27e00640 1 -- 172.21.15.46:0/3611434127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c281a12f0 con 0x7f5c28075330 2024-03-08T23:07:16.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.216+0000 7f5c2d600640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c27e00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 0x7f5c28199c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400e9f0 tx=0x7f5c1400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1400cd80 con 0x7f5c28075330 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c281a15e0 con 0x7f5c28075330 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c140035f0 con 0x7f5c28075330 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14010670 con 0x7f5c28075330 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.217+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c281a1b80 con 0x7f5c28075330 2024-03-08T23:07:16.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.218+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c28107de0 con 0x7f5c28075330 2024-03-08T23:07:16.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.221+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5c140037a0 con 0x7f5c28075330 2024-03-08T23:07:16.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.222+0000 7f5c25600640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 0x7f5c100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.222+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f5c140cf0b0 con 0x7f5c28075330 2024-03-08T23:07:16.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.222+0000 7f5c2cc00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 0x7f5c100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.223+0000 7f5c2cc00640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 0x7f5c100a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c02c7b0 tx=0x7f5c1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:16.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.224+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5c1409bd70 con 0x7f5c28075330 2024-03-08T23:07:16.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.425+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5c2802ce70 con 0x7f5c28075330 2024-03-08T23:07:16.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.426+0000 7f5c25600640 1 -- 172.21.15.46:0/3611434127 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5c1409b670 con 0x7f5c28075330 2024-03-08T23:07:16.429 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.428+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 msgr2=0x7f5c100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.428+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 0x7f5c100a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5c1c02c7b0 tx=0x7f5c1c005fd0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.428+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 msgr2=0x7f5c28199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.428+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 0x7f5c28199c10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5c1400e9f0 tx=0x7f5c1400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 shutdown_connections 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f5c100a2800 0x7f5c100a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c281090b0 0x7f5c281a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c28075330 0x7f5c28199c10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 --2- 172.21.15.46:0/3611434127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c28073d50 0x7f5c281996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 >> 172.21.15.46:0/3611434127 conn(0x7f5c280fc760 msgr2=0x7f5c28104fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 shutdown_connections 2024-03-08T23:07:16.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.429+0000 7f5c2e000640 1 -- 172.21.15.46:0/3611434127 wait complete. 2024-03-08T23:07:16.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:07:16.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:16 smithi046 ceph-mon[31859]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:16.536 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:16 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1976138714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:16 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/724743646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.537 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:16 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3611434127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 -- 172.21.15.46:0/3538526897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 msgr2=0x7fcda0104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 --2- 172.21.15.46:0/3538526897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0104ba0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009900 tx=0x7fcd9002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 -- 172.21.15.46:0/3538526897 shutdown_connections 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 --2- 172.21.15.46:0/3538526897 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 0x7fcda010b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 --2- 172.21.15.46:0/3538526897 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0104ba0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 --2- 172.21.15.46:0/3538526897 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 -- 172.21.15.46:0/3538526897 >> 172.21.15.46:0/3538526897 conn(0x7fcda00feae0 msgr2=0x7fcda0100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 -- 172.21.15.46:0/3538526897 shutdown_connections 2024-03-08T23:07:16.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.675+0000 7fcda7400640 1 -- 172.21.15.46:0/3538526897 wait complete. 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 Processor -- start 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 -- start start 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 0x7fcda01a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda010f210 con 0x7fcda0104720 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda010f090 con 0x7fcda01050e0 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda7400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda010f390 con 0x7fcda0103530 2024-03-08T23:07:16.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcda4a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.677+0000 7fcd97e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37564/0 (socket says 172.21.15.46:37564) 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 -- 172.21.15.46:0/1652266032 learned_addr learned my addr 172.21.15.46:0/1652266032 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda5400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 0x7fcda01a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 -- 172.21.15.46:0/1652266032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 msgr2=0x7fcda01a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 0x7fcda01a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 -- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 msgr2=0x7fcda0199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 -- 172.21.15.46:0/1652266032 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda01a1450 con 0x7fcda0103530 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcd97e00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:16.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.678+0000 7fcda4a00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800e9c0 tx=0x7fcd8800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.679+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8800cd80 con 0x7fcda0103530 2024-03-08T23:07:16.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.679+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda01a1740 con 0x7fcda0103530 2024-03-08T23:07:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.679+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd880035f0 con 0x7fcda0103530 2024-03-08T23:07:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.679+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd88010690 con 0x7fcda0103530 2024-03-08T23:07:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.679+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda01a1c60 con 0x7fcda0103530 2024-03-08T23:07:16.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.680+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda010a850 con 0x7fcda0103530 2024-03-08T23:07:16.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.682+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcd880037a0 con 0x7fcda0103530 2024-03-08T23:07:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.683+0000 7fcd9e000640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 0x7fcd800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.684+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fcd880cf270 con 0x7fcda0103530 2024-03-08T23:07:16.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.684+0000 7fcd97e00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 0x7fcd800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:16.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.684+0000 7fcd97e00640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 0x7fcd800a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd9002c240 tx=0x7fcd9002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:16.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.686+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcd8809bf30 con 0x7fcda0103530 2024-03-08T23:07:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:16 smithi192 ceph-mon[36700]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:16 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1976138714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:16 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/724743646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:16 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3611434127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:16 smithi062 ceph-mon[35761]: pgmap v401: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-08T23:07:16.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:16 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1976138714' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:16 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/724743646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:16 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3611434127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:16.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.930+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcda002ce70 con 0x7fcda0103530 2024-03-08T23:07:16.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.931+0000 7fcd9e000640 1 -- 172.21.15.46:0/1652266032 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fcd8809b830 con 0x7fcda0103530 2024-03-08T23:07:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:16.935 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:05:06.080225+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:07:16.935 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 23 2024-03-08T23:07:16.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 msgr2=0x7fcd800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 0x7fcd800a4cc0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcd9002c240 tx=0x7fcd9002df90 comp rx=0 tx=0).stop 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 msgr2=0x7fcda0199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fcd8800e9c0 tx=0x7fcd8800ee80 comp rx=0 tx=0).stop 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 shutdown_connections 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fcd800a2800 0x7fcd800a4cc0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcda01050e0 0x7fcda01a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcda0104720 0x7fcda0199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 --2- 172.21.15.46:0/1652266032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcda0103530 0x7fcda0199830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:16.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 >> 172.21.15.46:0/1652266032 conn(0x7fcda00feae0 msgr2=0x7fcda0107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:16.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 shutdown_connections 2024-03-08T23:07:16.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:16.934+0000 7fcda7400640 1 -- 172.21.15.46:0/1652266032 wait complete. 2024-03-08T23:07:16.957 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 -- 172.21.15.46:0/2485637052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc900073d50 msgr2=0x7fc9000741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 --2- 172.21.15.46:0/2485637052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc900073d50 0x7fc9000741d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec009930 tx=0x7fc8ec02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 -- 172.21.15.46:0/2485637052 shutdown_connections 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 --2- 172.21.15.46:0/2485637052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc90010d8e0 0x7fc90010fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 --2- 172.21.15.46:0/2485637052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc900073d50 0x7fc9000741d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 --2- 172.21.15.46:0/2485637052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 0x7fc900073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 -- 172.21.15.46:0/2485637052 >> 172.21.15.46:0/2485637052 conn(0x7fc9000fc8a0 msgr2=0x7fc9000fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 -- 172.21.15.46:0/2485637052 shutdown_connections 2024-03-08T23:07:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.178+0000 7fc908200640 1 -- 172.21.15.46:0/2485637052 wait complete. 2024-03-08T23:07:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.179+0000 7fc908200640 1 Processor -- start 2024-03-08T23:07:17.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 -- start start 2024-03-08T23:07:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 0x7fc90019dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 0x7fc90019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001135b0 con 0x7fc90010d8e0 2024-03-08T23:07:17.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900113430 con 0x7fc900073d50 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc908200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900113730 con 0x7fc900075330 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc906200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc905800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 0x7fc90019dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc906200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39246/0 (socket says 172.21.15.46:39246) 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.180+0000 7fc906200640 1 -- 172.21.15.46:0/2581903977 learned_addr learned my addr 172.21.15.46:0/2581903977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc904e00640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 0x7fc90019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 -- 172.21.15.46:0/2581903977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 msgr2=0x7fc90019e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 0x7fc90019e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 -- 172.21.15.46:0/2581903977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 msgr2=0x7fc90019dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 0x7fc90019dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 -- 172.21.15.46:0/2581903977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9001a5870 con 0x7fc90010d8e0 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc905800640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 0x7fc90019dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:17.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.181+0000 7fc906200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800eb10 tx=0x7fc8f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.182+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8004b20 con 0x7fc90010d8e0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.182+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8f8004cc0 con 0x7fc90010d8e0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.182+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8010760 con 0x7fc90010d8e0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.182+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9001a5b60 con 0x7fc90010d8e0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.182+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9001a6060 con 0x7fc90010d8e0 2024-03-08T23:07:17.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.183+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900103ab0 con 0x7fc90010d8e0 2024-03-08T23:07:17.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.185+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc8f8005020 con 0x7fc90010d8e0 2024-03-08T23:07:17.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.186+0000 7fc8fe000640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 0x7fc8e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.186+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fc8f80cf3e0 con 0x7fc90010d8e0 2024-03-08T23:07:17.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.187+0000 7fc905800640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 0x7fc8e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.187+0000 7fc905800640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 0x7fc8e80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc90019f130 tx=0x7fc8f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.191+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc8f809c100 con 0x7fc90010d8e0 2024-03-08T23:07:17.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.399+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc900061f90 con 0x7fc90010d8e0 2024-03-08T23:07:17.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.402+0000 7fc8fe000640 1 -- 172.21.15.46:0/2581903977 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7fc8f809ba00 con 0x7fc90010d8e0 2024-03-08T23:07:17.405 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:17.405 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 msgr2=0x7fc8e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 0x7fc8e80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc90019f130 tx=0x7fc8f4005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 msgr2=0x7fc9001a51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800eb10 tx=0x7fc8f800c2d0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 shutdown_connections 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc8e80a2800 0x7fc8e80a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc90010d8e0 0x7fc9001a51d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc900075330 0x7fc90019e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 --2- 172.21.15.46:0/2581903977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc900073d50 0x7fc90019dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 >> 172.21.15.46:0/2581903977 conn(0x7fc9000fc8a0 msgr2=0x7fc900100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 shutdown_connections 2024-03-08T23:07:17.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.405+0000 7fc908200640 1 -- 172.21.15.46:0/2581903977 wait complete. 2024-03-08T23:07:17.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-08T23:04:38.798478+0000', 'modified': '2024-03-08T23:05:06.080225+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24578, 'mds_1': 24560, 'mds_2': 24553}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24520': {'gid': 24520, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/3258548049', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 3258548049}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 3258548049}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.62:6837/2774760539', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6836', 'nonce': 2774760539}, {'type': 'v1', 'addr': '172.21.15.62:6837', 'nonce': 2774760539}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24532': {'gid': 24532, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6837/3408320280', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 3408320280}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 3408320280}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.192:6839/441315746', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 441315746}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 441315746}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24560': {'gid': 24560, 'name': 'a', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.46:6841/2979513578', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.46:6840', 'nonce': 2979513578}, {'type': 'v1', 'addr': '172.21.15.46:6841', 'nonce': 2979513578}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24578': {'gid': 24578, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.62:6839/1938633866', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.62:6838', 'nonce': 1938633866}, {'type': 'v1', 'addr': '172.21.15.62:6839', 'nonce': 1938633866}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-08T23:07:17.429 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-03-08T23:07:17.429 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-08T23:07:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 -- 172.21.15.46:0/4208760171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 msgr2=0x7fda58103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 --2- 172.21.15.46:0/4208760171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58103930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fda4c009900 tx=0x7fda4c02bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:17.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 -- 172.21.15.46:0/4208760171 shutdown_connections 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 --2- 172.21.15.46:0/4208760171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 0x7fda5810b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 --2- 172.21.15.46:0/4208760171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 --2- 172.21.15.46:0/4208760171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58103930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 -- 172.21.15.46:0/4208760171 >> 172.21.15.46:0/4208760171 conn(0x7fda580feaa0 msgr2=0x7fda58100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 -- 172.21.15.46:0/4208760171 shutdown_connections 2024-03-08T23:07:17.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.649+0000 7fda60200640 1 -- 172.21.15.46:0/4208760171 wait complete. 2024-03-08T23:07:17.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.650+0000 7fda60200640 1 Processor -- start 2024-03-08T23:07:17.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.650+0000 7fda60200640 1 -- start start 2024-03-08T23:07:17.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 0x7fda581a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5810f1c0 con 0x7fda58103510 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5810f040 con 0x7fda581050c0 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda60200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda5810f340 con 0x7fda58104700 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39262/0 (socket says 172.21.15.46:39262) 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 -- 172.21.15.46:0/2110577592 learned_addr learned my addr 172.21.15.46:0/2110577592 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5e200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 0x7fda581a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5ce00640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 -- 172.21.15.46:0/2110577592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 msgr2=0x7fda58199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 -- 172.21.15.46:0/2110577592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 msgr2=0x7fda581a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 0x7fda581a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5d800640 1 -- 172.21.15.46:0/2110577592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda581a14a0 con 0x7fda58103510 2024-03-08T23:07:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.651+0000 7fda5ce00640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:17.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda5d800640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02efd0 tx=0x7fda4c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda56000640 1 -- 172.21.15.46:0/2110577592 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c041070 con 0x7fda58103510 2024-03-08T23:07:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda56000640 1 -- 172.21.15.46:0/2110577592 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda4c0044d0 con 0x7fda58103510 2024-03-08T23:07:17.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda581a1730 con 0x7fda58103510 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda56000640 1 -- 172.21.15.46:0/2110577592 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c003e50 con 0x7fda58103510 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.652+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda581a1bf0 con 0x7fda58103510 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.655+0000 7fda62e3a740 1 Processor -- start 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 -- start start 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda56000640 1 -- 172.21.15.46:0/2110577592 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fda4c044070 con 0x7fda58103510 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576a406f0 con 0x55d576adbf80 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5769e99e0 con 0x55d576adb590 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda62e3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576a3c790 con 0x55d576ae10f0 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda5d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda5e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda5e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35006/0 (socket says 172.21.15.46:35006) 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.656+0000 7fda5e200640 1 -- 172.21.15.46:0/4256999876 learned_addr learned my addr 172.21.15.46:0/4256999876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5ce00640 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 -- 172.21.15.46:0/4256999876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 msgr2=0x55d576ae3580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae3580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 -- 172.21.15.46:0/4256999876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 msgr2=0x55d576ae0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 -- 172.21.15.46:0/4256999876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d5769dd720 con 0x55d576adb590 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda5e200640 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda50009f60 tx=0x7fda50009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4d1796435aeef57 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.657+0000 7fda4aa00640 1 -- 172.21.15.46:0/4256999876 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5001e070 con 0x55d576adb590 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda4aa00640 1 -- 172.21.15.46:0/4256999876 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda50003c30 con 0x55d576adb590 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda4aa00640 1 -- 172.21.15.46:0/4256999876 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5000fa20 con 0x55d576adb590 2024-03-08T23:07:17.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda56000640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 0x7fda400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda56000640 1 -- 172.21.15.46:0/2110577592 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fda4c0f0a90 con 0x7fda58103510 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda5d800640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 0x7fda400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 -- 172.21.15.46:0/4256999876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 msgr2=0x55d576adb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fda50009f60 tx=0x7fda50009f90 comp rx=0 tx=0).stop 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 -- 172.21.15.46:0/4256999876 shutdown_connections 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae3580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4256999876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576adb9b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.658+0000 7fda62e3a740 1 -- 172.21.15.46:0/4256999876 >> 172.21.15.46:0/4256999876 conn(0x55d576ad6b00 msgr2=0x55d576ad6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.659+0000 7fda62e3a740 1 -- 172.21.15.46:0/4256999876 shutdown_connections 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.659+0000 7fda62e3a740 1 -- 172.21.15.46:0/4256999876 wait complete. 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.659+0000 7fda5d800640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 0x7fda400a4e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0b94b0 tx=0x7fda4c0f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.659+0000 7fda62e3a740 1 -- start start 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.659+0000 7fda62e3a740 1 Processor -- start 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ae5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576aecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576ae40a0 con 0x55d576adbf80 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d5769fce20 con 0x55d576adb590 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576ae4220 con 0x55d576ae10f0 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda62e3a740 10 client.0 _ll_register_callbacks cb 0x55d576a378d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39280/0 (socket says 172.21.15.46:39280) 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda5ce00640 1 -- 172.21.15.46:0/3960719886 learned_addr learned my addr 172.21.15.46:0/3960719886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda5e200640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ae5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.660+0000 7fda5e200640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576aecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 -- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 msgr2=0x55d576aecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576aecd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 -- 172.21.15.46:0/3960719886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 msgr2=0x55d576ae5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ae5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 -- 172.21.15.46:0/3960719886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d5767552d0 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5e200640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ae5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda5ce00640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fda4400bad0 tx=0x7fda4400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fb6c23ad03a10e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda3f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4401b640 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fda4401b7e0 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.661+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda50003c30 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda44024050 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 10 client.15216 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d57681f6b0 con 0x55d576adbf80 2024-03-08T23:07:17.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda50003c30 con 0x55d576adbf80 2024-03-08T23:07:17.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fda44019020 con 0x55d576adbf80 2024-03-08T23:07:17.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 10 client.15216 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 10 client.15216 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 20 client.15216 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576af23b0 0x55d576af1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 4 client.15216 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.662+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d576a4dc30 con 0x55d576af23b0 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.663+0000 7fda5e200640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576af23b0 0x55d576af1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.663+0000 7fda5e200640 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576af23b0 0x55d576af1430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1e5bca5abf157019 server_cookie=55f64fade3fa712 in_seq=0 out_seq=0 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.663+0000 7fda3f400640 10 client.15216 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda3f400640 1 -- 172.21.15.46:0/3960719886 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d576a4dc30 con 0x55d576af23b0 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda3f400640 10 client.15216 handle_command_reply: tid=0 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda62e3a740 1 client.15216 shutdown 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 msgr2=0x55d576ae5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fda4400bad0 tx=0x7fda4400bf90 comp rx=0 tx=0).stop 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.664+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 shutdown_connections 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576af23b0 0x55d576af1430 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576aecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576ae5cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 --2- 172.21.15.46:0/3960719886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ae5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda3f400640 0 client.15216 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 >> 172.21.15.46:0/3960719886 conn(0x55d576ad6b00 msgr2=0x55d576ad5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 shutdown_connections 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 1 -- 172.21.15.46:0/3960719886 wait complete. 2024-03-08T23:07:17.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.665+0000 7fda62e3a740 20 client.15216 trim_cache size 0 max 16384 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 Processor -- start 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 -- start start 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ad18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576aeccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae5390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af1bd0 con 0x55d576adbf80 2024-03-08T23:07:17.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af2230 con 0x55d576adb590 2024-03-08T23:07:17.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.682+0000 7fda62e3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af2890 con 0x55d576ae10f0 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576aeccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ad18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576aeccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39292/0 (socket says 172.21.15.46:39292) 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5ce00640 1 -- 172.21.15.46:0/635801241 learned_addr learned my addr 172.21.15.46:0/635801241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 -- 172.21.15.46:0/635801241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 msgr2=0x55d576ae5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 -- 172.21.15.46:0/635801241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 msgr2=0x55d576aeccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.683+0000 7fda5e200640 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576aeccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.684+0000 7fda5e200640 1 -- 172.21.15.46:0/635801241 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d576adc810 con 0x55d576adb590 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.684+0000 7fda5e200640 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ad18b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda50009f30 tx=0x7fda50009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a69bd865eca11e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.684+0000 7fda4a000640 1 -- 172.21.15.46:0/635801241 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5001e040 con 0x55d576adb590 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.684+0000 7fda4a000640 1 -- 172.21.15.46:0/635801241 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda50004050 con 0x55d576adb590 2024-03-08T23:07:17.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.684+0000 7fda4a000640 1 -- 172.21.15.46:0/635801241 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5002a4c0 con 0x55d576adb590 2024-03-08T23:07:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- 172.21.15.46:0/635801241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 msgr2=0x55d576ad18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ad18b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fda50009f30 tx=0x7fda50009500 comp rx=0 tx=0).stop 2024-03-08T23:07:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- 172.21.15.46:0/635801241 shutdown_connections 2024-03-08T23:07:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576ae10f0 0x55d576ae5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576adbf80 0x55d576aeccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 --2- 172.21.15.46:0/635801241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adb590 0x55d576ad18b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- 172.21.15.46:0/635801241 >> 172.21.15.46:0/635801241 conn(0x55d576ad6b00 msgr2=0x55d576ad6ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- 172.21.15.46:0/635801241 shutdown_connections 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- 172.21.15.46:0/635801241 wait complete. 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.685+0000 7fda62e3a740 1 -- start start 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 Processor -- start 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576adb590 0x55d576ae9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576ae10f0 0x55d576aeb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af3550 con 0x55d576ae10f0 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af2ef0 con 0x55d576adbf80 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d576af3bb0 con 0x55d576adb590 2024-03-08T23:07:17.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.686+0000 7fda62e3a740 10 client.0 _ll_register_callbacks cb 0x7fda50009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35030/0 (socket says 172.21.15.46:35030) 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 -- 172.21.15.46:0/4250584099 learned_addr learned my addr 172.21.15.46:0/4250584099 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5e200640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576adb590 0x55d576ae9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5e200640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576ae10f0 0x55d576aeb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 -- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576adb590 msgr2=0x55d576ae9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576adb590 0x55d576ae9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 -- 172.21.15.46:0/4250584099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576ae10f0 msgr2=0x55d576aeb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576ae10f0 0x55d576aeb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 -- 172.21.15.46:0/4250584099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d576aeb900 con 0x55d576adbf80 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda5ce00640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fda440059f0 tx=0x7fda44004070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b97887714a2906bf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.687+0000 7fda3f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:07:17.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda44011650 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda50004050 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fda440117f0 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda44039050 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda62e3a740 10 client.25328 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55d576aecca0 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda50004050 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fda44019040 con 0x55d576adbf80 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.688+0000 7fda62e3a740 10 client.25328 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda62e3a740 10 client.25328 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda62e3a740 20 client.25328 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576b4e020 0x55d576b49fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda62e3a740 4 client.25328 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:07:17.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d576a47f80 con 0x55d576b4e020 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.689+0000 7fda5e200640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576b4e020 0x55d576b49fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda5e200640 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576b4e020 0x55d576b49fc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60cb9b302b461768 server_cookie=39317e215bda5826 in_seq=0 out_seq=0 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda3f400640 10 client.25328 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda3f400640 1 -- 172.21.15.46:0/4250584099 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55d576a47f80 con 0x55d576b4e020 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda3f400640 10 client.25328 handle_command_reply: tid=0 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 client.25328 shutdown 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 msgr2=0x55d576aea3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fda440059f0 tx=0x7fda44004070 comp rx=0 tx=0).stop 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 shutdown_connections 2024-03-08T23:07:17.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d576b4e020 0x55d576b49fc0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d576ae10f0 0x55d576aeb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda3f400640 0 client.25328 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d576adbf80 0x55d576aea3c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 --2- 172.21.15.46:0/4250584099 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d576adb590 0x55d576ae9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 >> 172.21.15.46:0/4250584099 conn(0x55d576ad6b00 msgr2=0x55d576ad18b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 shutdown_connections 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 1 -- 172.21.15.46:0/4250584099 wait complete. 2024-03-08T23:07:17.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.690+0000 7fda62e3a740 20 client.25328 trim_cache size 0 max 16384 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 2, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24553, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 132.618538828, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:07:17.698 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 177.00288604799999 2024-03-08T23:07:17.699 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:07:17.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 msgr2=0x7fda400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 0x7fda400a4e60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fda4c0b94b0 tx=0x7fda4c0f6530 comp rx=0 tx=0).stop 2024-03-08T23:07:17.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 msgr2=0x7fda58199880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fda4c02efd0 tx=0x7fda4c005d50 comp rx=0 tx=0).stop 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 shutdown_connections 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fda400a29a0 0x7fda400a4e60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda581050c0 0x7fda581a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda58104700 0x7fda58199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 --2- 172.21.15.46:0/2110577592 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda58103510 0x7fda58199880 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 >> 172.21.15.46:0/2110577592 conn(0x7fda580feaa0 msgr2=0x7fda58107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 shutdown_connections 2024-03-08T23:07:17.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.691+0000 7fda60200640 1 -- 172.21.15.46:0/2110577592 wait complete. 2024-03-08T23:07:17.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:17 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1652266032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.709 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:17 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2581903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-08T23:07:17.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:17 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1652266032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:17 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2581903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:17 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1652266032' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.892 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:17 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2581903977' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:17.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 -- 172.21.15.46:0/2392188969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81050e0 msgr2=0x7f14a810b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 --2- 172.21.15.46:0/2392188969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81050e0 0x7f14a810b970 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f14a0009860 tx=0x7f14a002bb70 comp rx=0 tx=0).stop 2024-03-08T23:07:17.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 -- 172.21.15.46:0/2392188969 shutdown_connections 2024-03-08T23:07:17.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 --2- 172.21.15.46:0/2392188969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a81050e0 0x7f14a810b970 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 --2- 172.21.15.46:0/2392188969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a8104720 0x7f14a8104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 --2- 172.21.15.46:0/2392188969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8103530 0x7f14a8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 -- 172.21.15.46:0/2392188969 >> 172.21.15.46:0/2392188969 conn(0x7f14a80feaa0 msgr2=0x7f14a8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 -- 172.21.15.46:0/2392188969 shutdown_connections 2024-03-08T23:07:17.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.934+0000 7f14af800640 1 -- 172.21.15.46:0/2392188969 wait complete. 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.935+0000 7f14af800640 1 Processor -- start 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.935+0000 7f14af800640 1 -- start start 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8104720 0x7f14a8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 0x7f14a81a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a810f190 con 0x7f14a8103530 2024-03-08T23:07:17.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a810f010 con 0x7f14a8104720 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14af800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a810f310 con 0x7f14a81050e0 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39320/0 (socket says 172.21.15.46:39320) 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 -- 172.21.15.46:0/2922705516 learned_addr learned my addr 172.21.15.46:0/2922705516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14a7e00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8104720 0x7f14a8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ad800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 0x7f14a81a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 -- 172.21.15.46:0/2922705516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 msgr2=0x7f14a81a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 0x7f14a81a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 -- 172.21.15.46:0/2922705516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8104720 msgr2=0x7f14a8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.936+0000 7f14ace00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8104720 0x7f14a8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14ace00640 1 -- 172.21.15.46:0/2922705516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a81a13f0 con 0x7f14a8103530 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14ad800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 0x7f14a81a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:17.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14ace00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f149800e9f0 tx=0x7f149800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14a5600640 1 -- 172.21.15.46:0/2922705516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800ccc0 con 0x7f14a8103530 2024-03-08T23:07:17.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14a5600640 1 -- 172.21.15.46:0/2922705516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14980035f0 con 0x7f14a8103530 2024-03-08T23:07:17.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a81a16e0 con 0x7f14a8103530 2024-03-08T23:07:17.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14a5600640 1 -- 172.21.15.46:0/2922705516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149800ce60 con 0x7f14a8103530 2024-03-08T23:07:17.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.937+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a81a1c80 con 0x7f14a8103530 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 Processor -- start 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14a5600640 1 -- 172.21.15.46:0/2922705516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f14980037a0 con 0x7f14a8103530 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 -- start start 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c7fba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c7fe580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c75b6f0 con 0x562b1c7f6590 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c7049e0 con 0x562b1c7fc0f0 2024-03-08T23:07:17.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14b259e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c757790 con 0x562b1c7f6f80 2024-03-08T23:07:17.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14ace00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c7fba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39328/0 (socket says 172.21.15.46:39328) 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 -- 172.21.15.46:0/227438436 learned_addr learned my addr 172.21.15.46:0/227438436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.941+0000 7f14a7e00640 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c7fe580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 -- 172.21.15.46:0/227438436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 msgr2=0x562b1c7fba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c7fba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 -- 172.21.15.46:0/227438436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 msgr2=0x562b1c7fe580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c7fe580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 -- 172.21.15.46:0/227438436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b1c6f8720 con 0x562b1c7f6590 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ace00640 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c7fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.942+0000 7f14ad800640 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14a00349e0 tx=0x7f14a0005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3d769cf17970b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f14a5600640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 0x7f148c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f14a5600640 1 -- 172.21.15.46:0/2922705516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f14980cf070 con 0x7f14a8103530 2024-03-08T23:07:17.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f149e000640 1 -- 172.21.15.46:0/227438436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0039040 con 0x562b1c7f6590 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f149e000640 1 -- 172.21.15.46:0/227438436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14a0043070 con 0x562b1c7f6590 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f14ace00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 0x7f148c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.943+0000 7f149e000640 1 -- 172.21.15.46:0/227438436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0005240 con 0x562b1c7f6590 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14ace00640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 0x7f148c0a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f14980ccb60 tx=0x7f14980d2b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- 172.21.15.46:0/227438436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 msgr2=0x562b1c7f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f14a00349e0 tx=0x7f14a0005da0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- 172.21.15.46:0/227438436 shutdown_connections 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c7fe580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c7fba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 --2- 172.21.15.46:0/227438436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c7f69b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- 172.21.15.46:0/227438436 >> 172.21.15.46:0/227438436 conn(0x562b1c7f1b00 msgr2=0x562b1c7f1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- 172.21.15.46:0/227438436 shutdown_connections 2024-03-08T23:07:17.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- 172.21.15.46:0/227438436 wait complete. 2024-03-08T23:07:17.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 -- start start 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.944+0000 7f14b259e740 1 Processor -- start 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c800c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c807c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c7ff040 con 0x562b1c7f6590 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c717e20 con 0x562b1c7fc0f0 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c7ff1c0 con 0x562b1c7f6f80 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14b259e740 10 client.0 _ll_register_callbacks cb 0x562b1c7528d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39344/0 (socket says 172.21.15.46:39344) 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14ad800640 1 -- 172.21.15.46:0/2858163924 learned_addr learned my addr 172.21.15.46:0/2858163924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.945+0000 7f14a7e00640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c800c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c807c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 -- 172.21.15.46:0/2858163924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 msgr2=0x562b1c800c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c800c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 -- 172.21.15.46:0/2858163924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 msgr2=0x562b1c807c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c807c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 -- 172.21.15.46:0/2858163924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b1c4702d0 con 0x562b1c7f6590 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14ad800640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f14a002c2e0 tx=0x7f14a002c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47769c0e5a56d28 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f1496a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0052060 con 0x562b1c7f6590 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f14a0005040 con 0x562b1c7f6590 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.946+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14a0043070 con 0x562b1c7f6590 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a0059050 con 0x562b1c7f6590 2024-03-08T23:07:17.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 10 client.15234 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562b1c53a6b0 con 0x562b1c7f6590 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14a0043070 con 0x562b1c7f6590 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f14a00046c0 con 0x562b1c7f6590 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 10 client.15234 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 10 client.15234 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 20 client.15234 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c80c150 0x562b1c8111b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 4 client.15234 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.947+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b1c768c30 con 0x562b1c80c150 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.948+0000 7f14a7e00640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c80c150 0x562b1c8111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.948+0000 7f14a7e00640 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c80c150 0x562b1c8111b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c8aeb53fe94382d7 server_cookie=75b56554594b48e9 in_seq=0 out_seq=0 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.948+0000 7f1496a00640 10 client.15234 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f1496a00640 1 -- 172.21.15.46:0/2858163924 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b1c768c30 con 0x562b1c80c150 2024-03-08T23:07:17.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f1496a00640 10 client.15234 handle_command_reply: tid=0 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 client.15234 shutdown 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 msgr2=0x562b1c8006e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f14a002c2e0 tx=0x7f14a002c270 comp rx=0 tx=0).stop 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 shutdown_connections 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7fc0f0 0x562b1c807c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c80c150 0x562b1c8111b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6f80 0x562b1c800c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 --2- 172.21.15.46:0/2858163924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6590 0x562b1c8006e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f1496a00640 0 client.15234 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 >> 172.21.15.46:0/2858163924 conn(0x562b1c7f1b00 msgr2=0x562b1c7f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 shutdown_connections 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.949+0000 7f14b259e740 1 -- 172.21.15.46:0/2858163924 wait complete. 2024-03-08T23:07:17.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.950+0000 7f14b259e740 20 client.15234 trim_cache size 0 max 16384 2024-03-08T23:07:17.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 Processor -- start 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 -- start start 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6590 0x562b1c7ec8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 0x562b1c8151d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80c550 con 0x562b1c7f6f80 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80cbb0 con 0x562b1c7f6590 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.967+0000 7f14b259e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80d210 con 0x562b1c7fc0f0 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14a7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14a7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39346/0 (socket says 172.21.15.46:39346) 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 0x562b1c8151d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14a7e00640 1 -- 172.21.15.46:0/218540727 learned_addr learned my addr 172.21.15.46:0/218540727 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6590 0x562b1c7ec8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 -- 172.21.15.46:0/218540727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6590 msgr2=0x562b1c7ec8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6590 0x562b1c7ec8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 -- 172.21.15.46:0/218540727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 msgr2=0x562b1c807c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14ad800640 1 -- 172.21.15.46:0/218540727 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b1c7f2870 con 0x562b1c7fc0f0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.968+0000 7f14a7e00640 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f14ad800640 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 0x562b1c8151d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f14a0004b60 tx=0x7f14a0034ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dbef9d3f30c6581 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f149d600640 1 -- 172.21.15.46:0/218540727 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0039070 con 0x562b1c7fc0f0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f149d600640 1 -- 172.21.15.46:0/218540727 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14a0043040 con 0x562b1c7fc0f0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f149d600640 1 -- 172.21.15.46:0/218540727 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a00328a0 con 0x562b1c7fc0f0 2024-03-08T23:07:17.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f14b259e740 1 -- 172.21.15.46:0/218540727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 msgr2=0x562b1c8151d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f14b259e740 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 0x562b1c8151d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f14a0004b60 tx=0x7f14a0034ab0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.969+0000 7f14b259e740 1 -- 172.21.15.46:0/218540727 shutdown_connections 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7fc0f0 0x562b1c8151d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7f6f80 0x562b1c807c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 --2- 172.21.15.46:0/218540727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6590 0x562b1c7ec8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 -- 172.21.15.46:0/218540727 >> 172.21.15.46:0/218540727 conn(0x562b1c7f1b00 msgr2=0x562b1c7f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 -- 172.21.15.46:0/218540727 shutdown_connections 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 -- 172.21.15.46:0/218540727 wait complete. 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 -- start start 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.970+0000 7f14b259e740 1 Processor -- start 2024-03-08T23:07:17.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6590 0x562b1c804800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 0x562b1c804d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80ded0 con 0x562b1c7fc0f0 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80d870 con 0x562b1c7f6f80 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b1c80e530 con 0x562b1c7f6590 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14b259e740 10 client.0 _ll_register_callbacks cb 0x7f14a0037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39358/0 (socket says 172.21.15.46:39358) 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 -- 172.21.15.46:0/270396657 learned_addr learned my addr 172.21.15.46:0/270396657 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14a7e00640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 0x562b1c804d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6590 0x562b1c804800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 -- 172.21.15.46:0/270396657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6590 msgr2=0x562b1c804800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6590 0x562b1c804800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 -- 172.21.15.46:0/270396657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 msgr2=0x562b1c804d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 0x562b1c804d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14ad800640 1 -- 172.21.15.46:0/270396657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b1c807e10 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.971+0000 7f14a7e00640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 0x562b1c804d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f14ad800640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f14a00049e0 tx=0x7f14a002c270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a284af91e034e873 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f1496a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14a0062020 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f14a002c9f0 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14a0043040 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14a0039040 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f14b259e740 10 client.15240 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.972+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562b1c864320 con 0x562b1c7fc0f0 2024-03-08T23:07:17.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14a0043040 con 0x562b1c7fc0f0 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f14a00035f0 con 0x562b1c7fc0f0 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 10 client.15240 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 10 client.15240 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 20 client.15240 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c8683c0 0x562b1c864540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 4 client.15240 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "status"}) v1 -- 0x562b1c762f80 con 0x562b1c8683c0 2024-03-08T23:07:17.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.973+0000 7f14a7e00640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c8683c0 0x562b1c864540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14a7e00640 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c8683c0 0x562b1c864540 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3940e7a0434ad3ec server_cookie=90fb4f0682178ba3 in_seq=0 out_seq=0 2024-03-08T23:07:17.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f1496a00640 10 client.15240 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f1496a00640 1 -- 172.21.15.46:0/270396657 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x562b1c762f80 con 0x562b1c8683c0 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f1496a00640 10 client.15240 handle_command_reply: tid=0 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 client.15240 shutdown 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 msgr2=0x562b1c862e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f14a00049e0 tx=0x7f14a002c270 comp rx=0 tx=0).stop 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 shutdown_connections 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b1c7fc0f0 0x562b1c862e70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b1c7f6f80 0x562b1c804d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x562b1c8683c0 0x562b1c864540 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 --2- 172.21.15.46:0/270396657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b1c7f6590 0x562b1c804800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 >> 172.21.15.46:0/270396657 conn(0x562b1c7f1b00 msgr2=0x562b1c7f0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.974+0000 7f1496a00640 0 client.15240 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 shutdown_connections 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14b259e740 1 -- 172.21.15.46:0/270396657 wait complete. 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14b259e740 20 client.15240 trim_cache size 0 max 16384 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 1, 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24560, 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:07:17.983 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 133.912737784, 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 169.15759848499999 2024-03-08T23:07:17.984 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 msgr2=0x7f148c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 0x7f148c0a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f14980ccb60 tx=0x7f14980d2b50 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 msgr2=0x7f14a81997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.975+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f149800e9f0 tx=0x7f149800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 shutdown_connections 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f148c0a2880 0x7f148c0a4d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f14a81050e0 0x7f14a81a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f14a8104720 0x7f14a8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 --2- 172.21.15.46:0/2922705516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f14a8103530 0x7f14a81997d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:17.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 >> 172.21.15.46:0/2922705516 conn(0x7f14a80feaa0 msgr2=0x7f14a81079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 shutdown_connections 2024-03-08T23:07:17.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:17.976+0000 7f14af800640 1 -- 172.21.15.46:0/2922705516 wait complete. 2024-03-08T23:07:17.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-08T23:07:18.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 -- 172.21.15.46:0/1273353908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1004104700 msgr2=0x7f1004104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 --2- 172.21.15.46:0/1273353908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1004104700 0x7f1004104b80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0009900 tx=0x7f0ff002bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 -- 172.21.15.46:0/1273353908 shutdown_connections 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 --2- 172.21.15.46:0/1273353908 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f10041050c0 0x7f100410b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 --2- 172.21.15.46:0/1273353908 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1004104700 0x7f1004104b80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 --2- 172.21.15.46:0/1273353908 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004103510 0x7f1004103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.206+0000 7f1009c00640 1 -- 172.21.15.46:0/1273353908 >> 172.21.15.46:0/1273353908 conn(0x7f10040feaa0 msgr2=0x7f1004100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.207+0000 7f1009c00640 1 -- 172.21.15.46:0/1273353908 shutdown_connections 2024-03-08T23:07:18.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.207+0000 7f1009c00640 1 -- 172.21.15.46:0/1273353908 wait complete. 2024-03-08T23:07:18.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 Processor -- start 2024-03-08T23:07:18.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 -- start start 2024-03-08T23:07:18.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1004103510 0x7f10040790a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004104700 0x7f10040795e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100410f1f0 con 0x7f10041050c0 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100410f070 con 0x7f1004104700 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1009c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f100410f370 con 0x7f1004103510 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1003400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1003400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39376/0 (socket says 172.21.15.46:39376) 2024-03-08T23:07:18.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1002a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1004103510 0x7f10040790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.208+0000 7f1003400640 1 -- 172.21.15.46:0/3459566436 learned_addr learned my addr 172.21.15.46:0/3459566436 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1002000640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004104700 0x7f10040795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 -- 172.21.15.46:0/3459566436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1004103510 msgr2=0x7f10040790a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1004103510 0x7f10040790a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 -- 172.21.15.46:0/3459566436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004104700 msgr2=0x7f10040795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004104700 0x7f10040795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 -- 172.21.15.46:0/3459566436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10040764d0 con 0x7f10041050c0 2024-03-08T23:07:18.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f1003400640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00efd0 tx=0x7f0ffc00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f0ffb400640 1 -- 172.21.15.46:0/3459566436 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc003b70 con 0x7f10041050c0 2024-03-08T23:07:18.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.209+0000 7f0ffb400640 1 -- 172.21.15.46:0/3459566436 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ffc003d10 con 0x7f10041050c0 2024-03-08T23:07:18.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.210+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10041a5c50 con 0x7f10041050c0 2024-03-08T23:07:18.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.210+0000 7f0ffb400640 1 -- 172.21.15.46:0/3459566436 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc0109f0 con 0x7f10041050c0 2024-03-08T23:07:18.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.210+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10041a6000 con 0x7f10041050c0 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 Processor -- start 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f0ffb400640 1 -- 172.21.15.46:0/3459566436 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ffc004020 con 0x7f10041050c0 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 -- start start 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d595199b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59521580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5947e6f0 con 0x563d59519590 2024-03-08T23:07:18.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d594279e0 con 0x563d5951f0f0 2024-03-08T23:07:18.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.213+0000 7f100c99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5947a790 con 0x563d59519f80 2024-03-08T23:07:18.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d595199b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:37686/0 (socket says 172.21.15.46:37686) 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 -- 172.21.15.46:0/649422260 learned_addr learned my addr 172.21.15.46:0/649422260 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002000640 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d595199b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39382/0 (socket says 172.21.15.46:39382) 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002000640 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59521580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 -- 172.21.15.46:0/649422260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 msgr2=0x563d59521580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59521580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 -- 172.21.15.46:0/649422260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 msgr2=0x563d595199b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d595199b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.214+0000 7f1002a00640 1 -- 172.21.15.46:0/649422260 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d5941b720 con 0x563d59519f80 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f1002a00640 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009ca0 tx=0x7f0ff4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a5a4feb75014b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f0ffb400640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 0x7f0fe80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f0fefe00640 1 -- 172.21.15.46:0/649422260 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4004220 con 0x563d59519f80 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f0fefe00640 1 -- 172.21.15.46:0/649422260 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ff40043c0 con 0x563d59519f80 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f0fefe00640 1 -- 172.21.15.46:0/649422260 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff40198d0 con 0x563d59519f80 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f0ffb400640 1 -- 172.21.15.46:0/3459566436 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f0ffc0cf240 con 0x7f10041050c0 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.215+0000 7f1002000640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 0x7f0fe80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- 172.21.15.46:0/649422260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 msgr2=0x563d5951ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009ca0 tx=0x7f0ff4011810 comp rx=0 tx=0).stop 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- 172.21.15.46:0/649422260 shutdown_connections 2024-03-08T23:07:18.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59521580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519f80 0x563d5951ea80 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 --2- 172.21.15.46:0/649422260 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d595199b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- 172.21.15.46:0/649422260 >> 172.21.15.46:0/649422260 conn(0x563d59514b00 msgr2=0x563d59514f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- 172.21.15.46:0/649422260 shutdown_connections 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f1002000640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 0x7f0fe80a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002c250 tx=0x7f0ff0043040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- 172.21.15.46:0/649422260 wait complete. 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 -- start start 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.216+0000 7f100c99e740 1 Processor -- start 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519f80 0x563d59523c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d5952acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d595220a0 con 0x563d59519590 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5943ae20 con 0x563d59519f80 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d59522220 con 0x563d5951f0f0 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f100c99e740 10 client.0 _ll_register_callbacks cb 0x563d594758d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f1002a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f1002a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39388/0 (socket says 172.21.15.46:39388) 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f1002a00640 1 -- 172.21.15.46:0/1066689483 learned_addr learned my addr 172.21.15.46:0/1066689483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f1003400640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d5952acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.217+0000 7f1002000640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519f80 0x563d59523c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 -- 172.21.15.46:0/1066689483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 msgr2=0x563d5952acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d5952acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 -- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519f80 msgr2=0x563d59523c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519f80 0x563d59523c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 -- 172.21.15.46:0/1066689483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d591932d0 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f1002a00640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009380 tx=0x7f0ff4003800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56edbdf4c8ef2af server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f0fecc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff4028070 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f0ff4005040 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.218+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ff40043c0 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ff4034050 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 10 client.15252 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563d5925d6b0 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ff40043c0 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ff4019b40 con 0x563d59519590 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 10 client.15252 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 10 client.15252 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 20 client.15252 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d59530020 0x563d5952f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 4 client.15252 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:07:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.219+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563d5948bc30 con 0x563d59530020 2024-03-08T23:07:18.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.220+0000 7f1002000640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d59530020 0x563d5952f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.220+0000 7f1002000640 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d59530020 0x563d5952f270 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da4b960f47a6bf63 server_cookie=4c9847cb10c2f6d8 in_seq=0 out_seq=0 2024-03-08T23:07:18.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.220+0000 7f0fecc00640 10 client.15252 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f0fecc00640 1 -- 172.21.15.46:0/1066689483 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563d5948bc30 con 0x563d59530020 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f0fecc00640 10 client.15252 handle_command_reply: tid=0 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 client.15252 shutdown 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 msgr2=0x563d59523740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0ff4009380 tx=0x7f0ff4003800 comp rx=0 tx=0).stop 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 shutdown_connections 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d59530020 0x563d5952f270 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d5952acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519f80 0x563d59523c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 --2- 172.21.15.46:0/1066689483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519590 0x563d59523740 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f0fecc00640 0 client.15252 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:18.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.221+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 >> 172.21.15.46:0/1066689483 conn(0x563d59514b00 msgr2=0x563d59513c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.222+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 shutdown_connections 2024-03-08T23:07:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.222+0000 7f100c99e740 1 -- 172.21.15.46:0/1066689483 wait complete. 2024-03-08T23:07:18.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.222+0000 7f100c99e740 20 client.15252 trim_cache size 0 max 16384 2024-03-08T23:07:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 Processor -- start 2024-03-08T23:07:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 -- start start 2024-03-08T23:07:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 0x563d595133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d595138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5952f840 con 0x563d59519f80 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d5952fea0 con 0x563d59519590 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.239+0000 7f100c99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d59530500 con 0x563d5951f0f0 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39392/0 (socket says 172.21.15.46:39392) 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 -- 172.21.15.46:0/418826235 learned_addr learned my addr 172.21.15.46:0/418826235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002a00640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 0x563d595133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1003400640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d595138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 -- 172.21.15.46:0/418826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 msgr2=0x563d595138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d595138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 -- 172.21.15.46:0/418826235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 msgr2=0x563d595133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 0x563d595133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 -- 172.21.15.46:0/418826235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563d59524870 con 0x563d59519f80 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002a00640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 0x563d595133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.240+0000 7f1002000640 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0005ab0 tx=0x7f0ff0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cd85ef82f407d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f0fef400640 1 -- 172.21.15.46:0/418826235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff005c070 con 0x563d59519f80 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f0fef400640 1 -- 172.21.15.46:0/418826235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ff0056de0 con 0x563d59519f80 2024-03-08T23:07:18.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f0fef400640 1 -- 172.21.15.46:0/418826235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0004ed0 con 0x563d59519f80 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 -- 172.21.15.46:0/418826235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 msgr2=0x563d5952ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0005ab0 tx=0x7f0ff0000bc0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 -- 172.21.15.46:0/418826235 shutdown_connections 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d5951f0f0 0x563d595138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d5952ac80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 --2- 172.21.15.46:0/418826235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d59519590 0x563d595133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 -- 172.21.15.46:0/418826235 >> 172.21.15.46:0/418826235 conn(0x563d59514b00 msgr2=0x563d59514ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.241+0000 7f100c99e740 1 -- 172.21.15.46:0/418826235 shutdown_connections 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 -- 172.21.15.46:0/418826235 wait complete. 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 -- start start 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 Processor -- start 2024-03-08T23:07:18.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519590 0x563d59527b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.242+0000 7f100c99e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59585c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f100c99e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d595311c0 con 0x563d59519f80 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f100c99e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d59530b60 con 0x563d5951f0f0 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f100c99e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563d59531820 con 0x563d59519590 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f100c99e740 10 client.0 _ll_register_callbacks cb 0x7f0ff40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39396/0 (socket says 172.21.15.46:39396) 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 -- 172.21.15.46:0/3714496149 learned_addr learned my addr 172.21.15.46:0/3714496149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002a00640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519590 0x563d59527b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1003400640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59585c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 -- 172.21.15.46:0/3714496149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519590 msgr2=0x563d59527b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519590 0x563d59527b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 -- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 msgr2=0x563d59585c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59585c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.243+0000 7f1002000640 1 -- 172.21.15.46:0/3714496149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563d59586160 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f1002000640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002c180 tx=0x7f0ff0056c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca500b0870cbe9f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f0fecc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff005c070 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f0ff0056330 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ff0056de0 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ff0056530 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f100c99e740 10 client.15258 fetch_fsmap learned FSMap version 23 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x563d59587360 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.244+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ff0056de0 con 0x563d59519f80 2024-03-08T23:07:18.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 23) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ff0003680 con 0x563d59519f80 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 10 client.15258 fetch_fsmap finished waiting for FSMap version 23 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 10 client.15258 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 20 client.15258 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d5958b4d0 0x563d59587530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 4 client.15258 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563d59485f80 con 0x563d5958b4d0 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.245+0000 7f1002a00640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d5958b4d0 0x563d59587530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f1002a00640 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d5958b4d0 0x563d59587530 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb29a661ea310b80 server_cookie=5bb3a75770918a74 in_seq=0 out_seq=0 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f0fecc00640 10 client.15258 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:18.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f0fecc00640 1 -- 172.21.15.46:0/3714496149 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x563d59485f80 con 0x563d5958b4d0 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f0fecc00640 10 client.15258 handle_command_reply: tid=0 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 client.15258 shutdown 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 msgr2=0x563d59528090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002c180 tx=0x7f0ff0056c70 comp rx=0 tx=0).stop 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 shutdown_connections 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563d5958b4d0 0x563d59587530 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563d5951f0f0 0x563d59585c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f0fecc00640 0 client.15258 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:18.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563d59519f80 0x563d59528090 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 --2- 172.21.15.46:0/3714496149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563d59519590 0x563d59527b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 >> 172.21.15.46:0/3714496149 conn(0x563d59514b00 msgr2=0x563d595133b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 shutdown_connections 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.246+0000 7f100c99e740 1 -- 172.21.15.46:0/3714496149 wait complete. 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f100c99e740 20 client.15258 trim_cache size 0 max 16384 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stdout: "cluster_fsid": "ced7dd96-dd9e-11ee-95c6-87774f69a715", 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stdout: "whoami": 0, 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stdout: "id": 24578, 2024-03-08T23:07:18.257 INFO:teuthology.orchestra.run.smithi046.stdout: "want_state": "up:active", 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "state": "up:active", 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "fs_name": "cephfs", 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "rank_uptime": 158.89062862700001, 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "mdsmap_epoch": 23, 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch": 93, 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_epoch_barrier": 89, 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout: "uptime": 166.44516062400001 2024-03-08T23:07:18.258 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 msgr2=0x7f0fe80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 0x7f0fe80a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0ff002c250 tx=0x7f0ff0043040 comp rx=0 tx=0).stop 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 msgr2=0x7f1004075c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00efd0 tx=0x7f0ffc00c6a0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 shutdown_connections 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0fe80a2800 0x7f0fe80a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f10041050c0 0x7f1004075c10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1004104700 0x7f10040795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 --2- 172.21.15.46:0/3459566436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1004103510 0x7f10040790a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 >> 172.21.15.46:0/3459566436 conn(0x7f10040feaa0 msgr2=0x7f1004107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 shutdown_connections 2024-03-08T23:07:18.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:18.247+0000 7f1009c00640 1 -- 172.21.15.46:0/3459566436 wait complete. 2024-03-08T23:07:18.267 INFO:teuthology.run_tasks:Running task kclient... 2024-03-08T23:07:18.284 INFO:tasks.kclient:Mounting kernel clients... 2024-03-08T23:07:18.284 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-08T23:07:18.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:18.285 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-08T23:07:18.342 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-08T23:07:18.345 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:18.345 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-08T23:07:18.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:18.445 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-03-08T23:07:18.522 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:18 smithi046 ceph-mon[31859]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:18.576 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-08T23:07:18.576 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-08T23:07:18.577 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:18.577 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-08T23:07:18.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:18 smithi192 ceph-mon[36700]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:18.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:18 smithi062 ceph-mon[35761]: pgmap v402: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:07:19.037 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:07:20.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:20 smithi192 ceph-mon[36700]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:20.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:20 smithi062 ceph-mon[35761]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:20.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:20 smithi046 ceph-mon[31859]: pgmap v403: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:21.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 -- 172.21.15.46:0/310256762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102a40 msgr2=0x7f7d04106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:21.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 --2- 172.21.15.46:0/310256762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102a40 0x7f7d04106a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009920 tx=0x7f7cf0009c30 comp rx=0 tx=0).stop 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 -- 172.21.15.46:0/310256762 shutdown_connections 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 --2- 172.21.15.46:0/310256762 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 0x7f7d0410ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 --2- 172.21.15.46:0/310256762 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102090 0x7f7d04102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 --2- 172.21.15.46:0/310256762 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102a40 0x7f7d04106a80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 -- 172.21.15.46:0/310256762 >> 172.21.15.46:0/310256762 conn(0x7f7d04078f40 msgr2=0x7f7d04079350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.420+0000 7f7d08e00700 1 -- 172.21.15.46:0/310256762 shutdown_connections 2024-03-08T23:07:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.421+0000 7f7d08e00700 1 -- 172.21.15.46:0/310256762 wait complete. 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 Processor -- start 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 -- start start 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102090 0x7f7d041a0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 0x7f7d0419a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d04112170 con 0x7f7d04102a40 2024-03-08T23:07:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d04111ff0 con 0x7f7d04107160 2024-03-08T23:07:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d08e00700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d041122f0 con 0x7f7d04102090 2024-03-08T23:07:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.422+0000 7f7d01400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39416/0 (socket says 172.21.15.46:39416) 2024-03-08T23:07:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 -- 172.21.15.46:0/1843813194 learned_addr learned my addr 172.21.15.46:0/1843813194 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102090 0x7f7d041a0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d02800700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 0x7f7d0419a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 -- 172.21.15.46:0/1843813194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102090 msgr2=0x7f7d041a0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102090 0x7f7d041a0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 -- 172.21.15.46:0/1843813194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 msgr2=0x7f7d0419a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 0x7f7d0419a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 -- 172.21.15.46:0/1843813194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0419afb0 con 0x7f7d04102a40 2024-03-08T23:07:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.423+0000 7f7d01400700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000a010 tx=0x7f7cf000fa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.424+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0005180 con 0x7f7d04102a40 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.424+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cf000b560 con 0x7f7d04102a40 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.424+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0419b240 con 0x7f7d04102a40 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.424+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0003a00 con 0x7f7d04102a40 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.424+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d041a73b0 con 0x7f7d04102a40 2024-03-08T23:07:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.425+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d04103560 con 0x7f7d04102a40 2024-03-08T23:07:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.427+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7cf0003c00 con 0x7f7d04102a40 2024-03-08T23:07:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.428+0000 7f7cfaa00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 0x7f7ce80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.429+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f7cf00cfdf0 con 0x7f7d04102a40 2024-03-08T23:07:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.429+0000 7f7d01e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 0x7f7ce80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:21.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.429+0000 7f7d01e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 0x7f7ce80a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009750 tx=0x7f7cf4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.433+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7cf0013030 con 0x7f7d04102a40 2024-03-08T23:07:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.710+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7d041a7580 con 0x7f7d04102a40 2024-03-08T23:07:21.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.711+0000 7f7cfaa00700 1 -- 172.21.15.46:0/1843813194 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7cf009cb40 con 0x7f7d04102a40 2024-03-08T23:07:21.716 INFO:teuthology.orchestra.run.smithi046.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.714+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 msgr2=0x7f7ce80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.714+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 0x7f7ce80a4d50 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009750 tx=0x7f7cf4009310 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.714+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 msgr2=0x7f7d041a0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.714+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000a010 tx=0x7f7cf000fa70 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 shutdown_connections 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7ce80a2890 0x7f7ce80a4d50 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d04107160 0x7f7d0419a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d04102090 0x7f7d041a0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 --2- 172.21.15.46:0/1843813194 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d04102a40 0x7f7d041a0b40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:21.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 >> 172.21.15.46:0/1843813194 conn(0x7f7d04078f40 msgr2=0x7f7d040ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 shutdown_connections 2024-03-08T23:07:21.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:21.715+0000 7f7d08e00700 1 -- 172.21.15.46:0/1843813194 wait complete. 2024-03-08T23:07:22.587 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi046.front.sepia.ceph.com 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-08T23:07:22.588 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-08T23:07:22.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:22.588 DEBUG:teuthology.orchestra.run.smithi046:> ip addr 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: inet 127.0.0.1/8 scope host lo 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 ::1/128 scope host 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:10 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f0 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:07:22.647 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:6b:f6:11 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp7s0f1 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f8 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f0 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet 172.21.15.46/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft 41836sec preferred_lft 41836sec 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: inet6 fe80::ec4:7aff:febd:15f8/64 scope link noprefixroute 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: valid_lft forever preferred_lft forever 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: link/ether 0c:c4:7a:bd:15:f9 brd ff:ff:ff:ff:ff:ff 2024-03-08T23:07:22.648 INFO:teuthology.orchestra.run.smithi046.stdout: altname enp1s0f1 2024-03-08T23:07:22.648 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-08T23:07:22.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add name ceph-brx type bridge 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr flush dev ceph-brx 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set ceph-brx up 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-08T23:07:22.649 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:07:22.788 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:22 smithi046 ceph-mon[31859]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:22.788 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:22 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1843813194' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:07:22.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:22 smithi192 ceph-mon[36700]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:22.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:22 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1843813194' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:07:22.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:22.886 DEBUG:teuthology.orchestra.run.smithi046:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-08T23:07:22.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:22 smithi062 ceph-mon[35761]: pgmap v404: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-08T23:07:22.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:22 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1843813194' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-08T23:07:22.969 INFO:teuthology.orchestra.run.smithi046.stdout:1 2024-03-08T23:07:22.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:22.973 DEBUG:teuthology.orchestra.run.smithi046:> route 2024-03-08T23:07:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:Kernel IP routing table 2024-03-08T23:07:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-08T23:07:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-08T23:07:23.023 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-08T23:07:23.024 INFO:teuthology.orchestra.run.smithi046.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-08T23:07:23.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-08T23:07:23.024 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:07:23.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.214 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-08T23:07:23.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.270 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list-id 2024-03-08T23:07:23.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.323 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:07:23.323 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:07:23.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T23:07:23.323 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-08T23:07:23.323 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:07:23.480 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-08T23:07:23.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-08T23:07:23.480 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-08T23:07:23.481 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:07:23.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.726 DEBUG:teuthology.orchestra.run.smithi046:> (cd / && exec bash -c ' 2024-03-08T23:07:23.726 DEBUG:teuthology.orchestra.run.smithi046:> set -e 2024-03-08T23:07:23.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set brx.0 up 2024-03-08T23:07:23.726 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link set dev brx.0 master ceph-brx 2024-03-08T23:07:23.726 DEBUG:teuthology.orchestra.run.smithi046:> ') 2024-03-08T23:07:23.889 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T23:07:23.890 DEBUG:teuthology.orchestra.run.smithi046:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:23.941 INFO:teuthology.orchestra.run.smithi046.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-08T23:07:23.941 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-08T23:07:23.941 DEBUG:teuthology.orchestra.run.smithi046:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:23.993 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-08T23:07:23.993 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-08T23:07:23.994 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/fd24f677-ddfc-4a4a-a669-a06e16ce0e48 2024-03-08T23:07:23.994 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:23.994 DEBUG:teuthology.orchestra.run.smithi046:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/fd24f677-ddfc-4a4a-a669-a06e16ce0e48 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-08T23:07:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-08T23:07:24.138 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-03-08T23:07:24.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:24 smithi046 ceph-mon[31859]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:24.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:24 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:24 smithi192 ceph-mon[36700]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:24 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:24.840 INFO:teuthology.orchestra.run.smithi046.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,wsync,nofallback 2024-03-08T23:07:24.840 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync". 2024-03-08T23:07:24.840 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: resolved to: "172.21.15.46:3300,172.21.15.62:3300,172.21.15.192:3300" 2024-03-08T23:07:24.841 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: trying mount with new device syntax: 0@ced7dd96-dd9e-11ee-95c6-87774f69a715.cephfs=/volumes/qa/sv_1/fd24f677-ddfc-4a4a-a669-a06e16ce0e48 2024-03-08T23:07:24.841 INFO:teuthology.orchestra.run.smithi046.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,wsync,name=0,key=0,mon_addr=172.21.15.46:3300/172.21.15.62:3300/172.21.15.192:3300" will pass to kernel 2024-03-08T23:07:24.845 INFO:teuthology.orchestra.run.smithi046.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-08T23:07:24.845 INFO:teuthology.orchestra.run.smithi046.stdout: You just mounted a file system that supports labels which does not 2024-03-08T23:07:24.845 INFO:teuthology.orchestra.run.smithi046.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-08T23:07:24.845 INFO:teuthology.orchestra.run.smithi046.stdout: applications will generate AVC messages and not be allowed access to 2024-03-08T23:07:24.845 INFO:teuthology.orchestra.run.smithi046.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-08T23:07:24.850 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-08T23:07:24.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:24.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:24.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:24 smithi062 ceph-mon[35761]: pgmap v405: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:07:24.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:24 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:24.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:24.945 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-08T23:07:24.999 INFO:teuthology.orchestra.run.smithi046.stdout:client25393 2024-03-08T23:07:24.999 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:07:24.999 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T23:07:25.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:25.051 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-08T23:07:25.097 INFO:teuthology.orchestra.run.smithi046.stdout:client25393 2024-03-08T23:07:25.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:25.097 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-08T23:07:25.186 INFO:teuthology.orchestra.run.smithi046.stdout:client25393 2024-03-08T23:07:25.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:25.186 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715.client25393/status') 2024-03-08T23:07:25.276 INFO:teuthology.orchestra.run.smithi046.stdout:instance: client.25393 (3)172.21.15.46:0/1837359673 2024-03-08T23:07:25.276 INFO:teuthology.orchestra.run.smithi046.stdout:blocklisted: false 2024-03-08T23:07:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records in 2024-03-08T23:07:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:0+1 records out 2024-03-08T23:07:25.276 INFO:teuthology.orchestra.run.smithi046.stderr:71 bytes copied, 7.0119e-05 s, 1.0 MB/s 2024-03-08T23:07:25.281 INFO:teuthology.run_tasks:Running task exec... 2024-03-08T23:07:25.291 INFO:teuthology.task.exec:Executing custom commands... 2024-03-08T23:07:25.291 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:07:25.291 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-08T23:07:25.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 -- 172.21.15.46:0/2830904202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc101da0 msgr2=0x7f0edc104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2830904202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc101da0 0x7f0edc104190 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009be0 tx=0x7f0ecc02bef0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 -- 172.21.15.46:0/2830904202 shutdown_connections 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2830904202 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc107000 0x7f0edc109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2830904202 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 0x7f0edc106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.485+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2830904202 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc101da0 0x7f0edc104190 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.486+0000 7f0ee1600640 1 -- 172.21.15.46:0/2830904202 >> 172.21.15.46:0/2830904202 conn(0x7f0edc0fb7b0 msgr2=0x7f0edc0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.486+0000 7f0ee1600640 1 -- 172.21.15.46:0/2830904202 shutdown_connections 2024-03-08T23:07:25.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.486+0000 7f0ee1600640 1 -- 172.21.15.46:0/2830904202 wait complete. 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 Processor -- start 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 -- start start 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 0x7f0edc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc107000 0x7f0edc1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc10d000 con 0x7f0edc107000 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc10ce80 con 0x7f0edc101da0 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0ee1600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc10d180 con 0x7f0edc1046d0 2024-03-08T23:07:25.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.487+0000 7f0edaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0eda000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 0x7f0edc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55086/0 (socket says 172.21.15.46:55086) 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 -- 172.21.15.46:0/2892035943 learned_addr learned my addr 172.21.15.46:0/2892035943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edb400640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc107000 0x7f0edc1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 -- 172.21.15.46:0/2892035943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 msgr2=0x7f0edc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 0x7f0edc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 -- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc107000 msgr2=0x7f0edc1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc107000 0x7f0edc1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 -- 172.21.15.46:0/2892035943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0edc1a1410 con 0x7f0edc101da0 2024-03-08T23:07:25.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.488+0000 7f0edaa00640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000f80 tx=0x7f0ecc0337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.489+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc043020 con 0x7f0edc101da0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.489+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ecc02ca60 con 0x7f0edc101da0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.489+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc034690 con 0x7f0edc101da0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.489+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc1a16a0 con 0x7f0edc101da0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.489+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc1a1c40 con 0x7f0edc101da0 2024-03-08T23:07:25.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.490+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc108b50 con 0x7f0edc101da0 2024-03-08T23:07:25.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.492+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0ecc046020 con 0x7f0edc101da0 2024-03-08T23:07:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.493+0000 7f0ed3400640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 0x7f0ec00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.493+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f0ecc0f13f0 con 0x7f0edc101da0 2024-03-08T23:07:25.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.493+0000 7f0eda000640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 0x7f0ec00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:25.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.494+0000 7f0eda000640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 0x7f0ec00a4de0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0edc19ad00 tx=0x7f0ec8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:25.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.498+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0ecc04d020 con 0x7f0edc101da0 2024-03-08T23:07:25.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:25.746+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7f0edc02ce70 con 0x7f0edc101da0 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2892035943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:26 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.285+0000 7f0ed3400640 1 -- 172.21.15.46:0/2892035943 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v24) v1 ==== 117+0+0 (secure 0 0 0) 0x7f0ecc0be0b0 con 0x7f0edc101da0 2024-03-08T23:07:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 msgr2=0x7f0ec00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 0x7f0ec00a4de0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0edc19ad00 tx=0x7f0ec8005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:26.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 msgr2=0x7f0edc1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc000f80 tx=0x7f0ecc0337b0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 shutdown_connections 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0ec00a2920 0x7f0ec00a4de0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0edc107000 0x7f0edc1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0edc1046d0 0x7f0edc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 --2- 172.21.15.46:0/2892035943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0edc101da0 0x7f0edc1997f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.288+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 >> 172.21.15.46:0/2892035943 conn(0x7f0edc0fb7b0 msgr2=0x7f0edc1013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.289+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 shutdown_connections 2024-03-08T23:07:26.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.289+0000 7f0ee1600640 1 -- 172.21.15.46:0/2892035943 wait complete. 2024-03-08T23:07:26.309 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.309 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.309 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.310 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.310 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.310 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.310 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2892035943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.310 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:26 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.312 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-08T23:07:26.324 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-08T23:07:26.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:26.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.373 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:26.373 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2892035943' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.373 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:26 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-08T23:07:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 -- 172.21.15.46:0/2806823391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4038104720 msgr2=0x7f4038104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 --2- 172.21.15.46:0/2806823391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4038104720 0x7f4038104ba0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4020009900 tx=0x7f402002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 -- 172.21.15.46:0/2806823391 shutdown_connections 2024-03-08T23:07:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 --2- 172.21.15.46:0/2806823391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40381050e0 0x7f403810b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 --2- 172.21.15.46:0/2806823391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4038104720 0x7f4038104ba0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 --2- 172.21.15.46:0/2806823391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038103530 0x7f4038103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 -- 172.21.15.46:0/2806823391 >> 172.21.15.46:0/2806823391 conn(0x7f40380feaa0 msgr2=0x7f4038100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 -- 172.21.15.46:0/2806823391 shutdown_connections 2024-03-08T23:07:26.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.587+0000 7f403ca00640 1 -- 172.21.15.46:0/2806823391 wait complete. 2024-03-08T23:07:26.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.588+0000 7f403ca00640 1 Processor -- start 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.588+0000 7f403ca00640 1 -- start start 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 0x7f4038199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810f210 con 0x7f4038103530 2024-03-08T23:07:26.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810f090 con 0x7f40381050e0 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f403ca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810f390 con 0x7f4038104720 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4036800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4035e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4035400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 0x7f4038199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4036800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55098/0 (socket says 172.21.15.46:55098) 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4035e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36420/0 (socket says 172.21.15.46:36420) 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.589+0000 7f4036800640 1 -- 172.21.15.46:0/3715841319 learned_addr learned my addr 172.21.15.46:0/3715841319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 -- 172.21.15.46:0/3715841319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 msgr2=0x7f40381a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 -- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 msgr2=0x7f40381997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 -- 172.21.15.46:0/3715841319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40381a1410 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4036800640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035e00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f4035400640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 0x7f4038199d30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4020036040 tx=0x7f402002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020043020 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40200054e0 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40381a16a0 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.590+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4020003c30 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.591+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40381a1c10 con 0x7f4038104720 2024-03-08T23:07:26.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.592+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403810a7f0 con 0x7f4038104720 2024-03-08T23:07:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.594+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4020046020 con 0x7f4038104720 2024-03-08T23:07:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.595+0000 7f402ea00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 0x7f401c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.595+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f40200f0d30 con 0x7f4038104720 2024-03-08T23:07:26.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.595+0000 7f4035e00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 0x7f401c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.596+0000 7f4035e00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 0x7f401c0a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4038104580 tx=0x7f4028009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:26.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.599+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f402004d020 con 0x7f4038104720 2024-03-08T23:07:26.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.821+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4038061ef0 con 0x7f4038104720 2024-03-08T23:07:26.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.823+0000 7f402ea00640 1 -- 172.21.15.46:0/3715841319 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7f40200bd9f0 con 0x7f4038104720 2024-03-08T23:07:26.828 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:26.828 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 msgr2=0x7f401c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 0x7f401c0a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4038104580 tx=0x7f4028009500 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 msgr2=0x7f4038199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 0x7f4038199d30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4020036040 tx=0x7f402002c650 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 shutdown_connections 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f401c0a2880 0x7f401c0a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40381050e0 0x7f40381a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4038104720 0x7f4038199d30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 --2- 172.21.15.46:0/3715841319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4038103530 0x7f40381997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 >> 172.21.15.46:0/3715841319 conn(0x7f40380feaa0 msgr2=0x7f40381079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:26.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 shutdown_connections 2024-03-08T23:07:26.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:26.826+0000 7f403ca00640 1 -- 172.21.15.46:0/3715841319 wait complete. 2024-03-08T23:07:26.850 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:07:26.851 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:07:27.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 -- 172.21.15.46:0/884090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10101c50 msgr2=0x7fed10104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 --2- 172.21.15.46:0/884090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10101c50 0x7fed10104040 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fed00009900 tx=0x7fed0002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:27.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 -- 172.21.15.46:0/884090104 shutdown_connections 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 --2- 172.21.15.46:0/884090104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10106eb0 0x7fed10109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 --2- 172.21.15.46:0/884090104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10104580 0x7fed10106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 --2- 172.21.15.46:0/884090104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10101c50 0x7fed10104040 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 -- 172.21.15.46:0/884090104 >> 172.21.15.46:0/884090104 conn(0x7fed100fb660 msgr2=0x7fed100fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 -- 172.21.15.46:0/884090104 shutdown_connections 2024-03-08T23:07:27.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.074+0000 7fed0f200640 1 -- 172.21.15.46:0/884090104 wait complete. 2024-03-08T23:07:27.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 Processor -- start 2024-03-08T23:07:27.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 -- start start 2024-03-08T23:07:27.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 0x7fed101a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1010ceb0 con 0x7fed10104580 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1010cd30 con 0x7fed10106eb0 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.075+0000 7fed0f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1010d030 con 0x7fed10101c50 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36438/0 (socket says 172.21.15.46:36438) 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:33840/0 (socket says 172.21.15.46:33840) 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0d400640 1 -- 172.21.15.46:0/224055180 learned_addr learned my addr 172.21.15.46:0/224055180 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0e800640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 0x7fed101a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 -- 172.21.15.46:0/224055180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 msgr2=0x7fed101a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 0x7fed101a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 -- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 msgr2=0x7fed10199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0de00640 1 -- 172.21.15.46:0/224055180 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed101a1380 con 0x7fed10101c50 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0e800640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 0x7fed101a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.076+0000 7fed0d400640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed0de00640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fed00036040 tx=0x7fed0002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00043020 con 0x7fed10101c50 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed000054e0 con 0x7fed10101c50 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed101a1610 con 0x7fed10101c50 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00003ce0 con 0x7fed10101c50 2024-03-08T23:07:27.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.077+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed101a1b80 con 0x7fed10101c50 2024-03-08T23:07:27.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.079+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed10108a00 con 0x7fed10101c50 2024-03-08T23:07:27.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.081+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fed00046020 con 0x7fed10101c50 2024-03-08T23:07:27.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.083+0000 7fed06a00640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 0x7fecf40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.083+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fed000f0dd0 con 0x7fed10101c50 2024-03-08T23:07:27.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.083+0000 7fed0d400640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 0x7fecf40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.084+0000 7fed0d400640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 0x7fecf40a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed1019ac70 tx=0x7fecfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:27.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.086+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fed0004d020 con 0x7fed10101c50 2024-03-08T23:07:27.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:07:27.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-08T23:07:27.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:07:27.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:27 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3715841319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:27 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3715841319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.310+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fed1002d090 con 0x7fed10101c50 2024-03-08T23:07:27.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.311+0000 7fed06a00640 1 -- 172.21.15.46:0/224055180 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fed000f2020 con 0x7fed10101c50 2024-03-08T23:07:27.315 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.313+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 msgr2=0x7fecf40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.313+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 0x7fecf40a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fed1019ac70 tx=0x7fecfc005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.313+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 msgr2=0x7fed10199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.313+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fed00036040 tx=0x7fed0002c6c0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 shutdown_connections 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fecf40a2880 0x7fecf40a4d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fed10106eb0 0x7fed101a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fed10104580 0x7fed10199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 --2- 172.21.15.46:0/224055180 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fed10101c50 0x7fed10199760 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 >> 172.21.15.46:0/224055180 conn(0x7fed100fb660 msgr2=0x7fed101012d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 shutdown_connections 2024-03-08T23:07:27.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.314+0000 7fed0f200640 1 -- 172.21.15.46:0/224055180 wait complete. 2024-03-08T23:07:27.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: pgmap v406: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:27.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:27 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3715841319' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 -- 172.21.15.46:0/3002311056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 msgr2=0x7fa0e8104170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 --2- 172.21.15.46:0/3002311056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e8104170 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8009900 tx=0x7fa0d802bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 -- 172.21.15.46:0/3002311056 shutdown_connections 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 --2- 172.21.15.46:0/3002311056 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 --2- 172.21.15.46:0/3002311056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 0x7fa0e8106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 --2- 172.21.15.46:0/3002311056 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e8104170 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 -- 172.21.15.46:0/3002311056 >> 172.21.15.46:0/3002311056 conn(0x7fa0e80fb790 msgr2=0x7fa0e80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 -- 172.21.15.46:0/3002311056 shutdown_connections 2024-03-08T23:07:27.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.546+0000 7fa0ee000640 1 -- 172.21.15.46:0/3002311056 wait complete. 2024-03-08T23:07:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.548+0000 7fa0ee000640 1 Processor -- start 2024-03-08T23:07:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.548+0000 7fa0ee000640 1 -- start start 2024-03-08T23:07:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.548+0000 7fa0ee000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.548+0000 7fa0ee000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 0x7fa0e80ffd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0ee000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8100270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0ee000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e810cfe0 con 0x7fa0e8106fe0 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0ee000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e810ce60 con 0x7fa0e8101d80 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0ee000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e810d160 con 0x7fa0e81046b0 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55132/0 (socket says 172.21.15.46:55132) 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 -- 172.21.15.46:0/568994075 learned_addr learned my addr 172.21.15.46:0/568994075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:27.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e6a00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 0x7fa0e80ffd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7e00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8100270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 -- 172.21.15.46:0/568994075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 msgr2=0x7fa0e80ffd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 0x7fa0e80ffd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 -- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 msgr2=0x7fa0e8100270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8100270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7400640 1 -- 172.21.15.46:0/568994075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e8100b00 con 0x7fa0e8101d80 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.549+0000 7fa0e7e00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0e7400640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8000c00 tx=0x7fa0d802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8043020 con 0x7fa0e8101d80 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0d80054e0 con 0x7fa0e8101d80 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0d8003c40 con 0x7fa0e8101d80 2024-03-08T23:07:27.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e81a17d0 con 0x7fa0e8101d80 2024-03-08T23:07:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.550+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e81a1d70 con 0x7fa0e8101d80 2024-03-08T23:07:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.551+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e80fd280 con 0x7fa0e8101d80 2024-03-08T23:07:27.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.553+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa0d8046020 con 0x7fa0e8101d80 2024-03-08T23:07:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.554+0000 7fa0dfe00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 0x7fa0cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.554+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fa0d80f0e50 con 0x7fa0e8101d80 2024-03-08T23:07:27.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.555+0000 7fa0e6a00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 0x7fa0cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:27.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.555+0000 7fa0e6a00640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 0x7fa0cc0a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8101460 tx=0x7fa0d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:27.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.558+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa0d804d020 con 0x7fa0e8101d80 2024-03-08T23:07:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.797+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa0e8061ef0 con 0x7fa0e8101d80 2024-03-08T23:07:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.797+0000 7fa0dfe00640 1 -- 172.21.15.46:0/568994075 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa0d80bdb70 con 0x7fa0e8101d80 2024-03-08T23:07:27.801 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 msgr2=0x7fa0cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 0x7fa0cc0a4d40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8101460 tx=0x7fa0d4005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 msgr2=0x7fa0e81016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8000c00 tx=0x7fa0d802c650 comp rx=0 tx=0).stop 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 shutdown_connections 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa0cc0a2880 0x7fa0cc0a4d40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa0e8106fe0 0x7fa0e8100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa0e81046b0 0x7fa0e80ffd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 --2- 172.21.15.46:0/568994075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa0e8101d80 0x7fa0e81016f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 >> 172.21.15.46:0/568994075 conn(0x7fa0e80fb790 msgr2=0x7fa0e81026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 shutdown_connections 2024-03-08T23:07:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:27.800+0000 7fa0ee000640 1 -- 172.21.15.46:0/568994075 wait complete. 2024-03-08T23:07:27.824 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.027+0000 7f56de000640 1 -- 172.21.15.46:0/4245157023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d81050e0 msgr2=0x7f56d810b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.027+0000 7f56de000640 1 --2- 172.21.15.46:0/4245157023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d81050e0 0x7f56d810b970 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f56d0009ca0 tx=0x7f56d002c100 comp rx=0 tx=0).stop 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 -- 172.21.15.46:0/4245157023 shutdown_connections 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 --2- 172.21.15.46:0/4245157023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d81050e0 0x7f56d810b970 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 --2- 172.21.15.46:0/4245157023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d8104720 0x7f56d8104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 --2- 172.21.15.46:0/4245157023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8103530 0x7f56d8103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 -- 172.21.15.46:0/4245157023 >> 172.21.15.46:0/4245157023 conn(0x7f56d80feaa0 msgr2=0x7f56d8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:28.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 -- 172.21.15.46:0/4245157023 shutdown_connections 2024-03-08T23:07:28.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.028+0000 7f56de000640 1 -- 172.21.15.46:0/4245157023 wait complete. 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 Processor -- start 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 -- start start 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8104720 0x7f56d8199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d810f1c0 con 0x7f56d8103530 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d810f040 con 0x7f56d81050e0 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.029+0000 7f56de000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d810f340 con 0x7f56d8104720 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55140/0 (socket says 172.21.15.46:55140) 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36468/0 (socket says 172.21.15.46:36468) 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 -- 172.21.15.46:0/2992539002 learned_addr learned my addr 172.21.15.46:0/2992539002 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56cea00640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8104720 0x7f56d8199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 -- 172.21.15.46:0/2992539002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8104720 msgr2=0x7f56d8199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8104720 0x7f56d8199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 -- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 msgr2=0x7f56d8199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7e00640 1 -- 172.21.15.46:0/2992539002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d81a13b0 con 0x7f56d81050e0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.030+0000 7f56d7400640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56d7e00640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56d002c640 tx=0x7f56d0035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0039040 con 0x7f56d81050e0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56d0044070 con 0x7f56d81050e0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d0004460 con 0x7f56d81050e0 2024-03-08T23:07:28.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d81a1640 con 0x7f56d81050e0 2024-03-08T23:07:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.031+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d81a1b30 con 0x7f56d81050e0 2024-03-08T23:07:28.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.033+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d810a7f0 con 0x7f56d81050e0 2024-03-08T23:07:28.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.035+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f56d0004600 con 0x7f56d81050e0 2024-03-08T23:07:28.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.037+0000 7f56d4c00640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.037+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f56d00f12b0 con 0x7f56d81050e0 2024-03-08T23:07:28.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.037+0000 7f56d7400640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.038+0000 7f56d7400640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f56c8005fd0 tx=0x7f56c800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:28.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.041+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f56d004e020 con 0x7f56d81050e0 2024-03-08T23:07:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:28 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/224055180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:28 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/568994075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:28 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.250+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f56d8061ef0 con 0x7f56d81050e0 2024-03-08T23:07:28.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.251+0000 7f56d4c00640 1 -- 172.21.15.46:0/2992539002 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f56d00bdf70 con 0x7f56d81050e0 2024-03-08T23:07:28.255 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 msgr2=0x7f56bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f56c8005fd0 tx=0x7f56c800b410 comp rx=0 tx=0).stop 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 msgr2=0x7f56d81a0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f56d002c640 tx=0x7f56d0035d10 comp rx=0 tx=0).stop 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 shutdown_connections 2024-03-08T23:07:28.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f56bc0a2800 0x7f56bc0a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56d81050e0 0x7f56d81a0d40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56d8104720 0x7f56d8199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 --2- 172.21.15.46:0/2992539002 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56d8103530 0x7f56d8199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 >> 172.21.15.46:0/2992539002 conn(0x7f56d80feaa0 msgr2=0x7f56d81079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 shutdown_connections 2024-03-08T23:07:28.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.254+0000 7f56de000640 1 -- 172.21.15.46:0/2992539002 wait complete. 2024-03-08T23:07:28.277 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:07:28.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:28 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/224055180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:28 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/568994075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:28 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:28 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/224055180' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:28 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/568994075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:28 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.486+0000 7f605d200640 1 -- 172.21.15.46:0/1249912725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 msgr2=0x7f6058109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.486+0000 7f605d200640 1 --2- 172.21.15.46:0/1249912725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f6058109740 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6050009ca0 tx=0x7f605002c100 comp rx=0 tx=0).stop 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 -- 172.21.15.46:0/1249912725 shutdown_connections 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 --2- 172.21.15.46:0/1249912725 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f6058109740 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 --2- 172.21.15.46:0/1249912725 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 --2- 172.21.15.46:0/1249912725 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 0x7f6058101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 -- 172.21.15.46:0/1249912725 >> 172.21.15.46:0/1249912725 conn(0x7f60580fc890 msgr2=0x7f60580fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 -- 172.21.15.46:0/1249912725 shutdown_connections 2024-03-08T23:07:28.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.487+0000 7f605d200640 1 -- 172.21.15.46:0/1249912725 wait complete. 2024-03-08T23:07:28.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 Processor -- start 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 -- start start 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 0x7f6058199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f60581a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605810cf90 con 0x7f60581024f0 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605810ce10 con 0x7f6058102eb0 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f605d200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605810d110 con 0x7f6058101300 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.488+0000 7f604fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36482/0 (socket says 172.21.15.46:36482) 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 -- 172.21.15.46:0/3673963249 learned_addr learned my addr 172.21.15.46:0/3673963249 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f6055e00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 0x7f6058199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f6056800640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f60581a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 -- 172.21.15.46:0/3673963249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 msgr2=0x7f6058199800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 0x7f6058199800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 -- 172.21.15.46:0/3673963249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 msgr2=0x7f60581a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f60581a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 -- 172.21.15.46:0/3673963249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60581a1420 con 0x7f60581024f0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.489+0000 7f604fe00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f604000e9f0 tx=0x7f604000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.490+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604000cd80 con 0x7f60581024f0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.490+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60581a1710 con 0x7f60581024f0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.490+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6040010910 con 0x7f60581024f0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.490+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400188f0 con 0x7f60581024f0 2024-03-08T23:07:28.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.490+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60581a1cb0 con 0x7f60581024f0 2024-03-08T23:07:28.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.491+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60581a1fd0 con 0x7f60581024f0 2024-03-08T23:07:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.493+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6040010ac0 con 0x7f60581024f0 2024-03-08T23:07:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.494+0000 7f604ea00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.494+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f60400cf1d0 con 0x7f60581024f0 2024-03-08T23:07:28.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.495+0000 7f6055e00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.495+0000 7f6055e00640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 0x7f603c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6058102350 tx=0x7f6048005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:28.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.498+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f604009b840 con 0x7f60581024f0 2024-03-08T23:07:28.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.702+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f60581a21a0 con 0x7f60581024f0 2024-03-08T23:07:28.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.702+0000 7f604ea00640 1 -- 172.21.15.46:0/3673963249 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f604009f8e0 con 0x7f60581024f0 2024-03-08T23:07:28.780 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:28.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 msgr2=0x7f603c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 0x7f603c0a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f6058102350 tx=0x7f6048005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 msgr2=0x7f6058199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f604000e9f0 tx=0x7f604000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 shutdown_connections 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f603c0a2800 0x7f603c0a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6058102eb0 0x7f60581a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f60581024f0 0x7f6058199d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 --2- 172.21.15.46:0/3673963249 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6058101300 0x7f6058199800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 >> 172.21.15.46:0/3673963249 conn(0x7f60580fc890 msgr2=0x7f6058073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 shutdown_connections 2024-03-08T23:07:28.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.705+0000 7f605d200640 1 -- 172.21.15.46:0/3673963249 wait complete. 2024-03-08T23:07:28.783 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 -- 172.21.15.46:0/2536764400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960106ff0 msgr2=0x7fc960109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 --2- 172.21.15.46:0/2536764400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960106ff0 0x7fc960109440 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc958009be0 tx=0x7fc95802bef0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 -- 172.21.15.46:0/2536764400 shutdown_connections 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 --2- 172.21.15.46:0/2536764400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960106ff0 0x7fc960109440 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 --2- 172.21.15.46:0/2536764400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9601046c0 0x7fc960106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 --2- 172.21.15.46:0/2536764400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc960101d90 0x7fc960104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 -- 172.21.15.46:0/2536764400 >> 172.21.15.46:0/2536764400 conn(0x7fc9600fb780 msgr2=0x7fc9600fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 -- 172.21.15.46:0/2536764400 shutdown_connections 2024-03-08T23:07:28.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.988+0000 7fc965200640 1 -- 172.21.15.46:0/2536764400 wait complete. 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.989+0000 7fc965200640 1 Processor -- start 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.989+0000 7fc965200640 1 -- start start 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.989+0000 7fc965200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 0x7fc960101700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.989+0000 7fc965200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9601046c0 0x7fc9600ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc965200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc965200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010cff0 con 0x7fc960106ff0 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc965200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010ce70 con 0x7fc9601046c0 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc965200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d170 con 0x7fc960101d90 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36502/0 (socket says 172.21.15.46:36502) 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95e800640 1 -- 172.21.15.46:0/2924938108 learned_addr learned my addr 172.21.15.46:0/2924938108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 0x7fc960101700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95d400640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9601046c0 0x7fc9600ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 -- 172.21.15.46:0/2924938108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9601046c0 msgr2=0x7fc9600ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9601046c0 0x7fc9600ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 -- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 msgr2=0x7fc960100280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95de00640 1 -- 172.21.15.46:0/2924938108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc960100b10 con 0x7fc960101d90 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.990+0000 7fc95e800640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc95de00640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 0x7fc960101700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e790 tx=0x7fc95000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000eea0 con 0x7fc960101d90 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9601a16e0 con 0x7fc960101d90 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc95000ccc0 con 0x7fc960101d90 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950018760 con 0x7fc960101d90 2024-03-08T23:07:28.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.991+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9601a1c80 con 0x7fc960101d90 2024-03-08T23:07:28.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.993+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9600fd270 con 0x7fc960101d90 2024-03-08T23:07:28.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.995+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc950010430 con 0x7fc960101d90 2024-03-08T23:07:29.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.996+0000 7fc956a00640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.996+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fc9500cf490 con 0x7fc960101d90 2024-03-08T23:07:29.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.996+0000 7fc95d400640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.997+0000 7fc95d400640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 0x7fc9440a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc960101470 tx=0x7fc94c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:29.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:28.998+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc95009bb00 con 0x7fc960101d90 2024-03-08T23:07:29.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.210+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc96002d100 con 0x7fc960101d90 2024-03-08T23:07:29.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.213+0000 7fc956a00640 1 -- 172.21.15.46:0/2924938108 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7fc95009fba0 con 0x7fc960101d90 2024-03-08T23:07:29.217 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:29.217 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 msgr2=0x7fc9440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 0x7fc9440a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc960101470 tx=0x7fc94c00b390 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 msgr2=0x7fc960101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 0x7fc960101700 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e790 tx=0x7fc95000ec50 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 shutdown_connections 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc9440a2800 0x7fc9440a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc960106ff0 0x7fc960100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9601046c0 0x7fc9600ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 --2- 172.21.15.46:0/2924938108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc960101d90 0x7fc960101700 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 >> 172.21.15.46:0/2924938108 conn(0x7fc9600fb780 msgr2=0x7fc9601026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 shutdown_connections 2024-03-08T23:07:29.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.216+0000 7fc965200640 1 -- 172.21.15.46:0/2924938108 wait complete. 2024-03-08T23:07:29.241 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:07:29.242 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:07:29.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:29 smithi192 ceph-mon[36700]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:07:29.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:29 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2992539002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:29 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3673963249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:29 smithi046 ceph-mon[31859]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:07:29.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2992539002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.316 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:29 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3673963249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:29 smithi062 ceph-mon[35761]: pgmap v407: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:07:29.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:29 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2992539002' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:29 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3673963249' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:29.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 -- 172.21.15.46:0/2909991955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 msgr2=0x7f11c0073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 --2- 172.21.15.46:0/2909991955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0073780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009ca0 tx=0x7f11b002c100 comp rx=0 tx=0).stop 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 -- 172.21.15.46:0/2909991955 shutdown_connections 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 --2- 172.21.15.46:0/2909991955 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 0x7f11c010b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 --2- 172.21.15.46:0/2909991955 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 0x7f11c00741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 --2- 172.21.15.46:0/2909991955 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0073780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.457+0000 7f11c5a00640 1 -- 172.21.15.46:0/2909991955 >> 172.21.15.46:0/2909991955 conn(0x7f11c00fc760 msgr2=0x7f11c00feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.458+0000 7f11c5a00640 1 -- 172.21.15.46:0/2909991955 shutdown_connections 2024-03-08T23:07:29.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.458+0000 7f11c5a00640 1 -- 172.21.15.46:0/2909991955 wait complete. 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.458+0000 7f11c5a00640 1 Processor -- start 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 -- start start 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 0x7f11c0199660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 0x7f11c01a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c010f050 con 0x7f11c0073d50 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c010eed0 con 0x7f11c0075330 2024-03-08T23:07:29.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c010f1d0 con 0x7f11c0109090 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11b7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11bfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 0x7f11c0199660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11c5000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 0x7f11c01a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11b7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55204/0 (socket says 172.21.15.46:55204) 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.459+0000 7f11b7400640 1 -- 172.21.15.46:0/973867465 learned_addr learned my addr 172.21.15.46:0/973867465 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 -- 172.21.15.46:0/973867465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 msgr2=0x7f11c01a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 0x7f11c01a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 -- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 msgr2=0x7f11c0199660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 0x7f11c0199660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 -- 172.21.15.46:0/973867465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c01a1280 con 0x7f11c0075330 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11b7400640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11a800e9f0 tx=0x7f11a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a800cd80 con 0x7f11c0075330 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c01a1570 con 0x7f11c0075330 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11a8010910 con 0x7f11c0075330 2024-03-08T23:07:29.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a80188f0 con 0x7f11c0075330 2024-03-08T23:07:29.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.460+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c01a1ac0 con 0x7f11c0075330 2024-03-08T23:07:29.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.462+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c0107dc0 con 0x7f11c0075330 2024-03-08T23:07:29.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.463+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f11a8010ac0 con 0x7f11c0075330 2024-03-08T23:07:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.465+0000 7f11bd600640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 0x7f11a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.465+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f11a80cf1f0 con 0x7f11c0075330 2024-03-08T23:07:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.465+0000 7f11bfe00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 0x7f11a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.466+0000 7f11bfe00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 0x7f11a40a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f11b002c710 tx=0x7f11b00323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.469+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f11a809b860 con 0x7f11c0075330 2024-03-08T23:07:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.670+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f11c0061ef0 con 0x7f11c0075330 2024-03-08T23:07:29.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.670+0000 7f11bd600640 1 -- 172.21.15.46:0/973867465 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f11a809f900 con 0x7f11c0075330 2024-03-08T23:07:29.674 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:07:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 msgr2=0x7f11a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 0x7f11a40a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f11b002c710 tx=0x7f11b00323c0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 msgr2=0x7f11c0199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f11a800e9f0 tx=0x7f11a800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 shutdown_connections 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f11a40a2800 0x7f11a40a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11c0109090 0x7f11c01a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11c0075330 0x7f11c0199ba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 --2- 172.21.15.46:0/973867465 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11c0073d50 0x7f11c0199660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.673+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 >> 172.21.15.46:0/973867465 conn(0x7f11c00fc760 msgr2=0x7f11c0104f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.674+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 shutdown_connections 2024-03-08T23:07:29.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.674+0000 7f11c5a00640 1 -- 172.21.15.46:0/973867465 wait complete. 2024-03-08T23:07:29.697 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:07:29.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 -- 172.21.15.46:0/3217695360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 msgr2=0x7fdbfc103810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/3217695360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc103810 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009900 tx=0x7fdbec02bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 -- 172.21.15.46:0/3217695360 shutdown_connections 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/3217695360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc104fa0 0x7fdbfc10b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/3217695360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc1045e0 0x7fdbfc104a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/3217695360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc103810 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 -- 172.21.15.46:0/3217695360 >> 172.21.15.46:0/3217695360 conn(0x7fdbfc0fe960 msgr2=0x7fdbfc100da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 -- 172.21.15.46:0/3217695360 shutdown_connections 2024-03-08T23:07:29.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.910+0000 7fdbfbe00640 1 -- 172.21.15.46:0/3217695360 wait complete. 2024-03-08T23:07:29.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.911+0000 7fdbfbe00640 1 Processor -- start 2024-03-08T23:07:29.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.911+0000 7fdbfbe00640 1 -- start start 2024-03-08T23:07:29.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc1045e0 0x7fdbfc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc104fa0 0x7fdbfc1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc10f0d0 con 0x7fdbfc1033f0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc10ef50 con 0x7fdbfc104fa0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfbe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc10f250 con 0x7fdbfc1045e0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36538/0 (socket says 172.21.15.46:36538) 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfaa00640 1 -- 172.21.15.46:0/692378771 learned_addr learned my addr 172.21.15.46:0/692378771 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbf2000640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc1045e0 0x7fdbfc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.912+0000 7fdbfb400640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc104fa0 0x7fdbfc1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 -- 172.21.15.46:0/692378771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc1045e0 msgr2=0x7fdbfc199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc1045e0 0x7fdbfc199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 -- 172.21.15.46:0/692378771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc104fa0 msgr2=0x7fdbfc1a0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc104fa0 0x7fdbfc1a0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 -- 172.21.15.46:0/692378771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc1a1300 con 0x7fdbfc1033f0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfaa00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02efd0 tx=0x7fdbec02c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec041070 con 0x7fdbfc1033f0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbec004450 con 0x7fdbfc1033f0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbec003e50 con 0x7fdbfc1033f0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.913+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc1a1590 con 0x7fdbfc1033f0 2024-03-08T23:07:29.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.914+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc1a1aa0 con 0x7fdbfc1033f0 2024-03-08T23:07:29.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.915+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc10a6b0 con 0x7fdbfc1033f0 2024-03-08T23:07:29.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.917+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdbec044070 con 0x7fdbfc1033f0 2024-03-08T23:07:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.918+0000 7fdc00a00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 0x7fdbe00a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.918+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fdbec0f0950 con 0x7fdbfc1033f0 2024-03-08T23:07:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.918+0000 7fdbf2000640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 0x7fdbe00a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:29.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.919+0000 7fdbf2000640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 0x7fdbe00a4ee0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc19abf0 tx=0x7fdbe4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:29.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:29.923+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdbec0bcfc0 con 0x7fdbfc1033f0 2024-03-08T23:07:30.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.126+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdbfc061ef0 con 0x7fdbfc1033f0 2024-03-08T23:07:30.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.127+0000 7fdc00a00640 1 -- 172.21.15.46:0/692378771 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdbec0c1060 con 0x7fdbfc1033f0 2024-03-08T23:07:30.131 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 msgr2=0x7fdbe00a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 0x7fdbe00a4ee0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc19abf0 tx=0x7fdbe4005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 msgr2=0x7fdbfc1996e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdbec02efd0 tx=0x7fdbec02c660 comp rx=0 tx=0).stop 2024-03-08T23:07:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 shutdown_connections 2024-03-08T23:07:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fdbe00a2a20 0x7fdbe00a4ee0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbfc104fa0 0x7fdbfc1a0c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbfc1045e0 0x7fdbfc199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 --2- 172.21.15.46:0/692378771 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbfc1033f0 0x7fdbfc1996e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 >> 172.21.15.46:0/692378771 conn(0x7fdbfc0fe960 msgr2=0x7fdbfc107880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 shutdown_connections 2024-03-08T23:07:30.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.129+0000 7fdbfbe00640 1 -- 172.21.15.46:0/692378771 wait complete. 2024-03-08T23:07:30.154 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:07:30.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 -- 172.21.15.46:0/381775959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 msgr2=0x7f23cc1005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 --2- 172.21.15.46:0/381775959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc1005f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009cd0 tx=0x7f23bc02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 -- 172.21.15.46:0/381775959 shutdown_connections 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 --2- 172.21.15.46:0/381775959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc105830 0x7f23cc107c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 --2- 172.21.15.46:0/381775959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc100b30 0x7f23cc100fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 --2- 172.21.15.46:0/381775959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc1005f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.381+0000 7f23cbe00640 1 -- 172.21.15.46:0/381775959 >> 172.21.15.46:0/381775959 conn(0x7f23cc0fa970 msgr2=0x7f23cc0fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.382+0000 7f23cbe00640 1 -- 172.21.15.46:0/381775959 shutdown_connections 2024-03-08T23:07:30.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.382+0000 7f23cbe00640 1 -- 172.21.15.46:0/381775959 wait complete. 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 Processor -- start 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 -- start start 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc100b30 0x7f23cc199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc105830 0x7f23cc1a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc10ac20 con 0x7f23cc100b30 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc10aaa0 con 0x7f23cc067770 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cbe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23cc10ada0 con 0x7f23cc105830 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23caa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55234/0 (socket says 172.21.15.46:55234) 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23cb400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc105830 0x7f23cc1a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23caa00640 1 -- 172.21.15.46:0/1468113476 learned_addr learned my addr 172.21.15.46:0/1468113476 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:30.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.383+0000 7f23c2000640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc100b30 0x7f23cc199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 -- 172.21.15.46:0/1468113476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc105830 msgr2=0x7f23cc1a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc105830 0x7f23cc1a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 -- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc100b30 msgr2=0x7f23cc199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc100b30 0x7f23cc199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 -- 172.21.15.46:0/1468113476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc1a1350 con 0x7f23cc067770 2024-03-08T23:07:30.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23caa00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009ca0 tx=0x7f23bc032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:30.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc041020 con 0x7f23cc067770 2024-03-08T23:07:30.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23bc034cd0 con 0x7f23cc067770 2024-03-08T23:07:30.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc039040 con 0x7f23cc067770 2024-03-08T23:07:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.384+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc1a15e0 con 0x7f23cc067770 2024-03-08T23:07:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.385+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc1a1a50 con 0x7f23cc067770 2024-03-08T23:07:30.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.386+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc104640 con 0x7f23cc067770 2024-03-08T23:07:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:30 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2924938108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:30 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:30 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:30 smithi062 ceph-mon[35761]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:30 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/973867465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:30.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.388+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f23bc044020 con 0x7f23cc067770 2024-03-08T23:07:30.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.389+0000 7f23c3e00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 0x7f23b00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.389+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f23bc0f0ba0 con 0x7f23cc067770 2024-03-08T23:07:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.389+0000 7f23c2000640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 0x7f23b00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.390+0000 7f23c2000640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 0x7f23b00a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23cc19ac40 tx=0x7f23b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:30.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.393+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f23bc04b020 con 0x7f23cc067770 2024-03-08T23:07:30.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:30 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2924938108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:30.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:30.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:30.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:30 smithi046 ceph-mon[31859]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:30.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:30 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/973867465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:30 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2924938108' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:30 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:30 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:30 smithi192 ceph-mon[36700]: from='mgr.14152 172.21.15.46:0/765931719' entity='mgr.x' 2024-03-08T23:07:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:30 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/973867465' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:30.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.596+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f23cc061ef0 con 0x7f23cc067770 2024-03-08T23:07:30.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.597+0000 7f23c3e00640 1 -- 172.21.15.46:0/1468113476 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f23bc0bd8f0 con 0x7f23cc067770 2024-03-08T23:07:30.600 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 msgr2=0x7f23b00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 0x7f23b00a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f23cc19ac40 tx=0x7f23b4005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 msgr2=0x7f23cc199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009ca0 tx=0x7f23bc032ec0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 shutdown_connections 2024-03-08T23:07:30.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f23b00a2800 0x7f23b00a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23cc105830 0x7f23cc1a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f23cc100b30 0x7f23cc199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 --2- 172.21.15.46:0/1468113476 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f23cc067770 0x7f23cc199730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 >> 172.21.15.46:0/1468113476 conn(0x7f23cc0fa970 msgr2=0x7f23cc101810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.599+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 shutdown_connections 2024-03-08T23:07:30.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.600+0000 7f23cbe00640 1 -- 172.21.15.46:0/1468113476 wait complete. 2024-03-08T23:07:30.623 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:07:30.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 -- 172.21.15.46:0/2688407603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 msgr2=0x7fa62c104820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 --2- 172.21.15.46:0/2688407603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c104820 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009900 tx=0x7fa61c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 -- 172.21.15.46:0/2688407603 shutdown_connections 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 --2- 172.21.15.46:0/2688407603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c104d60 0x7fa62c1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 --2- 172.21.15.46:0/2688407603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c104820 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 --2- 172.21.15.46:0/2688407603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c0ffa40 0x7fa62c101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 -- 172.21.15.46:0/2688407603 >> 172.21.15.46:0/2688407603 conn(0x7fa62c0fb660 msgr2=0x7fa62c0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 -- 172.21.15.46:0/2688407603 shutdown_connections 2024-03-08T23:07:30.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.850+0000 7fa632400640 1 -- 172.21.15.46:0/2688407603 wait complete. 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.851+0000 7fa632400640 1 Processor -- start 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 -- start start 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c19dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 0x7fa62c1a5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c113470 con 0x7fa62c0ffa40 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c1132f0 con 0x7fa62c1023b0 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa632400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa62c1135f0 con 0x7fa62c104d60 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa631000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa631000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa62be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c19dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa631000640 1 -- 172.21.15.46:0/4206514842 learned_addr learned my addr 172.21.15.46:0/4206514842 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.852+0000 7fa631a00640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 0x7fa62c1a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 -- 172.21.15.46:0/4206514842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 msgr2=0x7fa62c1a5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 0x7fa62c1a5010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 -- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 msgr2=0x7fa62c19da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 -- 172.21.15.46:0/4206514842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62c1a5680 con 0x7fa62c1023b0 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa631a00640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 0x7fa62c1a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa631000640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:30.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa62be00640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c19dfa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa61c036040 tx=0x7fa61c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c043020 con 0x7fa62c1023b0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa61c0054e0 con 0x7fa62c1023b0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61c003c10 con 0x7fa62c1023b0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.853+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa62c1a5910 con 0x7fa62c1023b0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.854+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa62c1a5de0 con 0x7fa62c1023b0 2024-03-08T23:07:30.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.855+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa62c0fd150 con 0x7fa62c1023b0 2024-03-08T23:07:30.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.857+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa61c046020 con 0x7fa62c1023b0 2024-03-08T23:07:30.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.858+0000 7fa629600640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 0x7fa6100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:30.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.858+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fa61c0f13d0 con 0x7fa62c1023b0 2024-03-08T23:07:30.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.858+0000 7fa631000640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 0x7fa6100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:30.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.859+0000 7fa631000640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 0x7fa6100a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa614011550 tx=0x7fa614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:30.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:30.860+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa61c04d020 con 0x7fa62c1023b0 2024-03-08T23:07:31.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.062+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa62c02d000 con 0x7fa62c1023b0 2024-03-08T23:07:31.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.068+0000 7fa629600640 1 -- 172.21.15.46:0/4206514842 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa61c0be0f0 con 0x7fa62c1023b0 2024-03-08T23:07:31.072 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.070+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 msgr2=0x7fa6100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 0x7fa6100a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa614011550 tx=0x7fa614005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 msgr2=0x7fa62c19dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c19dfa0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa61c036040 tx=0x7fa61c02c650 comp rx=0 tx=0).stop 2024-03-08T23:07:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 shutdown_connections 2024-03-08T23:07:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fa6100a2880 0x7fa6100a4d40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa62c104d60 0x7fa62c1a5010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa62c1023b0 0x7fa62c19dfa0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 --2- 172.21.15.46:0/4206514842 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa62c0ffa40 0x7fa62c19da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 >> 172.21.15.46:0/4206514842 conn(0x7fa62c0fb660 msgr2=0x7fa62c1003c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 shutdown_connections 2024-03-08T23:07:31.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.071+0000 7fa632400640 1 -- 172.21.15.46:0/4206514842 wait complete. 2024-03-08T23:07:31.096 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-08T23:07:31.096 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:07:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 -- 172.21.15.46:0/1582898207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8102320 msgr2=0x7f9fa81027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 --2- 172.21.15.46:0/1582898207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8102320 0x7f9fa81027a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9f90009900 tx=0x7f9f9002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 -- 172.21.15.46:0/1582898207 shutdown_connections 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 --2- 172.21.15.46:0/1582898207 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102ce0 0x7f9fa8109570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 --2- 172.21.15.46:0/1582898207 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8102320 0x7f9fa81027a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 --2- 172.21.15.46:0/1582898207 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8101130 0x7f9fa8101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 -- 172.21.15.46:0/1582898207 >> 172.21.15.46:0/1582898207 conn(0x7f9fa80fc6a0 msgr2=0x7f9fa80feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 -- 172.21.15.46:0/1582898207 shutdown_connections 2024-03-08T23:07:31.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.311+0000 7f9facc00640 1 -- 172.21.15.46:0/1582898207 wait complete. 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.312+0000 7f9facc00640 1 Processor -- start 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.312+0000 7f9facc00640 1 -- start start 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8101130 0x7f9fa81996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 0x7f9fa81a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa810ce10 con 0x7f9fa8102320 2024-03-08T23:07:31.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa810cc90 con 0x7f9fa8101130 2024-03-08T23:07:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9facc00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fa810cf90 con 0x7f9fa8102ce0 2024-03-08T23:07:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9f9ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9f9ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36560/0 (socket says 172.21.15.46:36560) 2024-03-08T23:07:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9f9ea00640 1 -- 172.21.15.46:0/4132078066 learned_addr learned my addr 172.21.15.46:0/4132078066 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:31.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9fa7400640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8101130 0x7f9fa81996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.313+0000 7f9fa7e00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 0x7f9fa81a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 -- 172.21.15.46:0/4132078066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 msgr2=0x7f9fa81a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 0x7f9fa81a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 -- 172.21.15.46:0/4132078066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8101130 msgr2=0x7f9fa81996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8101130 0x7f9fa81996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 -- 172.21.15.46:0/4132078066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa81a12d0 con 0x7f9fa8102320 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9fa7e00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 0x7f9fa81a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9f9ea00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f9002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90043020 con 0x7f9fa8102320 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa81a1560 con 0x7f9fa8102320 2024-03-08T23:07:31.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f900054e0 con 0x7f9fa8102320 2024-03-08T23:07:31.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f90003c40 con 0x7f9fa8102320 2024-03-08T23:07:31.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.314+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa81a1b00 con 0x7f9fa8102320 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.315+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa81a1e20 con 0x7f9fa8102320 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.317+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9f90046020 con 0x7f9fa8102320 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.319+0000 7f9fa4c00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 0x7f9f8c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.319+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f9f900f0ee0 con 0x7f9fa8102320 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.319+0000 7f9fa7400640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 0x7f9f8c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.320+0000 7f9fa7400640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 0x7f9f8c0a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8102180 tx=0x7f9f98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:31.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.323+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9f9004d020 con 0x7f9fa8102320 2024-03-08T23:07:31.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:31 smithi062 ceph-mon[35761]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-08T23:07:31.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:31 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/692378771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:31 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1468113476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:31 smithi046 ceph-mon[31859]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-08T23:07:31.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:31 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/692378771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:31 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1468113476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:31 smithi192 ceph-mon[36700]: pgmap v408: 129 pgs: 129 active+clean; 642 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-03-08T23:07:31.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:31 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/692378771' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:31 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1468113476' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:31.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.569+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fa81a1ff0 con 0x7f9fa8102320 2024-03-08T23:07:31.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.571+0000 7f9fa4c00640 1 -- 172.21.15.46:0/4132078066 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f9f900bdba0 con 0x7f9fa8102320 2024-03-08T23:07:31.575 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:31.575 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:07:25.750030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:07:31.576 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-08T23:07:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 msgr2=0x7f9f8c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 0x7f9f8c0a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8102180 tx=0x7f9f98005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:31.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 msgr2=0x7f9fa8199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9f90000c00 tx=0x7f9f9002c650 comp rx=0 tx=0).stop 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 shutdown_connections 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f8c0a2880 0x7f9f8c0a4d40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9fa8102ce0 0x7f9fa81a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9fa8102320 0x7f9fa8199bf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 --2- 172.21.15.46:0/4132078066 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9fa8101130 0x7f9fa81996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 >> 172.21.15.46:0/4132078066 conn(0x7f9fa80fc6a0 msgr2=0x7f9fa8073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.574+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 shutdown_connections 2024-03-08T23:07:31.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.575+0000 7f9facc00640 1 -- 172.21.15.46:0/4132078066 wait complete. 2024-03-08T23:07:31.599 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-08T23:07:31.599 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-08T23:07:31.599 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 -- 172.21.15.46:0/3909803210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4101150 msgr2=0x7f6fc4101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/3909803210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4101150 0x7f6fc4101570 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4009900 tx=0x7f6fb402bc50 comp rx=0 tx=0).stop 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 -- 172.21.15.46:0/3909803210 shutdown_connections 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/3909803210 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4102d00 0x7f6fc4109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/3909803210 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/3909803210 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4101150 0x7f6fc4101570 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.808+0000 7f6fc8e00640 1 -- 172.21.15.46:0/3909803210 >> 172.21.15.46:0/3909803210 conn(0x7f6fc40fc6c0 msgr2=0x7f6fc40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.809+0000 7f6fc8e00640 1 -- 172.21.15.46:0/3909803210 shutdown_connections 2024-03-08T23:07:31.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.809+0000 7f6fc8e00640 1 -- 172.21.15.46:0/3909803210 wait complete. 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 Processor -- start 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 -- start start 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4101150 0x7f6fc4197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 0x7f6fc419ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc410ce30 con 0x7f6fc4101150 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc410ccb0 con 0x7f6fc4102d00 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.810+0000 7f6fc8e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc410cfb0 con 0x7f6fc4102340 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fbaa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4101150 0x7f6fc4197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fbaa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:34016/0 (socket says 172.21.15.46:34016) 2024-03-08T23:07:31.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fbaa00640 1 -- 172.21.15.46:0/2552936870 learned_addr learned my addr 172.21.15.46:0/2552936870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 0x7f6fc419ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 -- 172.21.15.46:0/2552936870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 msgr2=0x7f6fc41979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 -- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4101150 msgr2=0x7f6fc4197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4101150 0x7f6fc4197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 -- 172.21.15.46:0/2552936870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc419f080 con 0x7f6fc4102d00 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fbaa00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.811+0000 7f6fc3e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 0x7f6fc419ea10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009fd0 tx=0x7f6fbc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.812+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00ce20 con 0x7f6fc4102d00 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.812+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fbc005020 con 0x7f6fc4102d00 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.812+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc010720 con 0x7f6fc4102d00 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.812+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc419f370 con 0x7f6fc4102d00 2024-03-08T23:07:31.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.812+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc419f870 con 0x7f6fc4102d00 2024-03-08T23:07:31.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.813+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc4108470 con 0x7f6fc4102d00 2024-03-08T23:07:31.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.815+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6fbc005540 con 0x7f6fc4102d00 2024-03-08T23:07:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.816+0000 7f6fc0c00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 0x7f6fa80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.816+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f6fbc0cf800 con 0x7f6fc4102d00 2024-03-08T23:07:31.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.817+0000 7f6fc3400640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 0x7f6fa80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:31.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.818+0000 7f6fc3400640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 0x7f6fa80a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6fb402c190 tx=0x7f6fb4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:31.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:31.819+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6fbc09c4c0 con 0x7f6fc4102d00 2024-03-08T23:07:32.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.019+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fc4061730 con 0x7f6fa80a2880 2024-03-08T23:07:32.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.036+0000 7f6fc0c00640 1 -- 172.21.15.46:0/2552936870 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340800 (secure 0 0 0) 0x7f6fc4061730 con 0x7f6fa80a2880 2024-03-08T23:07:32.045 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:32.047 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 msgr2=0x7f6fa80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 0x7f6fa80a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6fb402c190 tx=0x7f6fb4000bc0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 msgr2=0x7f6fc419ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 0x7f6fc419ea10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc009fd0 tx=0x7f6fbc00ef10 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 shutdown_connections 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6fa80a2880 0x7f6fa80a4d40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6fc4102d00 0x7f6fc419ea10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6fc4102340 0x7f6fc41979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 --2- 172.21.15.46:0/2552936870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6fc4101150 0x7f6fc4197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.042+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 >> 172.21.15.46:0/2552936870 conn(0x7f6fc40fc6c0 msgr2=0x7f6fc4105640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.043+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 shutdown_connections 2024-03-08T23:07:32.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.043+0000 7f6fc8e00640 1 -- 172.21.15.46:0/2552936870 wait complete. 2024-03-08T23:07:32.066 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-08T23:07:31.251232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663280,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1031,"num_read_kb":645,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365816,"kb_used_data":12452,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576008,"statfs":{"total":1151940427776,"available":1151565832192,"internally_reserved":0,"allocated":12750848,"data_stored":8199578,"data_compressed":432612,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5683,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005317"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295735+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:06:41.295735+0000","last_peered":"2024-03-08T23:06:41.295735+0000","last_clean":"2024-03-08T23:06:41.295735+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:06:41.295735+0000","last_undegraded":"2024-03-08T23:06:41.295735+0000","last_fullsized":"2024-03-08T23:06:41.295735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561180+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:06:41.561180+0000","last_peered":"2024-03-08T23:06:41.561180+0000","last_clean":"2024-03-08T23:06:41.561180+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:06:41.561180+0000","last_undegraded":"2024-03-08T23:06:41.561180+0000","last_fullsized":"2024-03-08T23:06:41.561180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560821+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:06:41.560821+0000","last_peered":"2024-03-08T23:06:41.560821+0000","last_clean":"2024-03-08T23:06:41.560821+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:06:41.560821+0000","last_undegraded":"2024-03-08T23:06:41.560821+0000","last_fullsized":"2024-03-08T23:06:41.560821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311586+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:06:41.311586+0000","last_peered":"2024-03-08T23:06:41.311586+0000","last_clean":"2024-03-08T23:06:41.311586+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:06:41.311586+0000","last_undegraded":"2024-03-08T23:06:41.311586+0000","last_fullsized":"2024-03-08T23:06:41.311586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297952+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:06:41.297952+0000","last_peered":"2024-03-08T23:06:41.297952+0000","last_clean":"2024-03-08T23:06:41.297952+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:06:41.297952+0000","last_undegraded":"2024-03-08T23:06:41.297952+0000","last_fullsized":"2024-03-08T23:06:41.297952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293959+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:06:41.293959+0000","last_peered":"2024-03-08T23:06:41.293959+0000","last_clean":"2024-03-08T23:06:41.293959+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:06:41.293959+0000","last_undegraded":"2024-03-08T23:06:41.293959+0000","last_fullsized":"2024-03-08T23:06:41.293959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434708+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:06:41.434708+0000","last_peered":"2024-03-08T23:06:41.434708+0000","last_clean":"2024-03-08T23:06:41.434708+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:06:41.434708+0000","last_undegraded":"2024-03-08T23:06:41.434708+0000","last_fullsized":"2024-03-08T23:06:41.434708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293197+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:06:41.293197+0000","last_peered":"2024-03-08T23:06:41.293197+0000","last_clean":"2024-03-08T23:06:41.293197+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:06:41.293197+0000","last_undegraded":"2024-03-08T23:06:41.293197+0000","last_fullsized":"2024-03-08T23:06:41.293197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298020+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:06:41.298020+0000","last_peered":"2024-03-08T23:06:41.298020+0000","last_clean":"2024-03-08T23:06:41.298020+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:06:41.298020+0000","last_undegraded":"2024-03-08T23:06:41.298020+0000","last_fullsized":"2024-03-08T23:06:41.298020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294363+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:06:41.294363+0000","last_peered":"2024-03-08T23:06:41.294363+0000","last_clean":"2024-03-08T23:06:41.294363+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:06:41.294363+0000","last_undegraded":"2024-03-08T23:06:41.294363+0000","last_fullsized":"2024-03-08T23:06:41.294363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435262+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:06:41.435262+0000","last_peered":"2024-03-08T23:06:41.435262+0000","last_clean":"2024-03-08T23:06:41.435262+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:06:41.435262+0000","last_undegraded":"2024-03-08T23:06:41.435262+0000","last_fullsized":"2024-03-08T23:06:41.435262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311614+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:06:41.311614+0000","last_peered":"2024-03-08T23:06:41.311614+0000","last_clean":"2024-03-08T23:06:41.311614+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:06:41.311614+0000","last_undegraded":"2024-03-08T23:06:41.311614+0000","last_fullsized":"2024-03-08T23:06:41.311614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308034+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:06:41.308034+0000","last_peered":"2024-03-08T23:06:41.308034+0000","last_clean":"2024-03-08T23:06:41.308034+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:06:41.308034+0000","last_undegraded":"2024-03-08T23:06:41.308034+0000","last_fullsized":"2024-03-08T23:06:41.308034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559587+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:06:41.559587+0000","last_peered":"2024-03-08T23:06:41.559587+0000","last_clean":"2024-03-08T23:06:41.559587+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:06:41.559587+0000","last_undegraded":"2024-03-08T23:06:41.559587+0000","last_fullsized":"2024-03-08T23:06:41.559587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559494+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:06:41.559494+0000","last_peered":"2024-03-08T23:06:41.559494+0000","last_clean":"2024-03-08T23:06:41.559494+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:06:41.559494+0000","last_undegraded":"2024-03-08T23:06:41.559494+0000","last_fullsized":"2024-03-08T23:06:41.559494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297182+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:06:41.297182+0000","last_peered":"2024-03-08T23:06:41.297182+0000","last_clean":"2024-03-08T23:06:41.297182+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:06:41.297182+0000","last_undegraded":"2024-03-08T23:06:41.297182+0000","last_fullsized":"2024-03-08T23:06:41.297182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295505+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:06:41.295505+0000","last_peered":"2024-03-08T23:06:41.295505+0000","last_clean":"2024-03-08T23:06:41.295505+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:06:41.295505+0000","last_undegraded":"2024-03-08T23:06:41.295505+0000","last_fullsized":"2024-03-08T23:06:41.295505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437926+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:06:41.437926+0000","last_peered":"2024-03-08T23:06:41.437926+0000","last_clean":"2024-03-08T23:06:41.437926+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:06:41.437926+0000","last_undegraded":"2024-03-08T23:06:41.437926+0000","last_fullsized":"2024-03-08T23:06:41.437926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298548+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:06:41.298548+0000","last_peered":"2024-03-08T23:06:41.298548+0000","last_clean":"2024-03-08T23:06:41.298548+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:06:41.298548+0000","last_undegraded":"2024-03-08T23:06:41.298548+0000","last_fullsized":"2024-03-08T23:06:41.298548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298912+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:06:41.298912+0000","last_peered":"2024-03-08T23:06:41.298912+0000","last_clean":"2024-03-08T23:06:41.298912+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:06:41.298912+0000","last_undegraded":"2024-03-08T23:06:41.298912+0000","last_fullsized":"2024-03-08T23:06:41.298912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.438077+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:06:41.438077+0000","last_peered":"2024-03-08T23:06:41.438077+0000","last_clean":"2024-03-08T23:06:41.438077+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:06:41.438077+0000","last_undegraded":"2024-03-08T23:06:41.438077+0000","last_fullsized":"2024-03-08T23:06:41.438077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294192+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:06:41.294192+0000","last_peered":"2024-03-08T23:06:41.294192+0000","last_clean":"2024-03-08T23:06:41.294192+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:06:41.294192+0000","last_undegraded":"2024-03-08T23:06:41.294192+0000","last_fullsized":"2024-03-08T23:06:41.294192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311090+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:06:41.311090+0000","last_peered":"2024-03-08T23:06:41.311090+0000","last_clean":"2024-03-08T23:06:41.311090+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:06:41.311090+0000","last_undegraded":"2024-03-08T23:06:41.311090+0000","last_fullsized":"2024-03-08T23:06:41.311090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294294+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:06:41.294294+0000","last_peered":"2024-03-08T23:06:41.294294+0000","last_clean":"2024-03-08T23:06:41.294294+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:06:41.294294+0000","last_undegraded":"2024-03-08T23:06:41.294294+0000","last_fullsized":"2024-03-08T23:06:41.294294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560057+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:06:41.560057+0000","last_peered":"2024-03-08T23:06:41.560057+0000","last_clean":"2024-03-08T23:06:41.560057+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:06:41.560057+0000","last_undegraded":"2024-03-08T23:06:41.560057+0000","last_fullsized":"2024-03-08T23:06:41.560057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311255+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:06:41.311255+0000","last_peered":"2024-03-08T23:06:41.311255+0000","last_clean":"2024-03-08T23:06:41.311255+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:06:41.311255+0000","last_undegraded":"2024-03-08T23:06:41.311255+0000","last_fullsized":"2024-03-08T23:06:41.311255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298754+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:06:41.298754+0000","last_peered":"2024-03-08T23:06:41.298754+0000","last_clean":"2024-03-08T23:06:41.298754+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:06:41.298754+0000","last_undegraded":"2024-03-08T23:06:41.298754+0000","last_fullsized":"2024-03-08T23:06:41.298754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297580+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:06:41.297580+0000","last_peered":"2024-03-08T23:06:41.297580+0000","last_clean":"2024-03-08T23:06:41.297580+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:06:41.297580+0000","last_undegraded":"2024-03-08T23:06:41.297580+0000","last_fullsized":"2024-03-08T23:06:41.297580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311187+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:06:41.311187+0000","last_peered":"2024-03-08T23:06:41.311187+0000","last_clean":"2024-03-08T23:06:41.311187+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:06:41.311187+0000","last_undegraded":"2024-03-08T23:06:41.311187+0000","last_fullsized":"2024-03-08T23:06:41.311187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.302293+0000","last_change":"2024-03-08T23:06:41.302293+0000","last_active":"2024-03-08T23:06:41.302293+0000","last_peered":"2024-03-08T23:06:41.302293+0000","last_clean":"2024-03-08T23:06:41.302293+0000","last_became_active":"2024-03-08T23:06:41.298282+0000","last_became_peered":"2024-03-08T23:06:41.298282+0000","last_unstale":"2024-03-08T23:06:41.302293+0000","last_undegraded":"2024-03-08T23:06:41.302293+0000","last_fullsized":"2024-03-08T23:06:41.302293+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:54:03.637871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293427+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:06:41.293427+0000","last_peered":"2024-03-08T23:06:41.293427+0000","last_clean":"2024-03-08T23:06:41.293427+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:06:41.293427+0000","last_undegraded":"2024-03-08T23:06:41.293427+0000","last_fullsized":"2024-03-08T23:06:41.293427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298352+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:06:41.298352+0000","last_peered":"2024-03-08T23:06:41.298352+0000","last_clean":"2024-03-08T23:06:41.298352+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:06:41.298352+0000","last_undegraded":"2024-03-08T23:06:41.298352+0000","last_fullsized":"2024-03-08T23:06:41.298352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:27.310084+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:07:27.310084+0000","last_peered":"2024-03-08T23:07:27.310084+0000","last_clean":"2024-03-08T23:07:27.310084+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:07:27.310084+0000","last_undegraded":"2024-03-08T23:07:27.310084+0000","last_fullsized":"2024-03-08T23:07:27.310084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298203+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:06:41.298203+0000","last_peered":"2024-03-08T23:06:41.298203+0000","last_clean":"2024-03-08T23:06:41.298203+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:06:41.298203+0000","last_undegraded":"2024-03-08T23:06:41.298203+0000","last_fullsized":"2024-03-08T23:06:41.298203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298560+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:06:41.298560+0000","last_peered":"2024-03-08T23:06:41.298560+0000","last_clean":"2024-03-08T23:06:41.298560+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:06:41.298560+0000","last_undegraded":"2024-03-08T23:06:41.298560+0000","last_fullsized":"2024-03-08T23:06:41.298560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293814+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:06:41.293814+0000","last_peered":"2024-03-08T23:06:41.293814+0000","last_clean":"2024-03-08T23:06:41.293814+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:06:41.293814+0000","last_undegraded":"2024-03-08T23:06:41.293814+0000","last_fullsized":"2024-03-08T23:06:41.293814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293533+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:06:41.293533+0000","last_peered":"2024-03-08T23:06:41.293533+0000","last_clean":"2024-03-08T23:06:41.293533+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:06:41.293533+0000","last_undegraded":"2024-03-08T23:06:41.293533+0000","last_fullsized":"2024-03-08T23:06:41.293533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295372+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:06:41.295372+0000","last_peered":"2024-03-08T23:06:41.295372+0000","last_clean":"2024-03-08T23:06:41.295372+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:06:41.295372+0000","last_undegraded":"2024-03-08T23:06:41.295372+0000","last_fullsized":"2024-03-08T23:06:41.295372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307289+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:06:41.307289+0000","last_peered":"2024-03-08T23:06:41.307289+0000","last_clean":"2024-03-08T23:06:41.307289+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:06:41.307289+0000","last_undegraded":"2024-03-08T23:06:41.307289+0000","last_fullsized":"2024-03-08T23:06:41.307289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293298+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:06:41.293298+0000","last_peered":"2024-03-08T23:06:41.293298+0000","last_clean":"2024-03-08T23:06:41.293298+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:06:41.293298+0000","last_undegraded":"2024-03-08T23:06:41.293298+0000","last_fullsized":"2024-03-08T23:06:41.293298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561119+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:06:41.561119+0000","last_peered":"2024-03-08T23:06:41.561119+0000","last_clean":"2024-03-08T23:06:41.561119+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:06:41.561119+0000","last_undegraded":"2024-03-08T23:06:41.561119+0000","last_fullsized":"2024-03-08T23:06:41.561119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293176+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:06:41.293176+0000","last_peered":"2024-03-08T23:06:41.293176+0000","last_clean":"2024-03-08T23:06:41.293176+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:06:41.293176+0000","last_undegraded":"2024-03-08T23:06:41.293176+0000","last_fullsized":"2024-03-08T23:06:41.293176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293191+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:06:41.293191+0000","last_peered":"2024-03-08T23:06:41.293191+0000","last_clean":"2024-03-08T23:06:41.293191+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:06:41.293191+0000","last_undegraded":"2024-03-08T23:06:41.293191+0000","last_fullsized":"2024-03-08T23:06:41.293191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":180,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.311328+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:07:28.311328+0000","last_peered":"2024-03-08T23:07:28.311328+0000","last_clean":"2024-03-08T23:07:28.311328+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:07:28.311328+0000","last_undegraded":"2024-03-08T23:07:28.311328+0000","last_fullsized":"2024-03-08T23:07:28.311328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297904+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:06:41.297904+0000","last_peered":"2024-03-08T23:06:41.297904+0000","last_clean":"2024-03-08T23:06:41.297904+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:06:41.297904+0000","last_undegraded":"2024-03-08T23:06:41.297904+0000","last_fullsized":"2024-03-08T23:06:41.297904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435215+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:06:41.435215+0000","last_peered":"2024-03-08T23:06:41.435215+0000","last_clean":"2024-03-08T23:06:41.435215+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:06:41.435215+0000","last_undegraded":"2024-03-08T23:06:41.435215+0000","last_fullsized":"2024-03-08T23:06:41.435215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560329+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:06:41.560329+0000","last_peered":"2024-03-08T23:06:41.560329+0000","last_clean":"2024-03-08T23:06:41.560329+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:06:41.560329+0000","last_undegraded":"2024-03-08T23:06:41.560329+0000","last_fullsized":"2024-03-08T23:06:41.560329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311591+0000","last_change":"2024-03-08T23:05:40.884963+0000","last_active":"2024-03-08T23:06:41.311591+0000","last_peered":"2024-03-08T23:06:41.311591+0000","last_clean":"2024-03-08T23:06:41.311591+0000","last_became_active":"2024-03-08T23:05:40.884402+0000","last_became_peered":"2024-03-08T23:05:40.884402+0000","last_unstale":"2024-03-08T23:06:41.311591+0000","last_undegraded":"2024-03-08T23:06:41.311591+0000","last_fullsized":"2024-03-08T23:06:41.311591+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:26.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293388+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:06:41.293388+0000","last_peered":"2024-03-08T23:06:41.293388+0000","last_clean":"2024-03-08T23:06:41.293388+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:06:41.293388+0000","last_undegraded":"2024-03-08T23:06:41.293388+0000","last_fullsized":"2024-03-08T23:06:41.293388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297713+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:06:41.297713+0000","last_peered":"2024-03-08T23:06:41.297713+0000","last_clean":"2024-03-08T23:06:41.297713+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:06:41.297713+0000","last_undegraded":"2024-03-08T23:06:41.297713+0000","last_fullsized":"2024-03-08T23:06:41.297713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293560+0000","last_change":"2024-03-08T23:05:41.148144+0000","last_active":"2024-03-08T23:06:41.293560+0000","last_peered":"2024-03-08T23:06:41.293560+0000","last_clean":"2024-03-08T23:06:41.293560+0000","last_became_active":"2024-03-08T23:05:41.147760+0000","last_became_peered":"2024-03-08T23:05:41.147760+0000","last_unstale":"2024-03-08T23:06:41.293560+0000","last_undegraded":"2024-03-08T23:06:41.293560+0000","last_fullsized":"2024-03-08T23:06:41.293560+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:00.345274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":369,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:29.312829+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:07:29.312829+0000","last_peered":"2024-03-08T23:07:29.312829+0000","last_clean":"2024-03-08T23:07:29.312829+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:07:29.312829+0000","last_undegraded":"2024-03-08T23:07:29.312829+0000","last_fullsized":"2024-03-08T23:07:29.312829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68004,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":300,"num_read_kb":218,"num_write":38,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298546+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:06:41.298546+0000","last_peered":"2024-03-08T23:06:41.298546+0000","last_clean":"2024-03-08T23:06:41.298546+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:06:41.298546+0000","last_undegraded":"2024-03-08T23:06:41.298546+0000","last_fullsized":"2024-03-08T23:06:41.298546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293944+0000","last_change":"2024-03-08T23:05:40.659738+0000","last_active":"2024-03-08T23:06:41.293944+0000","last_peered":"2024-03-08T23:06:41.293944+0000","last_clean":"2024-03-08T23:06:41.293944+0000","last_became_active":"2024-03-08T23:05:40.659391+0000","last_became_peered":"2024-03-08T23:05:40.659391+0000","last_unstale":"2024-03-08T23:06:41.293944+0000","last_undegraded":"2024-03-08T23:06:41.293944+0000","last_fullsized":"2024-03-08T23:06:41.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:42:03.247682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298136+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:06:41.298136+0000","last_peered":"2024-03-08T23:06:41.298136+0000","last_clean":"2024-03-08T23:06:41.298136+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:06:41.298136+0000","last_undegraded":"2024-03-08T23:06:41.298136+0000","last_fullsized":"2024-03-08T23:06:41.298136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307379+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:06:41.307379+0000","last_peered":"2024-03-08T23:06:41.307379+0000","last_clean":"2024-03-08T23:06:41.307379+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:06:41.307379+0000","last_undegraded":"2024-03-08T23:06:41.307379+0000","last_fullsized":"2024-03-08T23:06:41.307379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295980+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:06:41.295980+0000","last_peered":"2024-03-08T23:06:41.295980+0000","last_clean":"2024-03-08T23:06:41.295980+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:06:41.295980+0000","last_undegraded":"2024-03-08T23:06:41.295980+0000","last_fullsized":"2024-03-08T23:06:41.295980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293269+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:06:41.293269+0000","last_peered":"2024-03-08T23:06:41.293269+0000","last_clean":"2024-03-08T23:06:41.293269+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:06:41.293269+0000","last_undegraded":"2024-03-08T23:06:41.293269+0000","last_fullsized":"2024-03-08T23:06:41.293269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435039+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:06:41.435039+0000","last_peered":"2024-03-08T23:06:41.435039+0000","last_clean":"2024-03-08T23:06:41.435039+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:06:41.435039+0000","last_undegraded":"2024-03-08T23:06:41.435039+0000","last_fullsized":"2024-03-08T23:06:41.435039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437450+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:06:41.437450+0000","last_peered":"2024-03-08T23:06:41.437450+0000","last_clean":"2024-03-08T23:06:41.437450+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:06:41.437450+0000","last_undegraded":"2024-03-08T23:06:41.437450+0000","last_fullsized":"2024-03-08T23:06:41.437450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":185,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.418460+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:07:28.418460+0000","last_peered":"2024-03-08T23:07:28.418460+0000","last_clean":"2024-03-08T23:07:28.418460+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:07:28.418460+0000","last_undegraded":"2024-03-08T23:07:28.418460+0000","last_fullsized":"2024-03-08T23:07:28.418460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434833+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:06:41.434833+0000","last_peered":"2024-03-08T23:06:41.434833+0000","last_clean":"2024-03-08T23:06:41.434833+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:06:41.434833+0000","last_undegraded":"2024-03-08T23:06:41.434833+0000","last_fullsized":"2024-03-08T23:06:41.434833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308276+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:06:41.308276+0000","last_peered":"2024-03-08T23:06:41.308276+0000","last_clean":"2024-03-08T23:06:41.308276+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:06:41.308276+0000","last_undegraded":"2024-03-08T23:06:41.308276+0000","last_fullsized":"2024-03-08T23:06:41.308276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297956+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:06:41.297956+0000","last_peered":"2024-03-08T23:06:41.297956+0000","last_clean":"2024-03-08T23:06:41.297956+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:06:41.297956+0000","last_undegraded":"2024-03-08T23:06:41.297956+0000","last_fullsized":"2024-03-08T23:06:41.297956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292974+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:06:41.292974+0000","last_peered":"2024-03-08T23:06:41.292974+0000","last_clean":"2024-03-08T23:06:41.292974+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:06:41.292974+0000","last_undegraded":"2024-03-08T23:06:41.292974+0000","last_fullsized":"2024-03-08T23:06:41.292974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292888+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:06:41.292888+0000","last_peered":"2024-03-08T23:06:41.292888+0000","last_clean":"2024-03-08T23:06:41.292888+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:06:41.292888+0000","last_undegraded":"2024-03-08T23:06:41.292888+0000","last_fullsized":"2024-03-08T23:06:41.292888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298952+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:06:41.298952+0000","last_peered":"2024-03-08T23:06:41.298952+0000","last_clean":"2024-03-08T23:06:41.298952+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:06:41.298952+0000","last_undegraded":"2024-03-08T23:06:41.298952+0000","last_fullsized":"2024-03-08T23:06:41.298952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295447+0000","last_change":"2024-03-08T23:05:41.148194+0000","last_active":"2024-03-08T23:06:41.295447+0000","last_peered":"2024-03-08T23:06:41.295447+0000","last_clean":"2024-03-08T23:06:41.295447+0000","last_became_active":"2024-03-08T23:05:41.147832+0000","last_became_peered":"2024-03-08T23:05:41.147832+0000","last_unstale":"2024-03-08T23:06:41.295447+0000","last_undegraded":"2024-03-08T23:06:41.295447+0000","last_fullsized":"2024-03-08T23:06:41.295447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:38.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298118+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:06:41.298118+0000","last_peered":"2024-03-08T23:06:41.298118+0000","last_clean":"2024-03-08T23:06:41.298118+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:06:41.298118+0000","last_undegraded":"2024-03-08T23:06:41.298118+0000","last_fullsized":"2024-03-08T23:06:41.298118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437182+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:06:41.437182+0000","last_peered":"2024-03-08T23:06:41.437182+0000","last_clean":"2024-03-08T23:06:41.437182+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:06:41.437182+0000","last_undegraded":"2024-03-08T23:06:41.437182+0000","last_fullsized":"2024-03-08T23:06:41.437182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294258+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:06:41.294258+0000","last_peered":"2024-03-08T23:06:41.294258+0000","last_clean":"2024-03-08T23:06:41.294258+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:06:41.294258+0000","last_undegraded":"2024-03-08T23:06:41.294258+0000","last_fullsized":"2024-03-08T23:06:41.294258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307765+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:06:41.307765+0000","last_peered":"2024-03-08T23:06:41.307765+0000","last_clean":"2024-03-08T23:06:41.307765+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:06:41.307765+0000","last_undegraded":"2024-03-08T23:06:41.307765+0000","last_fullsized":"2024-03-08T23:06:41.307765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434782+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:06:41.434782+0000","last_peered":"2024-03-08T23:06:41.434782+0000","last_clean":"2024-03-08T23:06:41.434782+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:06:41.434782+0000","last_undegraded":"2024-03-08T23:06:41.434782+0000","last_fullsized":"2024-03-08T23:06:41.434782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293784+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:06:41.293784+0000","last_peered":"2024-03-08T23:06:41.293784+0000","last_clean":"2024-03-08T23:06:41.293784+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:06:41.293784+0000","last_undegraded":"2024-03-08T23:06:41.293784+0000","last_fullsized":"2024-03-08T23:06:41.293784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308278+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:06:41.308278+0000","last_peered":"2024-03-08T23:06:41.308278+0000","last_clean":"2024-03-08T23:06:41.308278+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:06:41.308278+0000","last_undegraded":"2024-03-08T23:06:41.308278+0000","last_fullsized":"2024-03-08T23:06:41.308278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293071+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:06:41.293071+0000","last_peered":"2024-03-08T23:06:41.293071+0000","last_clean":"2024-03-08T23:06:41.293071+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:06:41.293071+0000","last_undegraded":"2024-03-08T23:06:41.293071+0000","last_fullsized":"2024-03-08T23:06:41.293071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":179,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:26.414027+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:07:26.414027+0000","last_peered":"2024-03-08T23:07:26.414027+0000","last_clean":"2024-03-08T23:07:26.414027+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:07:26.414027+0000","last_undegraded":"2024-03-08T23:07:26.414027+0000","last_fullsized":"2024-03-08T23:07:26.414027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435137+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:06:41.435137+0000","last_peered":"2024-03-08T23:06:41.435137+0000","last_clean":"2024-03-08T23:06:41.435137+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:06:41.435137+0000","last_undegraded":"2024-03-08T23:06:41.435137+0000","last_fullsized":"2024-03-08T23:06:41.435137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293672+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:06:41.293672+0000","last_peered":"2024-03-08T23:06:41.293672+0000","last_clean":"2024-03-08T23:06:41.293672+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:06:41.293672+0000","last_undegraded":"2024-03-08T23:06:41.293672+0000","last_fullsized":"2024-03-08T23:06:41.293672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311338+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:06:41.311338+0000","last_peered":"2024-03-08T23:06:41.311338+0000","last_clean":"2024-03-08T23:06:41.311338+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:06:41.311338+0000","last_undegraded":"2024-03-08T23:06:41.311338+0000","last_fullsized":"2024-03-08T23:06:41.311338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434495+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:06:41.434495+0000","last_peered":"2024-03-08T23:06:41.434495+0000","last_clean":"2024-03-08T23:06:41.434495+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:06:41.434495+0000","last_undegraded":"2024-03-08T23:06:41.434495+0000","last_fullsized":"2024-03-08T23:06:41.434495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293727+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:06:41.293727+0000","last_peered":"2024-03-08T23:06:41.293727+0000","last_clean":"2024-03-08T23:06:41.293727+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:06:41.293727+0000","last_undegraded":"2024-03-08T23:06:41.293727+0000","last_fullsized":"2024-03-08T23:06:41.293727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307136+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:06:41.307136+0000","last_peered":"2024-03-08T23:06:41.307136+0000","last_clean":"2024-03-08T23:06:41.307136+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:06:41.307136+0000","last_undegraded":"2024-03-08T23:06:41.307136+0000","last_fullsized":"2024-03-08T23:06:41.307136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434597+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:06:41.434597+0000","last_peered":"2024-03-08T23:06:41.434597+0000","last_clean":"2024-03-08T23:06:41.434597+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:06:41.434597+0000","last_undegraded":"2024-03-08T23:06:41.434597+0000","last_fullsized":"2024-03-08T23:06:41.434597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295247+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:06:41.295247+0000","last_peered":"2024-03-08T23:06:41.295247+0000","last_clean":"2024-03-08T23:06:41.295247+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:06:41.295247+0000","last_undegraded":"2024-03-08T23:06:41.295247+0000","last_fullsized":"2024-03-08T23:06:41.295247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298642+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:06:41.298642+0000","last_peered":"2024-03-08T23:06:41.298642+0000","last_clean":"2024-03-08T23:06:41.298642+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:06:41.298642+0000","last_undegraded":"2024-03-08T23:06:41.298642+0000","last_fullsized":"2024-03-08T23:06:41.298642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307927+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:06:41.307927+0000","last_peered":"2024-03-08T23:06:41.307927+0000","last_clean":"2024-03-08T23:06:41.307927+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:06:41.307927+0000","last_undegraded":"2024-03-08T23:06:41.307927+0000","last_fullsized":"2024-03-08T23:06:41.307927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293158+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:06:41.293158+0000","last_peered":"2024-03-08T23:06:41.293158+0000","last_clean":"2024-03-08T23:06:41.293158+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:06:41.293158+0000","last_undegraded":"2024-03-08T23:06:41.293158+0000","last_fullsized":"2024-03-08T23:06:41.293158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'11","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:49.177474+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:06:49.177474+0000","last_peered":"2024-03-08T23:06:49.177474+0000","last_clean":"2024-03-08T23:06:49.177474+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:06:49.177474+0000","last_undegraded":"2024-03-08T23:06:49.177474+0000","last_fullsized":"2024-03-08T23:06:49.177474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294465+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:06:41.294465+0000","last_peered":"2024-03-08T23:06:41.294465+0000","last_clean":"2024-03-08T23:06:41.294465+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:06:41.294465+0000","last_undegraded":"2024-03-08T23:06:41.294465+0000","last_fullsized":"2024-03-08T23:06:41.294465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298315+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:06:41.298315+0000","last_peered":"2024-03-08T23:06:41.298315+0000","last_clean":"2024-03-08T23:06:41.298315+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:06:41.298315+0000","last_undegraded":"2024-03-08T23:06:41.298315+0000","last_fullsized":"2024-03-08T23:06:41.298315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311691+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:06:41.311691+0000","last_peered":"2024-03-08T23:06:41.311691+0000","last_clean":"2024-03-08T23:06:41.311691+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:06:41.311691+0000","last_undegraded":"2024-03-08T23:06:41.311691+0000","last_fullsized":"2024-03-08T23:06:41.311691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561285+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:06:41.561285+0000","last_peered":"2024-03-08T23:06:41.561285+0000","last_clean":"2024-03-08T23:06:41.561285+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:06:41.561285+0000","last_undegraded":"2024-03-08T23:06:41.561285+0000","last_fullsized":"2024-03-08T23:06:41.561285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311484+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:06:41.311484+0000","last_peered":"2024-03-08T23:06:41.311484+0000","last_clean":"2024-03-08T23:06:41.311484+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:06:41.311484+0000","last_undegraded":"2024-03-08T23:06:41.311484+0000","last_fullsized":"2024-03-08T23:06:41.311484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297243+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:06:41.297243+0000","last_peered":"2024-03-08T23:06:41.297243+0000","last_clean":"2024-03-08T23:06:41.297243+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:06:41.297243+0000","last_undegraded":"2024-03-08T23:06:41.297243+0000","last_fullsized":"2024-03-08T23:06:41.297243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297758+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:06:41.297758+0000","last_peered":"2024-03-08T23:06:41.297758+0000","last_clean":"2024-03-08T23:06:41.297758+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:06:41.297758+0000","last_undegraded":"2024-03-08T23:06:41.297758+0000","last_fullsized":"2024-03-08T23:06:41.297758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298880+0000","last_change":"2024-03-08T23:05:41.148196+0000","last_active":"2024-03-08T23:06:41.298880+0000","last_peered":"2024-03-08T23:06:41.298880+0000","last_clean":"2024-03-08T23:06:41.298880+0000","last_became_active":"2024-03-08T23:05:41.147730+0000","last_became_peered":"2024-03-08T23:05:41.147730+0000","last_unstale":"2024-03-08T23:06:41.298880+0000","last_undegraded":"2024-03-08T23:06:41.298880+0000","last_fullsized":"2024-03-08T23:06:41.298880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:21:05.284970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293328+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:06:41.293328+0000","last_peered":"2024-03-08T23:06:41.293328+0000","last_clean":"2024-03-08T23:06:41.293328+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:06:41.293328+0000","last_undegraded":"2024-03-08T23:06:41.293328+0000","last_fullsized":"2024-03-08T23:06:41.293328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295142+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:06:41.295142+0000","last_peered":"2024-03-08T23:06:41.295142+0000","last_clean":"2024-03-08T23:06:41.295142+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:06:41.295142+0000","last_undegraded":"2024-03-08T23:06:41.295142+0000","last_fullsized":"2024-03-08T23:06:41.295142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311375+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:06:41.311375+0000","last_peered":"2024-03-08T23:06:41.311375+0000","last_clean":"2024-03-08T23:06:41.311375+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:06:41.311375+0000","last_undegraded":"2024-03-08T23:06:41.311375+0000","last_fullsized":"2024-03-08T23:06:41.311375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.304979+0000","last_change":"2024-03-08T23:06:41.304979+0000","last_active":"2024-03-08T23:06:41.304979+0000","last_peered":"2024-03-08T23:06:41.304979+0000","last_clean":"2024-03-08T23:06:41.304979+0000","last_became_active":"2024-03-08T23:06:41.300018+0000","last_became_peered":"2024-03-08T23:06:41.300018+0000","last_unstale":"2024-03-08T23:06:41.304979+0000","last_undegraded":"2024-03-08T23:06:41.304979+0000","last_fullsized":"2024-03-08T23:06:41.304979+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:16:30.599602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293922+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:06:41.293922+0000","last_peered":"2024-03-08T23:06:41.293922+0000","last_clean":"2024-03-08T23:06:41.293922+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:06:41.293922+0000","last_undegraded":"2024-03-08T23:06:41.293922+0000","last_fullsized":"2024-03-08T23:06:41.293922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311564+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:06:41.311564+0000","last_peered":"2024-03-08T23:06:41.311564+0000","last_clean":"2024-03-08T23:06:41.311564+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:06:41.311564+0000","last_undegraded":"2024-03-08T23:06:41.311564+0000","last_fullsized":"2024-03-08T23:06:41.311564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297806+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:06:41.297806+0000","last_peered":"2024-03-08T23:06:41.297806+0000","last_clean":"2024-03-08T23:06:41.297806+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:06:41.297806+0000","last_undegraded":"2024-03-08T23:06:41.297806+0000","last_fullsized":"2024-03-08T23:06:41.297806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434958+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:06:41.434958+0000","last_peered":"2024-03-08T23:06:41.434958+0000","last_clean":"2024-03-08T23:06:41.434958+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:06:41.434958+0000","last_undegraded":"2024-03-08T23:06:41.434958+0000","last_fullsized":"2024-03-08T23:06:41.434958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297364+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:06:41.297364+0000","last_peered":"2024-03-08T23:06:41.297364+0000","last_clean":"2024-03-08T23:06:41.297364+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:06:41.297364+0000","last_undegraded":"2024-03-08T23:06:41.297364+0000","last_fullsized":"2024-03-08T23:06:41.297364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298721+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:06:41.298721+0000","last_peered":"2024-03-08T23:06:41.298721+0000","last_clean":"2024-03-08T23:06:41.298721+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:06:41.298721+0000","last_undegraded":"2024-03-08T23:06:41.298721+0000","last_fullsized":"2024-03-08T23:06:41.298721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435322+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:06:41.435322+0000","last_peered":"2024-03-08T23:06:41.435322+0000","last_clean":"2024-03-08T23:06:41.435322+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:06:41.435322+0000","last_undegraded":"2024-03-08T23:06:41.435322+0000","last_fullsized":"2024-03-08T23:06:41.435322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559863+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:06:41.559863+0000","last_peered":"2024-03-08T23:06:41.559863+0000","last_clean":"2024-03-08T23:06:41.559863+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:06:41.559863+0000","last_undegraded":"2024-03-08T23:06:41.559863+0000","last_fullsized":"2024-03-08T23:06:41.559863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293626+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:06:41.293626+0000","last_peered":"2024-03-08T23:06:41.293626+0000","last_clean":"2024-03-08T23:06:41.293626+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:06:41.293626+0000","last_undegraded":"2024-03-08T23:06:41.293626+0000","last_fullsized":"2024-03-08T23:06:41.293626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435008+0000","last_change":"2024-03-08T23:05:41.148243+0000","last_active":"2024-03-08T23:06:41.435008+0000","last_peered":"2024-03-08T23:06:41.435008+0000","last_clean":"2024-03-08T23:06:41.435008+0000","last_became_active":"2024-03-08T23:05:41.147924+0000","last_became_peered":"2024-03-08T23:05:41.147924+0000","last_unstale":"2024-03-08T23:06:41.435008+0000","last_undegraded":"2024-03-08T23:06:41.435008+0000","last_fullsized":"2024-03-08T23:06:41.435008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:47.129092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292704+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:06:41.292704+0000","last_peered":"2024-03-08T23:06:41.292704+0000","last_clean":"2024-03-08T23:06:41.292704+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:06:41.292704+0000","last_undegraded":"2024-03-08T23:06:41.292704+0000","last_fullsized":"2024-03-08T23:06:41.292704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308063+0000","last_change":"2024-03-08T23:05:40.661217+0000","last_active":"2024-03-08T23:06:41.308063+0000","last_peered":"2024-03-08T23:06:41.308063+0000","last_clean":"2024-03-08T23:06:41.308063+0000","last_became_active":"2024-03-08T23:05:40.657165+0000","last_became_peered":"2024-03-08T23:05:40.657165+0000","last_unstale":"2024-03-08T23:06:41.308063+0000","last_undegraded":"2024-03-08T23:06:41.308063+0000","last_fullsized":"2024-03-08T23:06:41.308063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:16.125758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435283+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:06:41.435283+0000","last_peered":"2024-03-08T23:06:41.435283+0000","last_clean":"2024-03-08T23:06:41.435283+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:06:41.435283+0000","last_undegraded":"2024-03-08T23:06:41.435283+0000","last_fullsized":"2024-03-08T23:06:41.435283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295157+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:06:41.295157+0000","last_peered":"2024-03-08T23:06:41.295157+0000","last_clean":"2024-03-08T23:06:41.295157+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:06:41.295157+0000","last_undegraded":"2024-03-08T23:06:41.295157+0000","last_fullsized":"2024-03-08T23:06:41.295157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.299154+0000","last_change":"2024-03-08T23:05:40.651674+0000","last_active":"2024-03-08T23:06:41.299154+0000","last_peered":"2024-03-08T23:06:41.299154+0000","last_clean":"2024-03-08T23:06:41.299154+0000","last_became_active":"2024-03-08T23:05:40.651393+0000","last_became_peered":"2024-03-08T23:05:40.651393+0000","last_unstale":"2024-03-08T23:06:41.299154+0000","last_undegraded":"2024-03-08T23:06:41.299154+0000","last_fullsized":"2024-03-08T23:06:41.299154+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:30:44.691787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298225+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:06:41.298225+0000","last_peered":"2024-03-08T23:06:41.298225+0000","last_clean":"2024-03-08T23:06:41.298225+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:06:41.298225+0000","last_undegraded":"2024-03-08T23:06:41.298225+0000","last_fullsized":"2024-03-08T23:06:41.298225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560171+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:06:41.560171+0000","last_peered":"2024-03-08T23:06:41.560171+0000","last_clean":"2024-03-08T23:06:41.560171+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:06:41.560171+0000","last_undegraded":"2024-03-08T23:06:41.560171+0000","last_fullsized":"2024-03-08T23:06:41.560171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308038+0000","last_change":"2024-03-08T23:05:40.993334+0000","last_active":"2024-03-08T23:06:41.308038+0000","last_peered":"2024-03-08T23:06:41.308038+0000","last_clean":"2024-03-08T23:06:41.308038+0000","last_became_active":"2024-03-08T23:05:40.991755+0000","last_became_peered":"2024-03-08T23:05:40.991755+0000","last_unstale":"2024-03-08T23:06:41.308038+0000","last_undegraded":"2024-03-08T23:06:41.308038+0000","last_fullsized":"2024-03-08T23:06:41.308038+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:31.085388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295481+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:06:41.295481+0000","last_peered":"2024-03-08T23:06:41.295481+0000","last_clean":"2024-03-08T23:06:41.295481+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:06:41.295481+0000","last_undegraded":"2024-03-08T23:06:41.295481+0000","last_fullsized":"2024-03-08T23:06:41.295481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298991+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:06:41.298991+0000","last_peered":"2024-03-08T23:06:41.298991+0000","last_clean":"2024-03-08T23:06:41.298991+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:06:41.298991+0000","last_undegraded":"2024-03-08T23:06:41.298991+0000","last_fullsized":"2024-03-08T23:06:41.298991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294678+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:06:41.294678+0000","last_peered":"2024-03-08T23:06:41.294678+0000","last_clean":"2024-03-08T23:06:41.294678+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:06:41.294678+0000","last_undegraded":"2024-03-08T23:06:41.294678+0000","last_fullsized":"2024-03-08T23:06:41.294678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292981+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:06:41.292981+0000","last_peered":"2024-03-08T23:06:41.292981+0000","last_clean":"2024-03-08T23:06:41.292981+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:06:41.292981+0000","last_undegraded":"2024-03-08T23:06:41.292981+0000","last_fullsized":"2024-03-08T23:06:41.292981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293449+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:06:41.293449+0000","last_peered":"2024-03-08T23:06:41.293449+0000","last_clean":"2024-03-08T23:06:41.293449+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:06:41.293449+0000","last_undegraded":"2024-03-08T23:06:41.293449+0000","last_fullsized":"2024-03-08T23:06:41.293449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297515+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:06:41.297515+0000","last_peered":"2024-03-08T23:06:41.297515+0000","last_clean":"2024-03-08T23:06:41.297515+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:06:41.297515+0000","last_undegraded":"2024-03-08T23:06:41.297515+0000","last_fullsized":"2024-03-08T23:06:41.297515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307876+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:06:41.307876+0000","last_peered":"2024-03-08T23:06:41.307876+0000","last_clean":"2024-03-08T23:06:41.307876+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:06:41.307876+0000","last_undegraded":"2024-03-08T23:06:41.307876+0000","last_fullsized":"2024-03-08T23:06:41.307876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.296016+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:06:41.296016+0000","last_peered":"2024-03-08T23:06:41.296016+0000","last_clean":"2024-03-08T23:06:41.296016+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:06:41.296016+0000","last_undegraded":"2024-03-08T23:06:41.296016+0000","last_fullsized":"2024-03-08T23:06:41.296016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72678,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":895,"num_read_kb":526,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":218034,"data_compressed":5463,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383728,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":519484,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645365,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1175288,"data_compressed":64572,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":523816,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":519640,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332166,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1116222,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":582810,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":516284,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":584692,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515090,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520078,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757330,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":1111278,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68004,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69886,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70114,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:07:32.073 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T23:07:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 -- 172.21.15.46:0/4141093102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 msgr2=0x7f59580741d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 --2- 172.21.15.46:0/4141093102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f59580741d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5940009ca0 tx=0x7f594002c100 comp rx=0 tx=0).stop 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 -- 172.21.15.46:0/4141093102 shutdown_connections 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 --2- 172.21.15.46:0/4141093102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 0x7f595810fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 --2- 172.21.15.46:0/4141093102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f59580741d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 --2- 172.21.15.46:0/4141093102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f5958073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 -- 172.21.15.46:0/4141093102 >> 172.21.15.46:0/4141093102 conn(0x7f59580fc8a0 msgr2=0x7f59580fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 -- 172.21.15.46:0/4141093102 shutdown_connections 2024-03-08T23:07:32.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.320+0000 7f595d000640 1 -- 172.21.15.46:0/4141093102 wait complete. 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.321+0000 7f595d000640 1 Processor -- start 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.321+0000 7f595d000640 1 -- start start 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f595819dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 0x7f59581a51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5958113580 con 0x7f5958075330 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5958113400 con 0x7f595810d8e0 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f595d000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5958113700 con 0x7f5958073d50 2024-03-08T23:07:32.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36598/0 (socket says 172.21.15.46:36598) 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 -- 172.21.15.46:0/1045082464 learned_addr learned my addr 172.21.15.46:0/1045082464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f5955e00640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f595819dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f5956800640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 0x7f59581a51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 -- 172.21.15.46:0/1045082464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 msgr2=0x7f595819dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f595819dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 -- 172.21.15.46:0/1045082464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 msgr2=0x7f59581a51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.322+0000 7f594d400640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 0x7f59581a51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f594d400640 1 -- 172.21.15.46:0/1045082464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59581a5850 con 0x7f5958075330 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f594d400640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59400325f0 tx=0x7f5940034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940039050 con 0x7f5958075330 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59581a5ae0 con 0x7f5958075330 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5940045070 con 0x7f5958075330 2024-03-08T23:07:32.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5940004640 con 0x7f5958075330 2024-03-08T23:07:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.323+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59581a5fd0 con 0x7f5958075330 2024-03-08T23:07:32.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.325+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5958103ab0 con 0x7f5958075330 2024-03-08T23:07:32.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.326+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5940003970 con 0x7f5958075330 2024-03-08T23:07:32.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.328+0000 7f594f400640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 0x7f593c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.328+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f59400f1640 con 0x7f5958075330 2024-03-08T23:07:32.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.328+0000 7f5955e00640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 0x7f593c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.329+0000 7f5955e00640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 0x7f593c0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f595819f110 tx=0x7f5948005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:32.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.332+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f594004f020 con 0x7f5958075330 2024-03-08T23:07:32.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:32 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4206514842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:32.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:32 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4132078066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:32.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:32 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4206514842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:32.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:32 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4132078066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:32.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.532+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5958061730 con 0x7f593c0a2800 2024-03-08T23:07:32.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:32 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4206514842' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:32.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:32 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4132078066' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:32.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.562+0000 7f594f400640 1 -- 172.21.15.46:0/1045082464 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340800 (secure 0 0 0) 0x7f5958061730 con 0x7f593c0a2800 2024-03-08T23:07:32.569 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:32.573 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:07:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.570+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 msgr2=0x7f593c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 0x7f593c0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f595819f110 tx=0x7f5948005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 msgr2=0x7f595819e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f59400325f0 tx=0x7f5940034670 comp rx=0 tx=0).stop 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 shutdown_connections 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f593c0a2800 0x7f593c0a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f595810d8e0 0x7f59581a51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5958075330 0x7f595819e180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 --2- 172.21.15.46:0/1045082464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5958073d50 0x7f595819dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 >> 172.21.15.46:0/1045082464 conn(0x7f59580fc8a0 msgr2=0x7f5958100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 shutdown_connections 2024-03-08T23:07:32.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.571+0000 7f595d000640 1 -- 172.21.15.46:0/1045082464 wait complete. 2024-03-08T23:07:32.598 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-08T23:07:31.251232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663280,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1031,"num_read_kb":645,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365816,"kb_used_data":12452,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576008,"statfs":{"total":1151940427776,"available":1151565832192,"internally_reserved":0,"allocated":12750848,"data_stored":8199578,"data_compressed":432612,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5683,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005317"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295735+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:06:41.295735+0000","last_peered":"2024-03-08T23:06:41.295735+0000","last_clean":"2024-03-08T23:06:41.295735+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:06:41.295735+0000","last_undegraded":"2024-03-08T23:06:41.295735+0000","last_fullsized":"2024-03-08T23:06:41.295735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561180+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:06:41.561180+0000","last_peered":"2024-03-08T23:06:41.561180+0000","last_clean":"2024-03-08T23:06:41.561180+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:06:41.561180+0000","last_undegraded":"2024-03-08T23:06:41.561180+0000","last_fullsized":"2024-03-08T23:06:41.561180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560821+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:06:41.560821+0000","last_peered":"2024-03-08T23:06:41.560821+0000","last_clean":"2024-03-08T23:06:41.560821+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:06:41.560821+0000","last_undegraded":"2024-03-08T23:06:41.560821+0000","last_fullsized":"2024-03-08T23:06:41.560821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311586+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:06:41.311586+0000","last_peered":"2024-03-08T23:06:41.311586+0000","last_clean":"2024-03-08T23:06:41.311586+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:06:41.311586+0000","last_undegraded":"2024-03-08T23:06:41.311586+0000","last_fullsized":"2024-03-08T23:06:41.311586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297952+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:06:41.297952+0000","last_peered":"2024-03-08T23:06:41.297952+0000","last_clean":"2024-03-08T23:06:41.297952+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:06:41.297952+0000","last_undegraded":"2024-03-08T23:06:41.297952+0000","last_fullsized":"2024-03-08T23:06:41.297952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293959+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:06:41.293959+0000","last_peered":"2024-03-08T23:06:41.293959+0000","last_clean":"2024-03-08T23:06:41.293959+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:06:41.293959+0000","last_undegraded":"2024-03-08T23:06:41.293959+0000","last_fullsized":"2024-03-08T23:06:41.293959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434708+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:06:41.434708+0000","last_peered":"2024-03-08T23:06:41.434708+0000","last_clean":"2024-03-08T23:06:41.434708+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:06:41.434708+0000","last_undegraded":"2024-03-08T23:06:41.434708+0000","last_fullsized":"2024-03-08T23:06:41.434708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293197+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:06:41.293197+0000","last_peered":"2024-03-08T23:06:41.293197+0000","last_clean":"2024-03-08T23:06:41.293197+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:06:41.293197+0000","last_undegraded":"2024-03-08T23:06:41.293197+0000","last_fullsized":"2024-03-08T23:06:41.293197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298020+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:06:41.298020+0000","last_peered":"2024-03-08T23:06:41.298020+0000","last_clean":"2024-03-08T23:06:41.298020+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:06:41.298020+0000","last_undegraded":"2024-03-08T23:06:41.298020+0000","last_fullsized":"2024-03-08T23:06:41.298020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294363+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:06:41.294363+0000","last_peered":"2024-03-08T23:06:41.294363+0000","last_clean":"2024-03-08T23:06:41.294363+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:06:41.294363+0000","last_undegraded":"2024-03-08T23:06:41.294363+0000","last_fullsized":"2024-03-08T23:06:41.294363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435262+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:06:41.435262+0000","last_peered":"2024-03-08T23:06:41.435262+0000","last_clean":"2024-03-08T23:06:41.435262+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:06:41.435262+0000","last_undegraded":"2024-03-08T23:06:41.435262+0000","last_fullsized":"2024-03-08T23:06:41.435262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311614+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:06:41.311614+0000","last_peered":"2024-03-08T23:06:41.311614+0000","last_clean":"2024-03-08T23:06:41.311614+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:06:41.311614+0000","last_undegraded":"2024-03-08T23:06:41.311614+0000","last_fullsized":"2024-03-08T23:06:41.311614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308034+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:06:41.308034+0000","last_peered":"2024-03-08T23:06:41.308034+0000","last_clean":"2024-03-08T23:06:41.308034+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:06:41.308034+0000","last_undegraded":"2024-03-08T23:06:41.308034+0000","last_fullsized":"2024-03-08T23:06:41.308034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559587+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:06:41.559587+0000","last_peered":"2024-03-08T23:06:41.559587+0000","last_clean":"2024-03-08T23:06:41.559587+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:06:41.559587+0000","last_undegraded":"2024-03-08T23:06:41.559587+0000","last_fullsized":"2024-03-08T23:06:41.559587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559494+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:06:41.559494+0000","last_peered":"2024-03-08T23:06:41.559494+0000","last_clean":"2024-03-08T23:06:41.559494+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:06:41.559494+0000","last_undegraded":"2024-03-08T23:06:41.559494+0000","last_fullsized":"2024-03-08T23:06:41.559494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297182+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:06:41.297182+0000","last_peered":"2024-03-08T23:06:41.297182+0000","last_clean":"2024-03-08T23:06:41.297182+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:06:41.297182+0000","last_undegraded":"2024-03-08T23:06:41.297182+0000","last_fullsized":"2024-03-08T23:06:41.297182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295505+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:06:41.295505+0000","last_peered":"2024-03-08T23:06:41.295505+0000","last_clean":"2024-03-08T23:06:41.295505+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:06:41.295505+0000","last_undegraded":"2024-03-08T23:06:41.295505+0000","last_fullsized":"2024-03-08T23:06:41.295505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437926+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:06:41.437926+0000","last_peered":"2024-03-08T23:06:41.437926+0000","last_clean":"2024-03-08T23:06:41.437926+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:06:41.437926+0000","last_undegraded":"2024-03-08T23:06:41.437926+0000","last_fullsized":"2024-03-08T23:06:41.437926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298548+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:06:41.298548+0000","last_peered":"2024-03-08T23:06:41.298548+0000","last_clean":"2024-03-08T23:06:41.298548+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:06:41.298548+0000","last_undegraded":"2024-03-08T23:06:41.298548+0000","last_fullsized":"2024-03-08T23:06:41.298548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298912+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:06:41.298912+0000","last_peered":"2024-03-08T23:06:41.298912+0000","last_clean":"2024-03-08T23:06:41.298912+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:06:41.298912+0000","last_undegraded":"2024-03-08T23:06:41.298912+0000","last_fullsized":"2024-03-08T23:06:41.298912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.438077+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:06:41.438077+0000","last_peered":"2024-03-08T23:06:41.438077+0000","last_clean":"2024-03-08T23:06:41.438077+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:06:41.438077+0000","last_undegraded":"2024-03-08T23:06:41.438077+0000","last_fullsized":"2024-03-08T23:06:41.438077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294192+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:06:41.294192+0000","last_peered":"2024-03-08T23:06:41.294192+0000","last_clean":"2024-03-08T23:06:41.294192+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:06:41.294192+0000","last_undegraded":"2024-03-08T23:06:41.294192+0000","last_fullsized":"2024-03-08T23:06:41.294192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311090+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:06:41.311090+0000","last_peered":"2024-03-08T23:06:41.311090+0000","last_clean":"2024-03-08T23:06:41.311090+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:06:41.311090+0000","last_undegraded":"2024-03-08T23:06:41.311090+0000","last_fullsized":"2024-03-08T23:06:41.311090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294294+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:06:41.294294+0000","last_peered":"2024-03-08T23:06:41.294294+0000","last_clean":"2024-03-08T23:06:41.294294+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:06:41.294294+0000","last_undegraded":"2024-03-08T23:06:41.294294+0000","last_fullsized":"2024-03-08T23:06:41.294294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560057+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:06:41.560057+0000","last_peered":"2024-03-08T23:06:41.560057+0000","last_clean":"2024-03-08T23:06:41.560057+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:06:41.560057+0000","last_undegraded":"2024-03-08T23:06:41.560057+0000","last_fullsized":"2024-03-08T23:06:41.560057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311255+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:06:41.311255+0000","last_peered":"2024-03-08T23:06:41.311255+0000","last_clean":"2024-03-08T23:06:41.311255+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:06:41.311255+0000","last_undegraded":"2024-03-08T23:06:41.311255+0000","last_fullsized":"2024-03-08T23:06:41.311255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298754+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:06:41.298754+0000","last_peered":"2024-03-08T23:06:41.298754+0000","last_clean":"2024-03-08T23:06:41.298754+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:06:41.298754+0000","last_undegraded":"2024-03-08T23:06:41.298754+0000","last_fullsized":"2024-03-08T23:06:41.298754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297580+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:06:41.297580+0000","last_peered":"2024-03-08T23:06:41.297580+0000","last_clean":"2024-03-08T23:06:41.297580+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:06:41.297580+0000","last_undegraded":"2024-03-08T23:06:41.297580+0000","last_fullsized":"2024-03-08T23:06:41.297580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311187+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:06:41.311187+0000","last_peered":"2024-03-08T23:06:41.311187+0000","last_clean":"2024-03-08T23:06:41.311187+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:06:41.311187+0000","last_undegraded":"2024-03-08T23:06:41.311187+0000","last_fullsized":"2024-03-08T23:06:41.311187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.302293+0000","last_change":"2024-03-08T23:06:41.302293+0000","last_active":"2024-03-08T23:06:41.302293+0000","last_peered":"2024-03-08T23:06:41.302293+0000","last_clean":"2024-03-08T23:06:41.302293+0000","last_became_active":"2024-03-08T23:06:41.298282+0000","last_became_peered":"2024-03-08T23:06:41.298282+0000","last_unstale":"2024-03-08T23:06:41.302293+0000","last_undegraded":"2024-03-08T23:06:41.302293+0000","last_fullsized":"2024-03-08T23:06:41.302293+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:54:03.637871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293427+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:06:41.293427+0000","last_peered":"2024-03-08T23:06:41.293427+0000","last_clean":"2024-03-08T23:06:41.293427+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:06:41.293427+0000","last_undegraded":"2024-03-08T23:06:41.293427+0000","last_fullsized":"2024-03-08T23:06:41.293427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298352+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:06:41.298352+0000","last_peered":"2024-03-08T23:06:41.298352+0000","last_clean":"2024-03-08T23:06:41.298352+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:06:41.298352+0000","last_undegraded":"2024-03-08T23:06:41.298352+0000","last_fullsized":"2024-03-08T23:06:41.298352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:27.310084+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:07:27.310084+0000","last_peered":"2024-03-08T23:07:27.310084+0000","last_clean":"2024-03-08T23:07:27.310084+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:07:27.310084+0000","last_undegraded":"2024-03-08T23:07:27.310084+0000","last_fullsized":"2024-03-08T23:07:27.310084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298203+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:06:41.298203+0000","last_peered":"2024-03-08T23:06:41.298203+0000","last_clean":"2024-03-08T23:06:41.298203+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:06:41.298203+0000","last_undegraded":"2024-03-08T23:06:41.298203+0000","last_fullsized":"2024-03-08T23:06:41.298203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298560+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:06:41.298560+0000","last_peered":"2024-03-08T23:06:41.298560+0000","last_clean":"2024-03-08T23:06:41.298560+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:06:41.298560+0000","last_undegraded":"2024-03-08T23:06:41.298560+0000","last_fullsized":"2024-03-08T23:06:41.298560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293814+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:06:41.293814+0000","last_peered":"2024-03-08T23:06:41.293814+0000","last_clean":"2024-03-08T23:06:41.293814+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:06:41.293814+0000","last_undegraded":"2024-03-08T23:06:41.293814+0000","last_fullsized":"2024-03-08T23:06:41.293814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293533+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:06:41.293533+0000","last_peered":"2024-03-08T23:06:41.293533+0000","last_clean":"2024-03-08T23:06:41.293533+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:06:41.293533+0000","last_undegraded":"2024-03-08T23:06:41.293533+0000","last_fullsized":"2024-03-08T23:06:41.293533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295372+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:06:41.295372+0000","last_peered":"2024-03-08T23:06:41.295372+0000","last_clean":"2024-03-08T23:06:41.295372+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:06:41.295372+0000","last_undegraded":"2024-03-08T23:06:41.295372+0000","last_fullsized":"2024-03-08T23:06:41.295372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307289+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:06:41.307289+0000","last_peered":"2024-03-08T23:06:41.307289+0000","last_clean":"2024-03-08T23:06:41.307289+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:06:41.307289+0000","last_undegraded":"2024-03-08T23:06:41.307289+0000","last_fullsized":"2024-03-08T23:06:41.307289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293298+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:06:41.293298+0000","last_peered":"2024-03-08T23:06:41.293298+0000","last_clean":"2024-03-08T23:06:41.293298+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:06:41.293298+0000","last_undegraded":"2024-03-08T23:06:41.293298+0000","last_fullsized":"2024-03-08T23:06:41.293298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561119+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:06:41.561119+0000","last_peered":"2024-03-08T23:06:41.561119+0000","last_clean":"2024-03-08T23:06:41.561119+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:06:41.561119+0000","last_undegraded":"2024-03-08T23:06:41.561119+0000","last_fullsized":"2024-03-08T23:06:41.561119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293176+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:06:41.293176+0000","last_peered":"2024-03-08T23:06:41.293176+0000","last_clean":"2024-03-08T23:06:41.293176+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:06:41.293176+0000","last_undegraded":"2024-03-08T23:06:41.293176+0000","last_fullsized":"2024-03-08T23:06:41.293176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293191+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:06:41.293191+0000","last_peered":"2024-03-08T23:06:41.293191+0000","last_clean":"2024-03-08T23:06:41.293191+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:06:41.293191+0000","last_undegraded":"2024-03-08T23:06:41.293191+0000","last_fullsized":"2024-03-08T23:06:41.293191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":180,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.311328+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:07:28.311328+0000","last_peered":"2024-03-08T23:07:28.311328+0000","last_clean":"2024-03-08T23:07:28.311328+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:07:28.311328+0000","last_undegraded":"2024-03-08T23:07:28.311328+0000","last_fullsized":"2024-03-08T23:07:28.311328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297904+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:06:41.297904+0000","last_peered":"2024-03-08T23:06:41.297904+0000","last_clean":"2024-03-08T23:06:41.297904+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:06:41.297904+0000","last_undegraded":"2024-03-08T23:06:41.297904+0000","last_fullsized":"2024-03-08T23:06:41.297904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435215+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:06:41.435215+0000","last_peered":"2024-03-08T23:06:41.435215+0000","last_clean":"2024-03-08T23:06:41.435215+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:06:41.435215+0000","last_undegraded":"2024-03-08T23:06:41.435215+0000","last_fullsized":"2024-03-08T23:06:41.435215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560329+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:06:41.560329+0000","last_peered":"2024-03-08T23:06:41.560329+0000","last_clean":"2024-03-08T23:06:41.560329+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:06:41.560329+0000","last_undegraded":"2024-03-08T23:06:41.560329+0000","last_fullsized":"2024-03-08T23:06:41.560329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311591+0000","last_change":"2024-03-08T23:05:40.884963+0000","last_active":"2024-03-08T23:06:41.311591+0000","last_peered":"2024-03-08T23:06:41.311591+0000","last_clean":"2024-03-08T23:06:41.311591+0000","last_became_active":"2024-03-08T23:05:40.884402+0000","last_became_peered":"2024-03-08T23:05:40.884402+0000","last_unstale":"2024-03-08T23:06:41.311591+0000","last_undegraded":"2024-03-08T23:06:41.311591+0000","last_fullsized":"2024-03-08T23:06:41.311591+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:26.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293388+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:06:41.293388+0000","last_peered":"2024-03-08T23:06:41.293388+0000","last_clean":"2024-03-08T23:06:41.293388+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:06:41.293388+0000","last_undegraded":"2024-03-08T23:06:41.293388+0000","last_fullsized":"2024-03-08T23:06:41.293388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297713+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:06:41.297713+0000","last_peered":"2024-03-08T23:06:41.297713+0000","last_clean":"2024-03-08T23:06:41.297713+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:06:41.297713+0000","last_undegraded":"2024-03-08T23:06:41.297713+0000","last_fullsized":"2024-03-08T23:06:41.297713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293560+0000","last_change":"2024-03-08T23:05:41.148144+0000","last_active":"2024-03-08T23:06:41.293560+0000","last_peered":"2024-03-08T23:06:41.293560+0000","last_clean":"2024-03-08T23:06:41.293560+0000","last_became_active":"2024-03-08T23:05:41.147760+0000","last_became_peered":"2024-03-08T23:05:41.147760+0000","last_unstale":"2024-03-08T23:06:41.293560+0000","last_undegraded":"2024-03-08T23:06:41.293560+0000","last_fullsized":"2024-03-08T23:06:41.293560+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:00.345274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":369,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:29.312829+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:07:29.312829+0000","last_peered":"2024-03-08T23:07:29.312829+0000","last_clean":"2024-03-08T23:07:29.312829+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:07:29.312829+0000","last_undegraded":"2024-03-08T23:07:29.312829+0000","last_fullsized":"2024-03-08T23:07:29.312829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68004,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":300,"num_read_kb":218,"num_write":38,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298546+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:06:41.298546+0000","last_peered":"2024-03-08T23:06:41.298546+0000","last_clean":"2024-03-08T23:06:41.298546+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:06:41.298546+0000","last_undegraded":"2024-03-08T23:06:41.298546+0000","last_fullsized":"2024-03-08T23:06:41.298546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293944+0000","last_change":"2024-03-08T23:05:40.659738+0000","last_active":"2024-03-08T23:06:41.293944+0000","last_peered":"2024-03-08T23:06:41.293944+0000","last_clean":"2024-03-08T23:06:41.293944+0000","last_became_active":"2024-03-08T23:05:40.659391+0000","last_became_peered":"2024-03-08T23:05:40.659391+0000","last_unstale":"2024-03-08T23:06:41.293944+0000","last_undegraded":"2024-03-08T23:06:41.293944+0000","last_fullsized":"2024-03-08T23:06:41.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:42:03.247682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298136+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:06:41.298136+0000","last_peered":"2024-03-08T23:06:41.298136+0000","last_clean":"2024-03-08T23:06:41.298136+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:06:41.298136+0000","last_undegraded":"2024-03-08T23:06:41.298136+0000","last_fullsized":"2024-03-08T23:06:41.298136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307379+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:06:41.307379+0000","last_peered":"2024-03-08T23:06:41.307379+0000","last_clean":"2024-03-08T23:06:41.307379+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:06:41.307379+0000","last_undegraded":"2024-03-08T23:06:41.307379+0000","last_fullsized":"2024-03-08T23:06:41.307379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295980+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:06:41.295980+0000","last_peered":"2024-03-08T23:06:41.295980+0000","last_clean":"2024-03-08T23:06:41.295980+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:06:41.295980+0000","last_undegraded":"2024-03-08T23:06:41.295980+0000","last_fullsized":"2024-03-08T23:06:41.295980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293269+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:06:41.293269+0000","last_peered":"2024-03-08T23:06:41.293269+0000","last_clean":"2024-03-08T23:06:41.293269+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:06:41.293269+0000","last_undegraded":"2024-03-08T23:06:41.293269+0000","last_fullsized":"2024-03-08T23:06:41.293269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435039+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:06:41.435039+0000","last_peered":"2024-03-08T23:06:41.435039+0000","last_clean":"2024-03-08T23:06:41.435039+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:06:41.435039+0000","last_undegraded":"2024-03-08T23:06:41.435039+0000","last_fullsized":"2024-03-08T23:06:41.435039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437450+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:06:41.437450+0000","last_peered":"2024-03-08T23:06:41.437450+0000","last_clean":"2024-03-08T23:06:41.437450+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:06:41.437450+0000","last_undegraded":"2024-03-08T23:06:41.437450+0000","last_fullsized":"2024-03-08T23:06:41.437450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":185,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.418460+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:07:28.418460+0000","last_peered":"2024-03-08T23:07:28.418460+0000","last_clean":"2024-03-08T23:07:28.418460+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:07:28.418460+0000","last_undegraded":"2024-03-08T23:07:28.418460+0000","last_fullsized":"2024-03-08T23:07:28.418460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434833+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:06:41.434833+0000","last_peered":"2024-03-08T23:06:41.434833+0000","last_clean":"2024-03-08T23:06:41.434833+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:06:41.434833+0000","last_undegraded":"2024-03-08T23:06:41.434833+0000","last_fullsized":"2024-03-08T23:06:41.434833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308276+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:06:41.308276+0000","last_peered":"2024-03-08T23:06:41.308276+0000","last_clean":"2024-03-08T23:06:41.308276+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:06:41.308276+0000","last_undegraded":"2024-03-08T23:06:41.308276+0000","last_fullsized":"2024-03-08T23:06:41.308276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297956+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:06:41.297956+0000","last_peered":"2024-03-08T23:06:41.297956+0000","last_clean":"2024-03-08T23:06:41.297956+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:06:41.297956+0000","last_undegraded":"2024-03-08T23:06:41.297956+0000","last_fullsized":"2024-03-08T23:06:41.297956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292974+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:06:41.292974+0000","last_peered":"2024-03-08T23:06:41.292974+0000","last_clean":"2024-03-08T23:06:41.292974+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:06:41.292974+0000","last_undegraded":"2024-03-08T23:06:41.292974+0000","last_fullsized":"2024-03-08T23:06:41.292974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292888+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:06:41.292888+0000","last_peered":"2024-03-08T23:06:41.292888+0000","last_clean":"2024-03-08T23:06:41.292888+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:06:41.292888+0000","last_undegraded":"2024-03-08T23:06:41.292888+0000","last_fullsized":"2024-03-08T23:06:41.292888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298952+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:06:41.298952+0000","last_peered":"2024-03-08T23:06:41.298952+0000","last_clean":"2024-03-08T23:06:41.298952+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:06:41.298952+0000","last_undegraded":"2024-03-08T23:06:41.298952+0000","last_fullsized":"2024-03-08T23:06:41.298952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295447+0000","last_change":"2024-03-08T23:05:41.148194+0000","last_active":"2024-03-08T23:06:41.295447+0000","last_peered":"2024-03-08T23:06:41.295447+0000","last_clean":"2024-03-08T23:06:41.295447+0000","last_became_active":"2024-03-08T23:05:41.147832+0000","last_became_peered":"2024-03-08T23:05:41.147832+0000","last_unstale":"2024-03-08T23:06:41.295447+0000","last_undegraded":"2024-03-08T23:06:41.295447+0000","last_fullsized":"2024-03-08T23:06:41.295447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:38.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298118+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:06:41.298118+0000","last_peered":"2024-03-08T23:06:41.298118+0000","last_clean":"2024-03-08T23:06:41.298118+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:06:41.298118+0000","last_undegraded":"2024-03-08T23:06:41.298118+0000","last_fullsized":"2024-03-08T23:06:41.298118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437182+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:06:41.437182+0000","last_peered":"2024-03-08T23:06:41.437182+0000","last_clean":"2024-03-08T23:06:41.437182+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:06:41.437182+0000","last_undegraded":"2024-03-08T23:06:41.437182+0000","last_fullsized":"2024-03-08T23:06:41.437182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294258+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:06:41.294258+0000","last_peered":"2024-03-08T23:06:41.294258+0000","last_clean":"2024-03-08T23:06:41.294258+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:06:41.294258+0000","last_undegraded":"2024-03-08T23:06:41.294258+0000","last_fullsized":"2024-03-08T23:06:41.294258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307765+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:06:41.307765+0000","last_peered":"2024-03-08T23:06:41.307765+0000","last_clean":"2024-03-08T23:06:41.307765+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:06:41.307765+0000","last_undegraded":"2024-03-08T23:06:41.307765+0000","last_fullsized":"2024-03-08T23:06:41.307765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434782+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:06:41.434782+0000","last_peered":"2024-03-08T23:06:41.434782+0000","last_clean":"2024-03-08T23:06:41.434782+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:06:41.434782+0000","last_undegraded":"2024-03-08T23:06:41.434782+0000","last_fullsized":"2024-03-08T23:06:41.434782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293784+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:06:41.293784+0000","last_peered":"2024-03-08T23:06:41.293784+0000","last_clean":"2024-03-08T23:06:41.293784+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:06:41.293784+0000","last_undegraded":"2024-03-08T23:06:41.293784+0000","last_fullsized":"2024-03-08T23:06:41.293784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308278+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:06:41.308278+0000","last_peered":"2024-03-08T23:06:41.308278+0000","last_clean":"2024-03-08T23:06:41.308278+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:06:41.308278+0000","last_undegraded":"2024-03-08T23:06:41.308278+0000","last_fullsized":"2024-03-08T23:06:41.308278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293071+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:06:41.293071+0000","last_peered":"2024-03-08T23:06:41.293071+0000","last_clean":"2024-03-08T23:06:41.293071+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:06:41.293071+0000","last_undegraded":"2024-03-08T23:06:41.293071+0000","last_fullsized":"2024-03-08T23:06:41.293071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":179,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:26.414027+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:07:26.414027+0000","last_peered":"2024-03-08T23:07:26.414027+0000","last_clean":"2024-03-08T23:07:26.414027+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:07:26.414027+0000","last_undegraded":"2024-03-08T23:07:26.414027+0000","last_fullsized":"2024-03-08T23:07:26.414027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435137+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:06:41.435137+0000","last_peered":"2024-03-08T23:06:41.435137+0000","last_clean":"2024-03-08T23:06:41.435137+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:06:41.435137+0000","last_undegraded":"2024-03-08T23:06:41.435137+0000","last_fullsized":"2024-03-08T23:06:41.435137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293672+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:06:41.293672+0000","last_peered":"2024-03-08T23:06:41.293672+0000","last_clean":"2024-03-08T23:06:41.293672+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:06:41.293672+0000","last_undegraded":"2024-03-08T23:06:41.293672+0000","last_fullsized":"2024-03-08T23:06:41.293672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311338+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:06:41.311338+0000","last_peered":"2024-03-08T23:06:41.311338+0000","last_clean":"2024-03-08T23:06:41.311338+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:06:41.311338+0000","last_undegraded":"2024-03-08T23:06:41.311338+0000","last_fullsized":"2024-03-08T23:06:41.311338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434495+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:06:41.434495+0000","last_peered":"2024-03-08T23:06:41.434495+0000","last_clean":"2024-03-08T23:06:41.434495+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:06:41.434495+0000","last_undegraded":"2024-03-08T23:06:41.434495+0000","last_fullsized":"2024-03-08T23:06:41.434495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293727+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:06:41.293727+0000","last_peered":"2024-03-08T23:06:41.293727+0000","last_clean":"2024-03-08T23:06:41.293727+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:06:41.293727+0000","last_undegraded":"2024-03-08T23:06:41.293727+0000","last_fullsized":"2024-03-08T23:06:41.293727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307136+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:06:41.307136+0000","last_peered":"2024-03-08T23:06:41.307136+0000","last_clean":"2024-03-08T23:06:41.307136+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:06:41.307136+0000","last_undegraded":"2024-03-08T23:06:41.307136+0000","last_fullsized":"2024-03-08T23:06:41.307136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434597+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:06:41.434597+0000","last_peered":"2024-03-08T23:06:41.434597+0000","last_clean":"2024-03-08T23:06:41.434597+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:06:41.434597+0000","last_undegraded":"2024-03-08T23:06:41.434597+0000","last_fullsized":"2024-03-08T23:06:41.434597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295247+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:06:41.295247+0000","last_peered":"2024-03-08T23:06:41.295247+0000","last_clean":"2024-03-08T23:06:41.295247+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:06:41.295247+0000","last_undegraded":"2024-03-08T23:06:41.295247+0000","last_fullsized":"2024-03-08T23:06:41.295247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298642+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:06:41.298642+0000","last_peered":"2024-03-08T23:06:41.298642+0000","last_clean":"2024-03-08T23:06:41.298642+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:06:41.298642+0000","last_undegraded":"2024-03-08T23:06:41.298642+0000","last_fullsized":"2024-03-08T23:06:41.298642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307927+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:06:41.307927+0000","last_peered":"2024-03-08T23:06:41.307927+0000","last_clean":"2024-03-08T23:06:41.307927+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:06:41.307927+0000","last_undegraded":"2024-03-08T23:06:41.307927+0000","last_fullsized":"2024-03-08T23:06:41.307927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293158+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:06:41.293158+0000","last_peered":"2024-03-08T23:06:41.293158+0000","last_clean":"2024-03-08T23:06:41.293158+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:06:41.293158+0000","last_undegraded":"2024-03-08T23:06:41.293158+0000","last_fullsized":"2024-03-08T23:06:41.293158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'11","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:49.177474+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:06:49.177474+0000","last_peered":"2024-03-08T23:06:49.177474+0000","last_clean":"2024-03-08T23:06:49.177474+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:06:49.177474+0000","last_undegraded":"2024-03-08T23:06:49.177474+0000","last_fullsized":"2024-03-08T23:06:49.177474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294465+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:06:41.294465+0000","last_peered":"2024-03-08T23:06:41.294465+0000","last_clean":"2024-03-08T23:06:41.294465+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:06:41.294465+0000","last_undegraded":"2024-03-08T23:06:41.294465+0000","last_fullsized":"2024-03-08T23:06:41.294465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298315+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:06:41.298315+0000","last_peered":"2024-03-08T23:06:41.298315+0000","last_clean":"2024-03-08T23:06:41.298315+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:06:41.298315+0000","last_undegraded":"2024-03-08T23:06:41.298315+0000","last_fullsized":"2024-03-08T23:06:41.298315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311691+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:06:41.311691+0000","last_peered":"2024-03-08T23:06:41.311691+0000","last_clean":"2024-03-08T23:06:41.311691+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:06:41.311691+0000","last_undegraded":"2024-03-08T23:06:41.311691+0000","last_fullsized":"2024-03-08T23:06:41.311691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561285+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:06:41.561285+0000","last_peered":"2024-03-08T23:06:41.561285+0000","last_clean":"2024-03-08T23:06:41.561285+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:06:41.561285+0000","last_undegraded":"2024-03-08T23:06:41.561285+0000","last_fullsized":"2024-03-08T23:06:41.561285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311484+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:06:41.311484+0000","last_peered":"2024-03-08T23:06:41.311484+0000","last_clean":"2024-03-08T23:06:41.311484+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:06:41.311484+0000","last_undegraded":"2024-03-08T23:06:41.311484+0000","last_fullsized":"2024-03-08T23:06:41.311484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297243+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:06:41.297243+0000","last_peered":"2024-03-08T23:06:41.297243+0000","last_clean":"2024-03-08T23:06:41.297243+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:06:41.297243+0000","last_undegraded":"2024-03-08T23:06:41.297243+0000","last_fullsized":"2024-03-08T23:06:41.297243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297758+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:06:41.297758+0000","last_peered":"2024-03-08T23:06:41.297758+0000","last_clean":"2024-03-08T23:06:41.297758+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:06:41.297758+0000","last_undegraded":"2024-03-08T23:06:41.297758+0000","last_fullsized":"2024-03-08T23:06:41.297758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298880+0000","last_change":"2024-03-08T23:05:41.148196+0000","last_active":"2024-03-08T23:06:41.298880+0000","last_peered":"2024-03-08T23:06:41.298880+0000","last_clean":"2024-03-08T23:06:41.298880+0000","last_became_active":"2024-03-08T23:05:41.147730+0000","last_became_peered":"2024-03-08T23:05:41.147730+0000","last_unstale":"2024-03-08T23:06:41.298880+0000","last_undegraded":"2024-03-08T23:06:41.298880+0000","last_fullsized":"2024-03-08T23:06:41.298880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:21:05.284970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293328+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:06:41.293328+0000","last_peered":"2024-03-08T23:06:41.293328+0000","last_clean":"2024-03-08T23:06:41.293328+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:06:41.293328+0000","last_undegraded":"2024-03-08T23:06:41.293328+0000","last_fullsized":"2024-03-08T23:06:41.293328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295142+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:06:41.295142+0000","last_peered":"2024-03-08T23:06:41.295142+0000","last_clean":"2024-03-08T23:06:41.295142+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:06:41.295142+0000","last_undegraded":"2024-03-08T23:06:41.295142+0000","last_fullsized":"2024-03-08T23:06:41.295142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311375+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:06:41.311375+0000","last_peered":"2024-03-08T23:06:41.311375+0000","last_clean":"2024-03-08T23:06:41.311375+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:06:41.311375+0000","last_undegraded":"2024-03-08T23:06:41.311375+0000","last_fullsized":"2024-03-08T23:06:41.311375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.304979+0000","last_change":"2024-03-08T23:06:41.304979+0000","last_active":"2024-03-08T23:06:41.304979+0000","last_peered":"2024-03-08T23:06:41.304979+0000","last_clean":"2024-03-08T23:06:41.304979+0000","last_became_active":"2024-03-08T23:06:41.300018+0000","last_became_peered":"2024-03-08T23:06:41.300018+0000","last_unstale":"2024-03-08T23:06:41.304979+0000","last_undegraded":"2024-03-08T23:06:41.304979+0000","last_fullsized":"2024-03-08T23:06:41.304979+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:16:30.599602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293922+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:06:41.293922+0000","last_peered":"2024-03-08T23:06:41.293922+0000","last_clean":"2024-03-08T23:06:41.293922+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:06:41.293922+0000","last_undegraded":"2024-03-08T23:06:41.293922+0000","last_fullsized":"2024-03-08T23:06:41.293922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311564+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:06:41.311564+0000","last_peered":"2024-03-08T23:06:41.311564+0000","last_clean":"2024-03-08T23:06:41.311564+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:06:41.311564+0000","last_undegraded":"2024-03-08T23:06:41.311564+0000","last_fullsized":"2024-03-08T23:06:41.311564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297806+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:06:41.297806+0000","last_peered":"2024-03-08T23:06:41.297806+0000","last_clean":"2024-03-08T23:06:41.297806+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:06:41.297806+0000","last_undegraded":"2024-03-08T23:06:41.297806+0000","last_fullsized":"2024-03-08T23:06:41.297806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434958+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:06:41.434958+0000","last_peered":"2024-03-08T23:06:41.434958+0000","last_clean":"2024-03-08T23:06:41.434958+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:06:41.434958+0000","last_undegraded":"2024-03-08T23:06:41.434958+0000","last_fullsized":"2024-03-08T23:06:41.434958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297364+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:06:41.297364+0000","last_peered":"2024-03-08T23:06:41.297364+0000","last_clean":"2024-03-08T23:06:41.297364+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:06:41.297364+0000","last_undegraded":"2024-03-08T23:06:41.297364+0000","last_fullsized":"2024-03-08T23:06:41.297364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298721+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:06:41.298721+0000","last_peered":"2024-03-08T23:06:41.298721+0000","last_clean":"2024-03-08T23:06:41.298721+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:06:41.298721+0000","last_undegraded":"2024-03-08T23:06:41.298721+0000","last_fullsized":"2024-03-08T23:06:41.298721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435322+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:06:41.435322+0000","last_peered":"2024-03-08T23:06:41.435322+0000","last_clean":"2024-03-08T23:06:41.435322+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:06:41.435322+0000","last_undegraded":"2024-03-08T23:06:41.435322+0000","last_fullsized":"2024-03-08T23:06:41.435322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559863+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:06:41.559863+0000","last_peered":"2024-03-08T23:06:41.559863+0000","last_clean":"2024-03-08T23:06:41.559863+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:06:41.559863+0000","last_undegraded":"2024-03-08T23:06:41.559863+0000","last_fullsized":"2024-03-08T23:06:41.559863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293626+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:06:41.293626+0000","last_peered":"2024-03-08T23:06:41.293626+0000","last_clean":"2024-03-08T23:06:41.293626+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:06:41.293626+0000","last_undegraded":"2024-03-08T23:06:41.293626+0000","last_fullsized":"2024-03-08T23:06:41.293626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435008+0000","last_change":"2024-03-08T23:05:41.148243+0000","last_active":"2024-03-08T23:06:41.435008+0000","last_peered":"2024-03-08T23:06:41.435008+0000","last_clean":"2024-03-08T23:06:41.435008+0000","last_became_active":"2024-03-08T23:05:41.147924+0000","last_became_peered":"2024-03-08T23:05:41.147924+0000","last_unstale":"2024-03-08T23:06:41.435008+0000","last_undegraded":"2024-03-08T23:06:41.435008+0000","last_fullsized":"2024-03-08T23:06:41.435008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:47.129092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292704+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:06:41.292704+0000","last_peered":"2024-03-08T23:06:41.292704+0000","last_clean":"2024-03-08T23:06:41.292704+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:06:41.292704+0000","last_undegraded":"2024-03-08T23:06:41.292704+0000","last_fullsized":"2024-03-08T23:06:41.292704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308063+0000","last_change":"2024-03-08T23:05:40.661217+0000","last_active":"2024-03-08T23:06:41.308063+0000","last_peered":"2024-03-08T23:06:41.308063+0000","last_clean":"2024-03-08T23:06:41.308063+0000","last_became_active":"2024-03-08T23:05:40.657165+0000","last_became_peered":"2024-03-08T23:05:40.657165+0000","last_unstale":"2024-03-08T23:06:41.308063+0000","last_undegraded":"2024-03-08T23:06:41.308063+0000","last_fullsized":"2024-03-08T23:06:41.308063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:16.125758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435283+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:06:41.435283+0000","last_peered":"2024-03-08T23:06:41.435283+0000","last_clean":"2024-03-08T23:06:41.435283+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:06:41.435283+0000","last_undegraded":"2024-03-08T23:06:41.435283+0000","last_fullsized":"2024-03-08T23:06:41.435283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295157+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:06:41.295157+0000","last_peered":"2024-03-08T23:06:41.295157+0000","last_clean":"2024-03-08T23:06:41.295157+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:06:41.295157+0000","last_undegraded":"2024-03-08T23:06:41.295157+0000","last_fullsized":"2024-03-08T23:06:41.295157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.299154+0000","last_change":"2024-03-08T23:05:40.651674+0000","last_active":"2024-03-08T23:06:41.299154+0000","last_peered":"2024-03-08T23:06:41.299154+0000","last_clean":"2024-03-08T23:06:41.299154+0000","last_became_active":"2024-03-08T23:05:40.651393+0000","last_became_peered":"2024-03-08T23:05:40.651393+0000","last_unstale":"2024-03-08T23:06:41.299154+0000","last_undegraded":"2024-03-08T23:06:41.299154+0000","last_fullsized":"2024-03-08T23:06:41.299154+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:30:44.691787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298225+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:06:41.298225+0000","last_peered":"2024-03-08T23:06:41.298225+0000","last_clean":"2024-03-08T23:06:41.298225+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:06:41.298225+0000","last_undegraded":"2024-03-08T23:06:41.298225+0000","last_fullsized":"2024-03-08T23:06:41.298225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560171+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:06:41.560171+0000","last_peered":"2024-03-08T23:06:41.560171+0000","last_clean":"2024-03-08T23:06:41.560171+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:06:41.560171+0000","last_undegraded":"2024-03-08T23:06:41.560171+0000","last_fullsized":"2024-03-08T23:06:41.560171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308038+0000","last_change":"2024-03-08T23:05:40.993334+0000","last_active":"2024-03-08T23:06:41.308038+0000","last_peered":"2024-03-08T23:06:41.308038+0000","last_clean":"2024-03-08T23:06:41.308038+0000","last_became_active":"2024-03-08T23:05:40.991755+0000","last_became_peered":"2024-03-08T23:05:40.991755+0000","last_unstale":"2024-03-08T23:06:41.308038+0000","last_undegraded":"2024-03-08T23:06:41.308038+0000","last_fullsized":"2024-03-08T23:06:41.308038+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:31.085388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295481+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:06:41.295481+0000","last_peered":"2024-03-08T23:06:41.295481+0000","last_clean":"2024-03-08T23:06:41.295481+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:06:41.295481+0000","last_undegraded":"2024-03-08T23:06:41.295481+0000","last_fullsized":"2024-03-08T23:06:41.295481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298991+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:06:41.298991+0000","last_peered":"2024-03-08T23:06:41.298991+0000","last_clean":"2024-03-08T23:06:41.298991+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:06:41.298991+0000","last_undegraded":"2024-03-08T23:06:41.298991+0000","last_fullsized":"2024-03-08T23:06:41.298991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294678+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:06:41.294678+0000","last_peered":"2024-03-08T23:06:41.294678+0000","last_clean":"2024-03-08T23:06:41.294678+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:06:41.294678+0000","last_undegraded":"2024-03-08T23:06:41.294678+0000","last_fullsized":"2024-03-08T23:06:41.294678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292981+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:06:41.292981+0000","last_peered":"2024-03-08T23:06:41.292981+0000","last_clean":"2024-03-08T23:06:41.292981+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:06:41.292981+0000","last_undegraded":"2024-03-08T23:06:41.292981+0000","last_fullsized":"2024-03-08T23:06:41.292981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293449+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:06:41.293449+0000","last_peered":"2024-03-08T23:06:41.293449+0000","last_clean":"2024-03-08T23:06:41.293449+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:06:41.293449+0000","last_undegraded":"2024-03-08T23:06:41.293449+0000","last_fullsized":"2024-03-08T23:06:41.293449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297515+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:06:41.297515+0000","last_peered":"2024-03-08T23:06:41.297515+0000","last_clean":"2024-03-08T23:06:41.297515+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:06:41.297515+0000","last_undegraded":"2024-03-08T23:06:41.297515+0000","last_fullsized":"2024-03-08T23:06:41.297515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307876+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:06:41.307876+0000","last_peered":"2024-03-08T23:06:41.307876+0000","last_clean":"2024-03-08T23:06:41.307876+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:06:41.307876+0000","last_undegraded":"2024-03-08T23:06:41.307876+0000","last_fullsized":"2024-03-08T23:06:41.307876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.296016+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:06:41.296016+0000","last_peered":"2024-03-08T23:06:41.296016+0000","last_clean":"2024-03-08T23:06:41.296016+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:06:41.296016+0000","last_undegraded":"2024-03-08T23:06:41.296016+0000","last_fullsized":"2024-03-08T23:06:41.296016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72678,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":895,"num_read_kb":526,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":218034,"data_compressed":5463,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383728,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":519484,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645365,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1175288,"data_compressed":64572,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":523816,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":519640,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332166,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1116222,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":582810,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":516284,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":584692,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515090,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520078,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757330,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":1111278,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68004,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69886,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70114,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:07:32.603 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-08T23:07:32.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.858+0000 7fc042e00640 1 -- 172.21.15.46:0/977391788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 msgr2=0x7fc03c103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.858+0000 7fc042e00640 1 --2- 172.21.15.46:0/977391788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c103950 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc02c009ca0 tx=0x7fc02c02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 -- 172.21.15.46:0/977391788 shutdown_connections 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 --2- 172.21.15.46:0/977391788 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 --2- 172.21.15.46:0/977391788 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c104ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 --2- 172.21.15.46:0/977391788 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c103950 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 -- 172.21.15.46:0/977391788 >> 172.21.15.46:0/977391788 conn(0x7fc03c0feae0 msgr2=0x7fc03c100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 -- 172.21.15.46:0/977391788 shutdown_connections 2024-03-08T23:07:32.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.859+0000 7fc042e00640 1 -- 172.21.15.46:0/977391788 wait complete. 2024-03-08T23:07:32.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 Processor -- start 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 -- start start 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c1a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c10f1c0 con 0x7fc03c103530 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c10f040 con 0x7fc03c104720 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc042e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03c10f340 con 0x7fc03c1050e0 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc03be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc03b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.860+0000 7fc03be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58060/0 (socket says 172.21.15.46:58060) 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc040e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c1a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 -- 172.21.15.46:0/272546364 learned_addr learned my addr 172.21.15.46:0/272546364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:32.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 -- 172.21.15.46:0/272546364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 msgr2=0x7fc03c1a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c1a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 -- 172.21.15.46:0/272546364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 msgr2=0x7fc03c199d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c199d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 -- 172.21.15.46:0/272546364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc03c1a1420 con 0x7fc03c103530 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc040e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c1a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03b400640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.861+0000 7fc03be00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0324c0 tx=0x7fc02c032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.862+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c041020 con 0x7fc03c103530 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.862+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc02c034cd0 con 0x7fc03c103530 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.862+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03c1a16b0 con 0x7fc03c103530 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.862+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c039040 con 0x7fc03c103530 2024-03-08T23:07:32.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.862+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03c1a1ba0 con 0x7fc03c103530 2024-03-08T23:07:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.863+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03c10a920 con 0x7fc03c103530 2024-03-08T23:07:32.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.865+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc02c044020 con 0x7fc03c103530 2024-03-08T23:07:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.866+0000 7fc038c00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.866+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fc02c0f0ce0 con 0x7fc03c103530 2024-03-08T23:07:32.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.867+0000 7fc03b400640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:32.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.868+0000 7fc03b400640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc0240095f0 tx=0x7fc024005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:32.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:32.871+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc02c04b020 con 0x7fc03c103530 2024-03-08T23:07:33.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.073+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc03c061730 con 0x7fc0200a2800 2024-03-08T23:07:33.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.103+0000 7fc038c00640 1 -- 172.21.15.46:0/272546364 <== mgr.14152 v2:172.21.15.46:6800/757291496 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340800 (secure 0 0 0) 0x7fc03c061730 con 0x7fc0200a2800 2024-03-08T23:07:33.108 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:33.113 INFO:teuthology.orchestra.run.smithi046.stderr:dumped all 2024-03-08T23:07:33.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 msgr2=0x7fc0200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 0x7fc0200a4cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc0240095f0 tx=0x7fc024005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 msgr2=0x7fc03c199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc02c0324c0 tx=0x7fc02c032f40 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 shutdown_connections 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fc0200a2800 0x7fc0200a4cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc03c1050e0 0x7fc03c1a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc03c104720 0x7fc03c199d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 --2- 172.21.15.46:0/272546364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc03c103530 0x7fc03c199800 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 >> 172.21.15.46:0/272546364 conn(0x7fc03c0feae0 msgr2=0x7fc03c107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 shutdown_connections 2024-03-08T23:07:33.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.111+0000 7fc042e00640 1 -- 172.21.15.46:0/272546364 wait complete. 2024-03-08T23:07:33.136 INFO:teuthology.orchestra.run.smithi046.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-03-08T23:07:31.251232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":663280,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":1031,"num_read_kb":645,"num_write":359,"num_write_kb":3071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365816,"kb_used_data":12452,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576008,"statfs":{"total":1151940427776,"available":1151565832192,"internally_reserved":0,"allocated":12750848,"data_stored":8199578,"data_compressed":432612,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19143,"internal_metadata":361674041},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5683,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":47,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005317"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295735+0000","last_change":"2024-03-08T23:04:39.100029+0000","last_active":"2024-03-08T23:06:41.295735+0000","last_peered":"2024-03-08T23:06:41.295735+0000","last_clean":"2024-03-08T23:06:41.295735+0000","last_became_active":"2024-03-08T23:04:39.099521+0000","last_became_peered":"2024-03-08T23:04:39.099521+0000","last_unstale":"2024-03-08T23:06:41.295735+0000","last_undegraded":"2024-03-08T23:06:41.295735+0000","last_fullsized":"2024-03-08T23:06:41.295735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:44.965815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561180+0000","last_change":"2024-03-08T23:04:39.095201+0000","last_active":"2024-03-08T23:06:41.561180+0000","last_peered":"2024-03-08T23:06:41.561180+0000","last_clean":"2024-03-08T23:06:41.561180+0000","last_became_active":"2024-03-08T23:04:39.094660+0000","last_became_peered":"2024-03-08T23:04:39.094660+0000","last_unstale":"2024-03-08T23:06:41.561180+0000","last_undegraded":"2024-03-08T23:06:41.561180+0000","last_fullsized":"2024-03-08T23:06:41.561180+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:38:23.619671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560821+0000","last_change":"2024-03-08T23:04:39.096837+0000","last_active":"2024-03-08T23:06:41.560821+0000","last_peered":"2024-03-08T23:06:41.560821+0000","last_clean":"2024-03-08T23:06:41.560821+0000","last_became_active":"2024-03-08T23:04:39.093466+0000","last_became_peered":"2024-03-08T23:04:39.093466+0000","last_unstale":"2024-03-08T23:06:41.560821+0000","last_undegraded":"2024-03-08T23:06:41.560821+0000","last_fullsized":"2024-03-08T23:06:41.560821+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:53.972101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311586+0000","last_change":"2024-03-08T23:04:39.075754+0000","last_active":"2024-03-08T23:06:41.311586+0000","last_peered":"2024-03-08T23:06:41.311586+0000","last_clean":"2024-03-08T23:06:41.311586+0000","last_became_active":"2024-03-08T23:04:39.075354+0000","last_became_peered":"2024-03-08T23:04:39.075354+0000","last_unstale":"2024-03-08T23:06:41.311586+0000","last_undegraded":"2024-03-08T23:06:41.311586+0000","last_fullsized":"2024-03-08T23:06:41.311586+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:37:23.889255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297952+0000","last_change":"2024-03-08T23:04:39.097169+0000","last_active":"2024-03-08T23:06:41.297952+0000","last_peered":"2024-03-08T23:06:41.297952+0000","last_clean":"2024-03-08T23:06:41.297952+0000","last_became_active":"2024-03-08T23:04:39.096896+0000","last_became_peered":"2024-03-08T23:04:39.096896+0000","last_unstale":"2024-03-08T23:06:41.297952+0000","last_undegraded":"2024-03-08T23:06:41.297952+0000","last_fullsized":"2024-03-08T23:06:41.297952+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:25:03.127427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293959+0000","last_change":"2024-03-08T23:04:39.082492+0000","last_active":"2024-03-08T23:06:41.293959+0000","last_peered":"2024-03-08T23:06:41.293959+0000","last_clean":"2024-03-08T23:06:41.293959+0000","last_became_active":"2024-03-08T23:04:39.082137+0000","last_became_peered":"2024-03-08T23:04:39.082137+0000","last_unstale":"2024-03-08T23:06:41.293959+0000","last_undegraded":"2024-03-08T23:06:41.293959+0000","last_fullsized":"2024-03-08T23:06:41.293959+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:41.743717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434708+0000","last_change":"2024-03-08T23:04:39.078987+0000","last_active":"2024-03-08T23:06:41.434708+0000","last_peered":"2024-03-08T23:06:41.434708+0000","last_clean":"2024-03-08T23:06:41.434708+0000","last_became_active":"2024-03-08T23:04:39.078140+0000","last_became_peered":"2024-03-08T23:04:39.078140+0000","last_unstale":"2024-03-08T23:06:41.434708+0000","last_undegraded":"2024-03-08T23:06:41.434708+0000","last_fullsized":"2024-03-08T23:06:41.434708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:10:56.481156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293197+0000","last_change":"2024-03-08T23:04:39.072457+0000","last_active":"2024-03-08T23:06:41.293197+0000","last_peered":"2024-03-08T23:06:41.293197+0000","last_clean":"2024-03-08T23:06:41.293197+0000","last_became_active":"2024-03-08T23:04:39.071808+0000","last_became_peered":"2024-03-08T23:04:39.071808+0000","last_unstale":"2024-03-08T23:06:41.293197+0000","last_undegraded":"2024-03-08T23:06:41.293197+0000","last_fullsized":"2024-03-08T23:06:41.293197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:38.136845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298020+0000","last_change":"2024-03-08T23:04:39.077581+0000","last_active":"2024-03-08T23:06:41.298020+0000","last_peered":"2024-03-08T23:06:41.298020+0000","last_clean":"2024-03-08T23:06:41.298020+0000","last_became_active":"2024-03-08T23:04:39.077227+0000","last_became_peered":"2024-03-08T23:04:39.077227+0000","last_unstale":"2024-03-08T23:06:41.298020+0000","last_undegraded":"2024-03-08T23:06:41.298020+0000","last_fullsized":"2024-03-08T23:06:41.298020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:04.275058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294363+0000","last_change":"2024-03-08T23:04:39.073415+0000","last_active":"2024-03-08T23:06:41.294363+0000","last_peered":"2024-03-08T23:06:41.294363+0000","last_clean":"2024-03-08T23:06:41.294363+0000","last_became_active":"2024-03-08T23:04:39.073085+0000","last_became_peered":"2024-03-08T23:04:39.073085+0000","last_unstale":"2024-03-08T23:06:41.294363+0000","last_undegraded":"2024-03-08T23:06:41.294363+0000","last_fullsized":"2024-03-08T23:06:41.294363+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:21:26.647895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435262+0000","last_change":"2024-03-08T23:04:39.077527+0000","last_active":"2024-03-08T23:06:41.435262+0000","last_peered":"2024-03-08T23:06:41.435262+0000","last_clean":"2024-03-08T23:06:41.435262+0000","last_became_active":"2024-03-08T23:04:39.077177+0000","last_became_peered":"2024-03-08T23:04:39.077177+0000","last_unstale":"2024-03-08T23:06:41.435262+0000","last_undegraded":"2024-03-08T23:06:41.435262+0000","last_fullsized":"2024-03-08T23:06:41.435262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:05:37.705695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311614+0000","last_change":"2024-03-08T23:04:39.074190+0000","last_active":"2024-03-08T23:06:41.311614+0000","last_peered":"2024-03-08T23:06:41.311614+0000","last_clean":"2024-03-08T23:06:41.311614+0000","last_became_active":"2024-03-08T23:04:39.073827+0000","last_became_peered":"2024-03-08T23:04:39.073827+0000","last_unstale":"2024-03-08T23:06:41.311614+0000","last_undegraded":"2024-03-08T23:06:41.311614+0000","last_fullsized":"2024-03-08T23:06:41.311614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:44:27.804176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308034+0000","last_change":"2024-03-08T23:04:39.092328+0000","last_active":"2024-03-08T23:06:41.308034+0000","last_peered":"2024-03-08T23:06:41.308034+0000","last_clean":"2024-03-08T23:06:41.308034+0000","last_became_active":"2024-03-08T23:04:39.090825+0000","last_became_peered":"2024-03-08T23:04:39.090825+0000","last_unstale":"2024-03-08T23:06:41.308034+0000","last_undegraded":"2024-03-08T23:06:41.308034+0000","last_fullsized":"2024-03-08T23:06:41.308034+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:51:09.208683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559587+0000","last_change":"2024-03-08T23:04:39.103692+0000","last_active":"2024-03-08T23:06:41.559587+0000","last_peered":"2024-03-08T23:06:41.559587+0000","last_clean":"2024-03-08T23:06:41.559587+0000","last_became_active":"2024-03-08T23:04:39.100679+0000","last_became_peered":"2024-03-08T23:04:39.100679+0000","last_unstale":"2024-03-08T23:06:41.559587+0000","last_undegraded":"2024-03-08T23:06:41.559587+0000","last_fullsized":"2024-03-08T23:06:41.559587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:36.217874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559494+0000","last_change":"2024-03-08T23:04:39.090063+0000","last_active":"2024-03-08T23:06:41.559494+0000","last_peered":"2024-03-08T23:06:41.559494+0000","last_clean":"2024-03-08T23:06:41.559494+0000","last_became_active":"2024-03-08T23:04:39.089552+0000","last_became_peered":"2024-03-08T23:04:39.089552+0000","last_unstale":"2024-03-08T23:06:41.559494+0000","last_undegraded":"2024-03-08T23:06:41.559494+0000","last_fullsized":"2024-03-08T23:06:41.559494+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:37.106135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297182+0000","last_change":"2024-03-08T23:04:39.100636+0000","last_active":"2024-03-08T23:06:41.297182+0000","last_peered":"2024-03-08T23:06:41.297182+0000","last_clean":"2024-03-08T23:06:41.297182+0000","last_became_active":"2024-03-08T23:04:39.100237+0000","last_became_peered":"2024-03-08T23:04:39.100237+0000","last_unstale":"2024-03-08T23:06:41.297182+0000","last_undegraded":"2024-03-08T23:06:41.297182+0000","last_fullsized":"2024-03-08T23:06:41.297182+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:01.971739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295505+0000","last_change":"2024-03-08T23:04:39.099902+0000","last_active":"2024-03-08T23:06:41.295505+0000","last_peered":"2024-03-08T23:06:41.295505+0000","last_clean":"2024-03-08T23:06:41.295505+0000","last_became_active":"2024-03-08T23:04:39.099581+0000","last_became_peered":"2024-03-08T23:04:39.099581+0000","last_unstale":"2024-03-08T23:06:41.295505+0000","last_undegraded":"2024-03-08T23:06:41.295505+0000","last_fullsized":"2024-03-08T23:06:41.295505+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:33.037344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437926+0000","last_change":"2024-03-08T23:04:39.085058+0000","last_active":"2024-03-08T23:06:41.437926+0000","last_peered":"2024-03-08T23:06:41.437926+0000","last_clean":"2024-03-08T23:06:41.437926+0000","last_became_active":"2024-03-08T23:04:39.081778+0000","last_became_peered":"2024-03-08T23:04:39.081778+0000","last_unstale":"2024-03-08T23:06:41.437926+0000","last_undegraded":"2024-03-08T23:06:41.437926+0000","last_fullsized":"2024-03-08T23:06:41.437926+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:18:17.006885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298548+0000","last_change":"2024-03-08T23:04:39.088588+0000","last_active":"2024-03-08T23:06:41.298548+0000","last_peered":"2024-03-08T23:06:41.298548+0000","last_clean":"2024-03-08T23:06:41.298548+0000","last_became_active":"2024-03-08T23:04:39.088251+0000","last_became_peered":"2024-03-08T23:04:39.088251+0000","last_unstale":"2024-03-08T23:06:41.298548+0000","last_undegraded":"2024-03-08T23:06:41.298548+0000","last_fullsized":"2024-03-08T23:06:41.298548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:39:54.406441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298912+0000","last_change":"2024-03-08T23:04:43.439528+0000","last_active":"2024-03-08T23:06:41.298912+0000","last_peered":"2024-03-08T23:06:41.298912+0000","last_clean":"2024-03-08T23:06:41.298912+0000","last_became_active":"2024-03-08T23:04:43.439022+0000","last_became_peered":"2024-03-08T23:04:43.439022+0000","last_unstale":"2024-03-08T23:06:41.298912+0000","last_undegraded":"2024-03-08T23:06:41.298912+0000","last_fullsized":"2024-03-08T23:06:41.298912+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:57.488717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.438077+0000","last_change":"2024-03-08T23:04:38.333274+0000","last_active":"2024-03-08T23:06:41.438077+0000","last_peered":"2024-03-08T23:06:41.438077+0000","last_clean":"2024-03-08T23:06:41.438077+0000","last_became_active":"2024-03-08T23:04:38.332646+0000","last_became_peered":"2024-03-08T23:04:38.332646+0000","last_unstale":"2024-03-08T23:06:41.438077+0000","last_undegraded":"2024-03-08T23:06:41.438077+0000","last_fullsized":"2024-03-08T23:06:41.438077+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:59:47.745462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294192+0000","last_change":"2024-03-08T23:04:39.084761+0000","last_active":"2024-03-08T23:06:41.294192+0000","last_peered":"2024-03-08T23:06:41.294192+0000","last_clean":"2024-03-08T23:06:41.294192+0000","last_became_active":"2024-03-08T23:04:39.084335+0000","last_became_peered":"2024-03-08T23:04:39.084335+0000","last_unstale":"2024-03-08T23:06:41.294192+0000","last_undegraded":"2024-03-08T23:06:41.294192+0000","last_fullsized":"2024-03-08T23:06:41.294192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:23:05.376313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311090+0000","last_change":"2024-03-08T23:04:43.435132+0000","last_active":"2024-03-08T23:06:41.311090+0000","last_peered":"2024-03-08T23:06:41.311090+0000","last_clean":"2024-03-08T23:06:41.311090+0000","last_became_active":"2024-03-08T23:04:43.434734+0000","last_became_peered":"2024-03-08T23:04:43.434734+0000","last_unstale":"2024-03-08T23:06:41.311090+0000","last_undegraded":"2024-03-08T23:06:41.311090+0000","last_fullsized":"2024-03-08T23:06:41.311090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:16.371203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294294+0000","last_change":"2024-03-08T23:04:38.346409+0000","last_active":"2024-03-08T23:06:41.294294+0000","last_peered":"2024-03-08T23:06:41.294294+0000","last_clean":"2024-03-08T23:06:41.294294+0000","last_became_active":"2024-03-08T23:04:38.345086+0000","last_became_peered":"2024-03-08T23:04:38.345086+0000","last_unstale":"2024-03-08T23:06:41.294294+0000","last_undegraded":"2024-03-08T23:06:41.294294+0000","last_fullsized":"2024-03-08T23:06:41.294294+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:04:16.883559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560057+0000","last_change":"2024-03-08T23:04:39.096002+0000","last_active":"2024-03-08T23:06:41.560057+0000","last_peered":"2024-03-08T23:06:41.560057+0000","last_clean":"2024-03-08T23:06:41.560057+0000","last_became_active":"2024-03-08T23:04:39.095670+0000","last_became_peered":"2024-03-08T23:04:39.095670+0000","last_unstale":"2024-03-08T23:06:41.560057+0000","last_undegraded":"2024-03-08T23:06:41.560057+0000","last_fullsized":"2024-03-08T23:06:41.560057+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:55:13.723306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311255+0000","last_change":"2024-03-08T23:04:43.447150+0000","last_active":"2024-03-08T23:06:41.311255+0000","last_peered":"2024-03-08T23:06:41.311255+0000","last_clean":"2024-03-08T23:06:41.311255+0000","last_became_active":"2024-03-08T23:04:43.446030+0000","last_became_peered":"2024-03-08T23:04:43.446030+0000","last_unstale":"2024-03-08T23:06:41.311255+0000","last_undegraded":"2024-03-08T23:06:41.311255+0000","last_fullsized":"2024-03-08T23:06:41.311255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:24.830222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298754+0000","last_change":"2024-03-08T23:04:38.328475+0000","last_active":"2024-03-08T23:06:41.298754+0000","last_peered":"2024-03-08T23:06:41.298754+0000","last_clean":"2024-03-08T23:06:41.298754+0000","last_became_active":"2024-03-08T23:04:38.328147+0000","last_became_peered":"2024-03-08T23:04:38.328147+0000","last_unstale":"2024-03-08T23:06:41.298754+0000","last_undegraded":"2024-03-08T23:06:41.298754+0000","last_fullsized":"2024-03-08T23:06:41.298754+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:20.443932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297580+0000","last_change":"2024-03-08T23:04:39.090780+0000","last_active":"2024-03-08T23:06:41.297580+0000","last_peered":"2024-03-08T23:06:41.297580+0000","last_clean":"2024-03-08T23:06:41.297580+0000","last_became_active":"2024-03-08T23:04:39.089766+0000","last_became_peered":"2024-03-08T23:04:39.089766+0000","last_unstale":"2024-03-08T23:06:41.297580+0000","last_undegraded":"2024-03-08T23:06:41.297580+0000","last_fullsized":"2024-03-08T23:06:41.297580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:00:42.146843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311187+0000","last_change":"2024-03-08T23:04:43.446038+0000","last_active":"2024-03-08T23:06:41.311187+0000","last_peered":"2024-03-08T23:06:41.311187+0000","last_clean":"2024-03-08T23:06:41.311187+0000","last_became_active":"2024-03-08T23:04:43.445596+0000","last_became_peered":"2024-03-08T23:04:43.445596+0000","last_unstale":"2024-03-08T23:06:41.311187+0000","last_undegraded":"2024-03-08T23:06:41.311187+0000","last_fullsized":"2024-03-08T23:06:41.311187+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:40:49.627130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.302293+0000","last_change":"2024-03-08T23:06:41.302293+0000","last_active":"2024-03-08T23:06:41.302293+0000","last_peered":"2024-03-08T23:06:41.302293+0000","last_clean":"2024-03-08T23:06:41.302293+0000","last_became_active":"2024-03-08T23:06:41.298282+0000","last_became_peered":"2024-03-08T23:06:41.298282+0000","last_unstale":"2024-03-08T23:06:41.302293+0000","last_undegraded":"2024-03-08T23:06:41.302293+0000","last_fullsized":"2024-03-08T23:06:41.302293+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:54:03.637871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,5],"acting":[11,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293427+0000","last_change":"2024-03-08T23:04:39.077503+0000","last_active":"2024-03-08T23:06:41.293427+0000","last_peered":"2024-03-08T23:06:41.293427+0000","last_clean":"2024-03-08T23:06:41.293427+0000","last_became_active":"2024-03-08T23:04:39.077003+0000","last_became_peered":"2024-03-08T23:04:39.077003+0000","last_unstale":"2024-03-08T23:06:41.293427+0000","last_undegraded":"2024-03-08T23:06:41.293427+0000","last_fullsized":"2024-03-08T23:06:41.293427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:25:50.494809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298352+0000","last_change":"2024-03-08T23:04:43.427524+0000","last_active":"2024-03-08T23:06:41.298352+0000","last_peered":"2024-03-08T23:06:41.298352+0000","last_clean":"2024-03-08T23:06:41.298352+0000","last_became_active":"2024-03-08T23:04:43.427051+0000","last_became_peered":"2024-03-08T23:04:43.427051+0000","last_unstale":"2024-03-08T23:06:41.298352+0000","last_undegraded":"2024-03-08T23:06:41.298352+0000","last_fullsized":"2024-03-08T23:06:41.298352+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:10:56.999685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":186,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:27.310084+0000","last_change":"2024-03-08T23:04:38.325253+0000","last_active":"2024-03-08T23:07:27.310084+0000","last_peered":"2024-03-08T23:07:27.310084+0000","last_clean":"2024-03-08T23:07:27.310084+0000","last_became_active":"2024-03-08T23:04:38.324886+0000","last_became_peered":"2024-03-08T23:04:38.324886+0000","last_unstale":"2024-03-08T23:07:27.310084+0000","last_undegraded":"2024-03-08T23:07:27.310084+0000","last_fullsized":"2024-03-08T23:07:27.310084+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:33:08.547620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":147,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298203+0000","last_change":"2024-03-08T23:04:39.076824+0000","last_active":"2024-03-08T23:06:41.298203+0000","last_peered":"2024-03-08T23:06:41.298203+0000","last_clean":"2024-03-08T23:06:41.298203+0000","last_became_active":"2024-03-08T23:04:39.076487+0000","last_became_peered":"2024-03-08T23:04:39.076487+0000","last_unstale":"2024-03-08T23:06:41.298203+0000","last_undegraded":"2024-03-08T23:06:41.298203+0000","last_fullsized":"2024-03-08T23:06:41.298203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:06.773886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298560+0000","last_change":"2024-03-08T23:04:43.445702+0000","last_active":"2024-03-08T23:06:41.298560+0000","last_peered":"2024-03-08T23:06:41.298560+0000","last_clean":"2024-03-08T23:06:41.298560+0000","last_became_active":"2024-03-08T23:04:43.445248+0000","last_became_peered":"2024-03-08T23:04:43.445248+0000","last_unstale":"2024-03-08T23:06:41.298560+0000","last_undegraded":"2024-03-08T23:06:41.298560+0000","last_fullsized":"2024-03-08T23:06:41.298560+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:26:12.376076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293814+0000","last_change":"2024-03-08T23:04:38.345470+0000","last_active":"2024-03-08T23:06:41.293814+0000","last_peered":"2024-03-08T23:06:41.293814+0000","last_clean":"2024-03-08T23:06:41.293814+0000","last_became_active":"2024-03-08T23:04:38.345078+0000","last_became_peered":"2024-03-08T23:04:38.345078+0000","last_unstale":"2024-03-08T23:06:41.293814+0000","last_undegraded":"2024-03-08T23:06:41.293814+0000","last_fullsized":"2024-03-08T23:06:41.293814+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:22:23.472918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293533+0000","last_change":"2024-03-08T23:04:39.090186+0000","last_active":"2024-03-08T23:06:41.293533+0000","last_peered":"2024-03-08T23:06:41.293533+0000","last_clean":"2024-03-08T23:06:41.293533+0000","last_became_active":"2024-03-08T23:04:39.089829+0000","last_became_peered":"2024-03-08T23:04:39.089829+0000","last_unstale":"2024-03-08T23:06:41.293533+0000","last_undegraded":"2024-03-08T23:06:41.293533+0000","last_fullsized":"2024-03-08T23:06:41.293533+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:09:38.370752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295372+0000","last_change":"2024-03-08T23:04:43.437300+0000","last_active":"2024-03-08T23:06:41.295372+0000","last_peered":"2024-03-08T23:06:41.295372+0000","last_clean":"2024-03-08T23:06:41.295372+0000","last_became_active":"2024-03-08T23:04:43.436876+0000","last_became_peered":"2024-03-08T23:04:43.436876+0000","last_unstale":"2024-03-08T23:06:41.295372+0000","last_undegraded":"2024-03-08T23:06:41.295372+0000","last_fullsized":"2024-03-08T23:06:41.295372+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:27:18.411287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307289+0000","last_change":"2024-03-08T23:04:38.340026+0000","last_active":"2024-03-08T23:06:41.307289+0000","last_peered":"2024-03-08T23:06:41.307289+0000","last_clean":"2024-03-08T23:06:41.307289+0000","last_became_active":"2024-03-08T23:04:38.337127+0000","last_became_peered":"2024-03-08T23:04:38.337127+0000","last_unstale":"2024-03-08T23:06:41.307289+0000","last_undegraded":"2024-03-08T23:06:41.307289+0000","last_fullsized":"2024-03-08T23:06:41.307289+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:13:17.026407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293298+0000","last_change":"2024-03-08T23:04:39.091243+0000","last_active":"2024-03-08T23:06:41.293298+0000","last_peered":"2024-03-08T23:06:41.293298+0000","last_clean":"2024-03-08T23:06:41.293298+0000","last_became_active":"2024-03-08T23:04:39.090899+0000","last_became_peered":"2024-03-08T23:04:39.090899+0000","last_unstale":"2024-03-08T23:06:41.293298+0000","last_undegraded":"2024-03-08T23:06:41.293298+0000","last_fullsized":"2024-03-08T23:06:41.293298+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:48.533546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561119+0000","last_change":"2024-03-08T23:04:43.449103+0000","last_active":"2024-03-08T23:06:41.561119+0000","last_peered":"2024-03-08T23:06:41.561119+0000","last_clean":"2024-03-08T23:06:41.561119+0000","last_became_active":"2024-03-08T23:04:43.447578+0000","last_became_peered":"2024-03-08T23:04:43.447578+0000","last_unstale":"2024-03-08T23:06:41.561119+0000","last_undegraded":"2024-03-08T23:06:41.561119+0000","last_fullsized":"2024-03-08T23:06:41.561119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:59.737013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293176+0000","last_change":"2024-03-08T23:04:38.325320+0000","last_active":"2024-03-08T23:06:41.293176+0000","last_peered":"2024-03-08T23:06:41.293176+0000","last_clean":"2024-03-08T23:06:41.293176+0000","last_became_active":"2024-03-08T23:04:38.324944+0000","last_became_peered":"2024-03-08T23:04:38.324944+0000","last_unstale":"2024-03-08T23:06:41.293176+0000","last_undegraded":"2024-03-08T23:06:41.293176+0000","last_fullsized":"2024-03-08T23:06:41.293176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:10:55.891029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:39.101793+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:39.101396+0000","last_became_peered":"2024-03-08T23:04:39.101396+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:35.751075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293191+0000","last_change":"2024-03-08T23:04:43.428063+0000","last_active":"2024-03-08T23:06:41.293191+0000","last_peered":"2024-03-08T23:06:41.293191+0000","last_clean":"2024-03-08T23:06:41.293191+0000","last_became_active":"2024-03-08T23:04:43.425189+0000","last_became_peered":"2024-03-08T23:04:43.425189+0000","last_unstale":"2024-03-08T23:06:41.293191+0000","last_undegraded":"2024-03-08T23:06:41.293191+0000","last_fullsized":"2024-03-08T23:06:41.293191+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:28.229853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":180,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.311328+0000","last_change":"2024-03-08T23:04:38.331580+0000","last_active":"2024-03-08T23:07:28.311328+0000","last_peered":"2024-03-08T23:07:28.311328+0000","last_clean":"2024-03-08T23:07:28.311328+0000","last_became_active":"2024-03-08T23:04:38.331076+0000","last_became_peered":"2024-03-08T23:04:38.331076+0000","last_unstale":"2024-03-08T23:07:28.311328+0000","last_undegraded":"2024-03-08T23:07:28.311328+0000","last_fullsized":"2024-03-08T23:07:28.311328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:33:25.812335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297904+0000","last_change":"2024-03-08T23:04:39.078004+0000","last_active":"2024-03-08T23:06:41.297904+0000","last_peered":"2024-03-08T23:06:41.297904+0000","last_clean":"2024-03-08T23:06:41.297904+0000","last_became_active":"2024-03-08T23:04:39.077640+0000","last_became_peered":"2024-03-08T23:04:39.077640+0000","last_unstale":"2024-03-08T23:06:41.297904+0000","last_undegraded":"2024-03-08T23:06:41.297904+0000","last_fullsized":"2024-03-08T23:06:41.297904+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:36:32.353462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.436742+0000","last_change":"2024-03-08T23:04:43.442756+0000","last_active":"2024-03-08T23:06:41.436742+0000","last_peered":"2024-03-08T23:06:41.436742+0000","last_clean":"2024-03-08T23:06:41.436742+0000","last_became_active":"2024-03-08T23:04:43.442232+0000","last_became_peered":"2024-03-08T23:04:43.442232+0000","last_unstale":"2024-03-08T23:06:41.436742+0000","last_undegraded":"2024-03-08T23:06:41.436742+0000","last_fullsized":"2024-03-08T23:06:41.436742+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:26:48.169711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435215+0000","last_change":"2024-03-08T23:04:38.327319+0000","last_active":"2024-03-08T23:06:41.435215+0000","last_peered":"2024-03-08T23:06:41.435215+0000","last_clean":"2024-03-08T23:06:41.435215+0000","last_became_active":"2024-03-08T23:04:38.326961+0000","last_became_peered":"2024-03-08T23:04:38.326961+0000","last_unstale":"2024-03-08T23:06:41.435215+0000","last_undegraded":"2024-03-08T23:06:41.435215+0000","last_fullsized":"2024-03-08T23:06:41.435215+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:14:11.719069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560329+0000","last_change":"2024-03-08T23:04:39.106417+0000","last_active":"2024-03-08T23:06:41.560329+0000","last_peered":"2024-03-08T23:06:41.560329+0000","last_clean":"2024-03-08T23:06:41.560329+0000","last_became_active":"2024-03-08T23:04:39.102226+0000","last_became_peered":"2024-03-08T23:04:39.102226+0000","last_unstale":"2024-03-08T23:06:41.560329+0000","last_undegraded":"2024-03-08T23:06:41.560329+0000","last_fullsized":"2024-03-08T23:06:41.560329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:28:39.765217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311591+0000","last_change":"2024-03-08T23:05:40.884963+0000","last_active":"2024-03-08T23:06:41.311591+0000","last_peered":"2024-03-08T23:06:41.311591+0000","last_clean":"2024-03-08T23:06:41.311591+0000","last_became_active":"2024-03-08T23:05:40.884402+0000","last_became_peered":"2024-03-08T23:05:40.884402+0000","last_unstale":"2024-03-08T23:06:41.311591+0000","last_undegraded":"2024-03-08T23:06:41.311591+0000","last_fullsized":"2024-03-08T23:06:41.311591+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:26.279789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293388+0000","last_change":"2024-03-08T23:04:38.339881+0000","last_active":"2024-03-08T23:06:41.293388+0000","last_peered":"2024-03-08T23:06:41.293388+0000","last_clean":"2024-03-08T23:06:41.293388+0000","last_became_active":"2024-03-08T23:04:38.339469+0000","last_became_peered":"2024-03-08T23:04:38.339469+0000","last_unstale":"2024-03-08T23:06:41.293388+0000","last_undegraded":"2024-03-08T23:06:41.293388+0000","last_fullsized":"2024-03-08T23:06:41.293388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:13.380409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297713+0000","last_change":"2024-03-08T23:04:39.075029+0000","last_active":"2024-03-08T23:06:41.297713+0000","last_peered":"2024-03-08T23:06:41.297713+0000","last_clean":"2024-03-08T23:06:41.297713+0000","last_became_active":"2024-03-08T23:04:39.074677+0000","last_became_peered":"2024-03-08T23:04:39.074677+0000","last_unstale":"2024-03-08T23:06:41.297713+0000","last_undegraded":"2024-03-08T23:06:41.297713+0000","last_fullsized":"2024-03-08T23:06:41.297713+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:07.528875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293560+0000","last_change":"2024-03-08T23:05:41.148144+0000","last_active":"2024-03-08T23:06:41.293560+0000","last_peered":"2024-03-08T23:06:41.293560+0000","last_clean":"2024-03-08T23:06:41.293560+0000","last_became_active":"2024-03-08T23:05:41.147760+0000","last_became_peered":"2024-03-08T23:05:41.147760+0000","last_unstale":"2024-03-08T23:06:41.293560+0000","last_undegraded":"2024-03-08T23:06:41.293560+0000","last_fullsized":"2024-03-08T23:06:41.293560+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:57:00.345274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":369,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:29.312829+0000","last_change":"2024-03-08T23:04:38.327010+0000","last_active":"2024-03-08T23:07:29.312829+0000","last_peered":"2024-03-08T23:07:29.312829+0000","last_clean":"2024-03-08T23:07:29.312829+0000","last_became_active":"2024-03-08T23:04:38.326642+0000","last_became_peered":"2024-03-08T23:04:38.326642+0000","last_unstale":"2024-03-08T23:07:29.312829+0000","last_undegraded":"2024-03-08T23:07:29.312829+0000","last_fullsized":"2024-03-08T23:07:29.312829+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:38.815918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68004,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":300,"num_read_kb":218,"num_write":38,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298546+0000","last_change":"2024-03-08T23:04:39.071354+0000","last_active":"2024-03-08T23:06:41.298546+0000","last_peered":"2024-03-08T23:06:41.298546+0000","last_clean":"2024-03-08T23:06:41.298546+0000","last_became_active":"2024-03-08T23:04:39.070964+0000","last_became_peered":"2024-03-08T23:04:39.070964+0000","last_unstale":"2024-03-08T23:06:41.298546+0000","last_undegraded":"2024-03-08T23:06:41.298546+0000","last_fullsized":"2024-03-08T23:06:41.298546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:07:47.430389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293944+0000","last_change":"2024-03-08T23:05:40.659738+0000","last_active":"2024-03-08T23:06:41.293944+0000","last_peered":"2024-03-08T23:06:41.293944+0000","last_clean":"2024-03-08T23:06:41.293944+0000","last_became_active":"2024-03-08T23:05:40.659391+0000","last_became_peered":"2024-03-08T23:05:40.659391+0000","last_unstale":"2024-03-08T23:06:41.293944+0000","last_undegraded":"2024-03-08T23:06:41.293944+0000","last_fullsized":"2024-03-08T23:06:41.293944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:42:03.247682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,3,6],"acting":[10,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298136+0000","last_change":"2024-03-08T23:04:38.330500+0000","last_active":"2024-03-08T23:06:41.298136+0000","last_peered":"2024-03-08T23:06:41.298136+0000","last_clean":"2024-03-08T23:06:41.298136+0000","last_became_active":"2024-03-08T23:04:38.330156+0000","last_became_peered":"2024-03-08T23:04:38.330156+0000","last_unstale":"2024-03-08T23:06:41.298136+0000","last_undegraded":"2024-03-08T23:06:41.298136+0000","last_fullsized":"2024-03-08T23:06:41.298136+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:48.204051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307379+0000","last_change":"2024-03-08T23:04:39.103364+0000","last_active":"2024-03-08T23:06:41.307379+0000","last_peered":"2024-03-08T23:06:41.307379+0000","last_clean":"2024-03-08T23:06:41.307379+0000","last_became_active":"2024-03-08T23:04:39.100096+0000","last_became_peered":"2024-03-08T23:04:39.100096+0000","last_unstale":"2024-03-08T23:06:41.307379+0000","last_undegraded":"2024-03-08T23:06:41.307379+0000","last_fullsized":"2024-03-08T23:06:41.307379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:55:08.038590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295980+0000","last_change":"2024-03-08T23:04:43.438538+0000","last_active":"2024-03-08T23:06:41.295980+0000","last_peered":"2024-03-08T23:06:41.295980+0000","last_clean":"2024-03-08T23:06:41.295980+0000","last_became_active":"2024-03-08T23:04:43.438051+0000","last_became_peered":"2024-03-08T23:04:43.438051+0000","last_unstale":"2024-03-08T23:06:41.295980+0000","last_undegraded":"2024-03-08T23:06:41.295980+0000","last_fullsized":"2024-03-08T23:06:41.295980+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:39:02.742455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293269+0000","last_change":"2024-03-08T23:04:38.336859+0000","last_active":"2024-03-08T23:06:41.293269+0000","last_peered":"2024-03-08T23:06:41.293269+0000","last_clean":"2024-03-08T23:06:41.293269+0000","last_became_active":"2024-03-08T23:04:38.336459+0000","last_became_peered":"2024-03-08T23:04:38.336459+0000","last_unstale":"2024-03-08T23:06:41.293269+0000","last_undegraded":"2024-03-08T23:06:41.293269+0000","last_fullsized":"2024-03-08T23:06:41.293269+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:48:29.762299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435039+0000","last_change":"2024-03-08T23:04:39.092956+0000","last_active":"2024-03-08T23:06:41.435039+0000","last_peered":"2024-03-08T23:06:41.435039+0000","last_clean":"2024-03-08T23:06:41.435039+0000","last_became_active":"2024-03-08T23:04:39.092566+0000","last_became_peered":"2024-03-08T23:04:39.092566+0000","last_unstale":"2024-03-08T23:06:41.435039+0000","last_undegraded":"2024-03-08T23:06:41.435039+0000","last_fullsized":"2024-03-08T23:06:41.435039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:59:41.972615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437450+0000","last_change":"2024-03-08T23:04:43.448551+0000","last_active":"2024-03-08T23:06:41.437450+0000","last_peered":"2024-03-08T23:06:41.437450+0000","last_clean":"2024-03-08T23:06:41.437450+0000","last_became_active":"2024-03-08T23:04:43.445442+0000","last_became_peered":"2024-03-08T23:04:43.445442+0000","last_unstale":"2024-03-08T23:06:41.437450+0000","last_undegraded":"2024-03-08T23:06:41.437450+0000","last_fullsized":"2024-03-08T23:06:41.437450+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:36:17.433216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"91'5","reported_seq":185,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:28.418460+0000","last_change":"2024-03-08T23:04:38.324125+0000","last_active":"2024-03-08T23:07:28.418460+0000","last_peered":"2024-03-08T23:07:28.418460+0000","last_clean":"2024-03-08T23:07:28.418460+0000","last_became_active":"2024-03-08T23:04:38.323553+0000","last_became_peered":"2024-03-08T23:04:38.323553+0000","last_unstale":"2024-03-08T23:07:28.418460+0000","last_undegraded":"2024-03-08T23:07:28.418460+0000","last_fullsized":"2024-03-08T23:07:28.418460+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:27:18.086483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434833+0000","last_change":"2024-03-08T23:04:39.075145+0000","last_active":"2024-03-08T23:06:41.434833+0000","last_peered":"2024-03-08T23:06:41.434833+0000","last_clean":"2024-03-08T23:06:41.434833+0000","last_became_active":"2024-03-08T23:04:39.071630+0000","last_became_peered":"2024-03-08T23:04:39.071630+0000","last_unstale":"2024-03-08T23:06:41.434833+0000","last_undegraded":"2024-03-08T23:06:41.434833+0000","last_fullsized":"2024-03-08T23:06:41.434833+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:43:17.545504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308276+0000","last_change":"2024-03-08T23:04:43.447165+0000","last_active":"2024-03-08T23:06:41.308276+0000","last_peered":"2024-03-08T23:06:41.308276+0000","last_clean":"2024-03-08T23:06:41.308276+0000","last_became_active":"2024-03-08T23:04:43.445985+0000","last_became_peered":"2024-03-08T23:04:43.445985+0000","last_unstale":"2024-03-08T23:06:41.308276+0000","last_undegraded":"2024-03-08T23:06:41.308276+0000","last_fullsized":"2024-03-08T23:06:41.308276+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:37:05.098022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297956+0000","last_change":"2024-03-08T23:04:38.339890+0000","last_active":"2024-03-08T23:06:41.297956+0000","last_peered":"2024-03-08T23:06:41.297956+0000","last_clean":"2024-03-08T23:06:41.297956+0000","last_became_active":"2024-03-08T23:04:38.339514+0000","last_became_peered":"2024-03-08T23:04:38.339514+0000","last_unstale":"2024-03-08T23:06:41.297956+0000","last_undegraded":"2024-03-08T23:06:41.297956+0000","last_fullsized":"2024-03-08T23:06:41.297956+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:18:24.231303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292974+0000","last_change":"2024-03-08T23:04:39.076680+0000","last_active":"2024-03-08T23:06:41.292974+0000","last_peered":"2024-03-08T23:06:41.292974+0000","last_clean":"2024-03-08T23:06:41.292974+0000","last_became_active":"2024-03-08T23:04:39.075848+0000","last_became_peered":"2024-03-08T23:04:39.075848+0000","last_unstale":"2024-03-08T23:06:41.292974+0000","last_undegraded":"2024-03-08T23:06:41.292974+0000","last_fullsized":"2024-03-08T23:06:41.292974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:26:05.842929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292888+0000","last_change":"2024-03-08T23:04:43.438210+0000","last_active":"2024-03-08T23:06:41.292888+0000","last_peered":"2024-03-08T23:06:41.292888+0000","last_clean":"2024-03-08T23:06:41.292888+0000","last_became_active":"2024-03-08T23:04:43.437743+0000","last_became_peered":"2024-03-08T23:04:43.437743+0000","last_unstale":"2024-03-08T23:06:41.292888+0000","last_undegraded":"2024-03-08T23:06:41.292888+0000","last_fullsized":"2024-03-08T23:06:41.292888+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:32:50.565868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298952+0000","last_change":"2024-03-08T23:04:38.323921+0000","last_active":"2024-03-08T23:06:41.298952+0000","last_peered":"2024-03-08T23:06:41.298952+0000","last_clean":"2024-03-08T23:06:41.298952+0000","last_became_active":"2024-03-08T23:04:38.323429+0000","last_became_peered":"2024-03-08T23:04:38.323429+0000","last_unstale":"2024-03-08T23:06:41.298952+0000","last_undegraded":"2024-03-08T23:06:41.298952+0000","last_fullsized":"2024-03-08T23:06:41.298952+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:20:49.939410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295447+0000","last_change":"2024-03-08T23:05:41.148194+0000","last_active":"2024-03-08T23:06:41.295447+0000","last_peered":"2024-03-08T23:06:41.295447+0000","last_clean":"2024-03-08T23:06:41.295447+0000","last_became_active":"2024-03-08T23:05:41.147832+0000","last_became_peered":"2024-03-08T23:05:41.147832+0000","last_unstale":"2024-03-08T23:06:41.295447+0000","last_undegraded":"2024-03-08T23:06:41.295447+0000","last_fullsized":"2024-03-08T23:06:41.295447+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:40:38.756856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,6],"acting":[11,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298118+0000","last_change":"2024-03-08T23:04:43.567132+0000","last_active":"2024-03-08T23:06:41.298118+0000","last_peered":"2024-03-08T23:06:41.298118+0000","last_clean":"2024-03-08T23:06:41.298118+0000","last_became_active":"2024-03-08T23:04:43.566678+0000","last_became_peered":"2024-03-08T23:04:43.566678+0000","last_unstale":"2024-03-08T23:06:41.298118+0000","last_undegraded":"2024-03-08T23:06:41.298118+0000","last_fullsized":"2024-03-08T23:06:41.298118+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:44.936352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.437182+0000","last_change":"2024-03-08T23:04:38.340248+0000","last_active":"2024-03-08T23:06:41.437182+0000","last_peered":"2024-03-08T23:06:41.437182+0000","last_clean":"2024-03-08T23:06:41.437182+0000","last_became_active":"2024-03-08T23:04:38.337644+0000","last_became_peered":"2024-03-08T23:04:38.337644+0000","last_unstale":"2024-03-08T23:06:41.437182+0000","last_undegraded":"2024-03-08T23:06:41.437182+0000","last_fullsized":"2024-03-08T23:06:41.437182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:37:19.022906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294258+0000","last_change":"2024-03-08T23:04:39.090048+0000","last_active":"2024-03-08T23:06:41.294258+0000","last_peered":"2024-03-08T23:06:41.294258+0000","last_clean":"2024-03-08T23:06:41.294258+0000","last_became_active":"2024-03-08T23:04:39.089642+0000","last_became_peered":"2024-03-08T23:04:39.089642+0000","last_unstale":"2024-03-08T23:06:41.294258+0000","last_undegraded":"2024-03-08T23:06:41.294258+0000","last_fullsized":"2024-03-08T23:06:41.294258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:26:44.471855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307765+0000","last_change":"2024-03-08T23:04:43.439810+0000","last_active":"2024-03-08T23:06:41.307765+0000","last_peered":"2024-03-08T23:06:41.307765+0000","last_clean":"2024-03-08T23:06:41.307765+0000","last_became_active":"2024-03-08T23:04:43.439117+0000","last_became_peered":"2024-03-08T23:04:43.439117+0000","last_unstale":"2024-03-08T23:06:41.307765+0000","last_undegraded":"2024-03-08T23:06:41.307765+0000","last_fullsized":"2024-03-08T23:06:41.307765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:26:30.872758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434782+0000","last_change":"2024-03-08T23:04:38.338438+0000","last_active":"2024-03-08T23:06:41.434782+0000","last_peered":"2024-03-08T23:06:41.434782+0000","last_clean":"2024-03-08T23:06:41.434782+0000","last_became_active":"2024-03-08T23:04:38.338059+0000","last_became_peered":"2024-03-08T23:04:38.338059+0000","last_unstale":"2024-03-08T23:06:41.434782+0000","last_undegraded":"2024-03-08T23:06:41.434782+0000","last_fullsized":"2024-03-08T23:06:41.434782+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:29:41.011840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293784+0000","last_change":"2024-03-08T23:04:39.092117+0000","last_active":"2024-03-08T23:06:41.293784+0000","last_peered":"2024-03-08T23:06:41.293784+0000","last_clean":"2024-03-08T23:06:41.293784+0000","last_became_active":"2024-03-08T23:04:39.091768+0000","last_became_peered":"2024-03-08T23:04:39.091768+0000","last_unstale":"2024-03-08T23:06:41.293784+0000","last_undegraded":"2024-03-08T23:06:41.293784+0000","last_fullsized":"2024-03-08T23:06:41.293784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:33:46.885902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308278+0000","last_change":"2024-03-08T23:04:43.567957+0000","last_active":"2024-03-08T23:06:41.308278+0000","last_peered":"2024-03-08T23:06:41.308278+0000","last_clean":"2024-03-08T23:06:41.308278+0000","last_became_active":"2024-03-08T23:04:43.567055+0000","last_became_peered":"2024-03-08T23:04:43.567055+0000","last_unstale":"2024-03-08T23:06:41.308278+0000","last_undegraded":"2024-03-08T23:06:41.308278+0000","last_fullsized":"2024-03-08T23:06:41.308278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:34:17.201953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293071+0000","last_change":"2024-03-08T23:04:39.073530+0000","last_active":"2024-03-08T23:06:41.293071+0000","last_peered":"2024-03-08T23:06:41.293071+0000","last_clean":"2024-03-08T23:06:41.293071+0000","last_became_active":"2024-03-08T23:04:39.073210+0000","last_became_peered":"2024-03-08T23:04:39.073210+0000","last_unstale":"2024-03-08T23:06:41.293071+0000","last_undegraded":"2024-03-08T23:06:41.293071+0000","last_fullsized":"2024-03-08T23:06:41.293071+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:43:50.266057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"91'5","reported_seq":179,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:07:26.414027+0000","last_change":"2024-03-08T23:04:38.347719+0000","last_active":"2024-03-08T23:07:26.414027+0000","last_peered":"2024-03-08T23:07:26.414027+0000","last_clean":"2024-03-08T23:07:26.414027+0000","last_became_active":"2024-03-08T23:04:38.347095+0000","last_became_peered":"2024-03-08T23:04:38.347095+0000","last_unstale":"2024-03-08T23:07:26.414027+0000","last_undegraded":"2024-03-08T23:07:26.414027+0000","last_fullsized":"2024-03-08T23:07:26.414027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:37.024428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":142,"num_read_kb":142,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435137+0000","last_change":"2024-03-08T23:04:43.433480+0000","last_active":"2024-03-08T23:06:41.435137+0000","last_peered":"2024-03-08T23:06:41.435137+0000","last_clean":"2024-03-08T23:06:41.435137+0000","last_became_active":"2024-03-08T23:04:43.433014+0000","last_became_peered":"2024-03-08T23:04:43.433014+0000","last_unstale":"2024-03-08T23:06:41.435137+0000","last_undegraded":"2024-03-08T23:06:41.435137+0000","last_fullsized":"2024-03-08T23:06:41.435137+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:03.193580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293672+0000","last_change":"2024-03-08T23:04:39.087749+0000","last_active":"2024-03-08T23:06:41.293672+0000","last_peered":"2024-03-08T23:06:41.293672+0000","last_clean":"2024-03-08T23:06:41.293672+0000","last_became_active":"2024-03-08T23:04:39.087392+0000","last_became_peered":"2024-03-08T23:04:39.087392+0000","last_unstale":"2024-03-08T23:06:41.293672+0000","last_undegraded":"2024-03-08T23:06:41.293672+0000","last_fullsized":"2024-03-08T23:06:41.293672+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:22:13.115344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311338+0000","last_change":"2024-03-08T23:04:38.331694+0000","last_active":"2024-03-08T23:06:41.311338+0000","last_peered":"2024-03-08T23:06:41.311338+0000","last_clean":"2024-03-08T23:06:41.311338+0000","last_became_active":"2024-03-08T23:04:38.331216+0000","last_became_peered":"2024-03-08T23:04:38.331216+0000","last_unstale":"2024-03-08T23:06:41.311338+0000","last_undegraded":"2024-03-08T23:06:41.311338+0000","last_fullsized":"2024-03-08T23:06:41.311338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:57:22.448180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":367,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434495+0000","last_change":"2024-03-08T23:03:18.482627+0000","last_active":"2024-03-08T23:06:41.434495+0000","last_peered":"2024-03-08T23:06:41.434495+0000","last_clean":"2024-03-08T23:06:41.434495+0000","last_became_active":"2024-03-08T23:03:18.482094+0000","last_became_peered":"2024-03-08T23:03:18.482094+0000","last_unstale":"2024-03-08T23:06:41.434495+0000","last_undegraded":"2024-03-08T23:06:41.434495+0000","last_fullsized":"2024-03-08T23:06:41.434495+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T22:59:25.736090+0000","last_clean_scrub_stamp":"2024-03-08T22:59:25.736090+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:24:37.120258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293727+0000","last_change":"2024-03-08T23:04:43.566957+0000","last_active":"2024-03-08T23:06:41.293727+0000","last_peered":"2024-03-08T23:06:41.293727+0000","last_clean":"2024-03-08T23:06:41.293727+0000","last_became_active":"2024-03-08T23:04:43.566500+0000","last_became_peered":"2024-03-08T23:04:43.566500+0000","last_unstale":"2024-03-08T23:06:41.293727+0000","last_undegraded":"2024-03-08T23:06:41.293727+0000","last_fullsized":"2024-03-08T23:06:41.293727+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:40:21.450321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307136+0000","last_change":"2024-03-08T23:04:39.090637+0000","last_active":"2024-03-08T23:06:41.307136+0000","last_peered":"2024-03-08T23:06:41.307136+0000","last_clean":"2024-03-08T23:06:41.307136+0000","last_became_active":"2024-03-08T23:04:39.088408+0000","last_became_peered":"2024-03-08T23:04:39.088408+0000","last_unstale":"2024-03-08T23:06:41.307136+0000","last_undegraded":"2024-03-08T23:06:41.307136+0000","last_fullsized":"2024-03-08T23:06:41.307136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:50.714259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434597+0000","last_change":"2024-03-08T23:04:38.327245+0000","last_active":"2024-03-08T23:06:41.434597+0000","last_peered":"2024-03-08T23:06:41.434597+0000","last_clean":"2024-03-08T23:06:41.434597+0000","last_became_active":"2024-03-08T23:04:38.326889+0000","last_became_peered":"2024-03-08T23:04:38.326889+0000","last_unstale":"2024-03-08T23:06:41.434597+0000","last_undegraded":"2024-03-08T23:06:41.434597+0000","last_fullsized":"2024-03-08T23:06:41.434597+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:43:49.247555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295247+0000","last_change":"2024-03-08T23:04:43.438362+0000","last_active":"2024-03-08T23:06:41.295247+0000","last_peered":"2024-03-08T23:06:41.295247+0000","last_clean":"2024-03-08T23:06:41.295247+0000","last_became_active":"2024-03-08T23:04:43.437806+0000","last_became_peered":"2024-03-08T23:04:43.437806+0000","last_unstale":"2024-03-08T23:06:41.295247+0000","last_undegraded":"2024-03-08T23:06:41.295247+0000","last_fullsized":"2024-03-08T23:06:41.295247+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:10:12.065341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298642+0000","last_change":"2024-03-08T23:04:38.340010+0000","last_active":"2024-03-08T23:06:41.298642+0000","last_peered":"2024-03-08T23:06:41.298642+0000","last_clean":"2024-03-08T23:06:41.298642+0000","last_became_active":"2024-03-08T23:04:38.339570+0000","last_became_peered":"2024-03-08T23:04:38.339570+0000","last_unstale":"2024-03-08T23:06:41.298642+0000","last_undegraded":"2024-03-08T23:06:41.298642+0000","last_fullsized":"2024-03-08T23:06:41.298642+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:49:06.450417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307927+0000","last_change":"2024-03-08T23:04:39.095115+0000","last_active":"2024-03-08T23:06:41.307927+0000","last_peered":"2024-03-08T23:06:41.307927+0000","last_clean":"2024-03-08T23:06:41.307927+0000","last_became_active":"2024-03-08T23:04:39.093873+0000","last_became_peered":"2024-03-08T23:04:39.093873+0000","last_unstale":"2024-03-08T23:06:41.307927+0000","last_undegraded":"2024-03-08T23:06:41.307927+0000","last_fullsized":"2024-03-08T23:06:41.307927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:30:10.046369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293158+0000","last_change":"2024-03-08T23:04:43.569113+0000","last_active":"2024-03-08T23:06:41.293158+0000","last_peered":"2024-03-08T23:06:41.293158+0000","last_clean":"2024-03-08T23:06:41.293158+0000","last_became_active":"2024-03-08T23:04:43.568281+0000","last_became_peered":"2024-03-08T23:04:43.568281+0000","last_unstale":"2024-03-08T23:06:41.293158+0000","last_undegraded":"2024-03-08T23:06:41.293158+0000","last_fullsized":"2024-03-08T23:06:41.293158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:00:04.925516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'11","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:49.177474+0000","last_change":"2024-03-08T23:04:38.339172+0000","last_active":"2024-03-08T23:06:49.177474+0000","last_peered":"2024-03-08T23:06:49.177474+0000","last_clean":"2024-03-08T23:06:49.177474+0000","last_became_active":"2024-03-08T23:04:38.338278+0000","last_became_peered":"2024-03-08T23:04:38.338278+0000","last_unstale":"2024-03-08T23:06:49.177474+0000","last_undegraded":"2024-03-08T23:06:49.177474+0000","last_fullsized":"2024-03-08T23:06:49.177474+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:30:50.163111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294465+0000","last_change":"2024-03-08T23:04:39.093480+0000","last_active":"2024-03-08T23:06:41.294465+0000","last_peered":"2024-03-08T23:06:41.294465+0000","last_clean":"2024-03-08T23:06:41.294465+0000","last_became_active":"2024-03-08T23:04:39.093039+0000","last_became_peered":"2024-03-08T23:04:39.093039+0000","last_unstale":"2024-03-08T23:06:41.294465+0000","last_undegraded":"2024-03-08T23:06:41.294465+0000","last_fullsized":"2024-03-08T23:06:41.294465+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:07:06.217188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298315+0000","last_change":"2024-03-08T23:04:43.424399+0000","last_active":"2024-03-08T23:06:41.298315+0000","last_peered":"2024-03-08T23:06:41.298315+0000","last_clean":"2024-03-08T23:06:41.298315+0000","last_became_active":"2024-03-08T23:04:43.423894+0000","last_became_peered":"2024-03-08T23:04:43.423894+0000","last_unstale":"2024-03-08T23:06:41.298315+0000","last_undegraded":"2024-03-08T23:06:41.298315+0000","last_fullsized":"2024-03-08T23:06:41.298315+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:47:17.987588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311691+0000","last_change":"2024-03-08T23:04:38.335791+0000","last_active":"2024-03-08T23:06:41.311691+0000","last_peered":"2024-03-08T23:06:41.311691+0000","last_clean":"2024-03-08T23:06:41.311691+0000","last_became_active":"2024-03-08T23:04:38.335427+0000","last_became_peered":"2024-03-08T23:04:38.335427+0000","last_unstale":"2024-03-08T23:06:41.311691+0000","last_undegraded":"2024-03-08T23:06:41.311691+0000","last_fullsized":"2024-03-08T23:06:41.311691+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:53:07.557725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.561285+0000","last_change":"2024-03-08T23:04:39.106233+0000","last_active":"2024-03-08T23:06:41.561285+0000","last_peered":"2024-03-08T23:06:41.561285+0000","last_clean":"2024-03-08T23:06:41.561285+0000","last_became_active":"2024-03-08T23:04:39.102493+0000","last_became_peered":"2024-03-08T23:04:39.102493+0000","last_unstale":"2024-03-08T23:06:41.561285+0000","last_undegraded":"2024-03-08T23:06:41.561285+0000","last_fullsized":"2024-03-08T23:06:41.561285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:24.627176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311484+0000","last_change":"2024-03-08T23:04:43.566948+0000","last_active":"2024-03-08T23:06:41.311484+0000","last_peered":"2024-03-08T23:06:41.311484+0000","last_clean":"2024-03-08T23:06:41.311484+0000","last_became_active":"2024-03-08T23:04:43.566430+0000","last_became_peered":"2024-03-08T23:04:43.566430+0000","last_unstale":"2024-03-08T23:06:41.311484+0000","last_undegraded":"2024-03-08T23:06:41.311484+0000","last_fullsized":"2024-03-08T23:06:41.311484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:00:25.086553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297243+0000","last_change":"2024-03-08T23:04:38.339945+0000","last_active":"2024-03-08T23:06:41.297243+0000","last_peered":"2024-03-08T23:06:41.297243+0000","last_clean":"2024-03-08T23:06:41.297243+0000","last_became_active":"2024-03-08T23:04:38.339602+0000","last_became_peered":"2024-03-08T23:04:38.339602+0000","last_unstale":"2024-03-08T23:06:41.297243+0000","last_undegraded":"2024-03-08T23:06:41.297243+0000","last_fullsized":"2024-03-08T23:06:41.297243+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:02:51.662438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297758+0000","last_change":"2024-03-08T23:04:39.070393+0000","last_active":"2024-03-08T23:06:41.297758+0000","last_peered":"2024-03-08T23:06:41.297758+0000","last_clean":"2024-03-08T23:06:41.297758+0000","last_became_active":"2024-03-08T23:04:39.070064+0000","last_became_peered":"2024-03-08T23:04:39.070064+0000","last_unstale":"2024-03-08T23:06:41.297758+0000","last_undegraded":"2024-03-08T23:06:41.297758+0000","last_fullsized":"2024-03-08T23:06:41.297758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:12.149130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298880+0000","last_change":"2024-03-08T23:05:41.148196+0000","last_active":"2024-03-08T23:06:41.298880+0000","last_peered":"2024-03-08T23:06:41.298880+0000","last_clean":"2024-03-08T23:06:41.298880+0000","last_became_active":"2024-03-08T23:05:41.147730+0000","last_became_peered":"2024-03-08T23:05:41.147730+0000","last_unstale":"2024-03-08T23:06:41.298880+0000","last_undegraded":"2024-03-08T23:06:41.298880+0000","last_fullsized":"2024-03-08T23:06:41.298880+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:21:05.284970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293328+0000","last_change":"2024-03-08T23:04:38.335735+0000","last_active":"2024-03-08T23:06:41.293328+0000","last_peered":"2024-03-08T23:06:41.293328+0000","last_clean":"2024-03-08T23:06:41.293328+0000","last_became_active":"2024-03-08T23:04:38.335260+0000","last_became_peered":"2024-03-08T23:04:38.335260+0000","last_unstale":"2024-03-08T23:06:41.293328+0000","last_undegraded":"2024-03-08T23:06:41.293328+0000","last_fullsized":"2024-03-08T23:06:41.293328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:20:48.022720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295142+0000","last_change":"2024-03-08T23:04:39.099860+0000","last_active":"2024-03-08T23:06:41.295142+0000","last_peered":"2024-03-08T23:06:41.295142+0000","last_clean":"2024-03-08T23:06:41.295142+0000","last_became_active":"2024-03-08T23:04:39.099484+0000","last_became_peered":"2024-03-08T23:04:39.099484+0000","last_unstale":"2024-03-08T23:06:41.295142+0000","last_undegraded":"2024-03-08T23:06:41.295142+0000","last_fullsized":"2024-03-08T23:06:41.295142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:18:23.172770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311375+0000","last_change":"2024-03-08T23:04:43.437536+0000","last_active":"2024-03-08T23:06:41.311375+0000","last_peered":"2024-03-08T23:06:41.311375+0000","last_clean":"2024-03-08T23:06:41.311375+0000","last_became_active":"2024-03-08T23:04:43.437132+0000","last_became_peered":"2024-03-08T23:04:43.437132+0000","last_unstale":"2024-03-08T23:06:41.311375+0000","last_undegraded":"2024-03-08T23:06:41.311375+0000","last_fullsized":"2024-03-08T23:06:41.311375+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:31:06.060842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":15,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.304979+0000","last_change":"2024-03-08T23:06:41.304979+0000","last_active":"2024-03-08T23:06:41.304979+0000","last_peered":"2024-03-08T23:06:41.304979+0000","last_clean":"2024-03-08T23:06:41.304979+0000","last_became_active":"2024-03-08T23:06:41.300018+0000","last_became_peered":"2024-03-08T23:06:41.300018+0000","last_unstale":"2024-03-08T23:06:41.304979+0000","last_undegraded":"2024-03-08T23:06:41.304979+0000","last_fullsized":"2024-03-08T23:06:41.304979+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:16:30.599602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293922+0000","last_change":"2024-03-08T23:04:39.096118+0000","last_active":"2024-03-08T23:06:41.293922+0000","last_peered":"2024-03-08T23:06:41.293922+0000","last_clean":"2024-03-08T23:06:41.293922+0000","last_became_active":"2024-03-08T23:04:39.095748+0000","last_became_peered":"2024-03-08T23:04:39.095748+0000","last_unstale":"2024-03-08T23:06:41.293922+0000","last_undegraded":"2024-03-08T23:06:41.293922+0000","last_fullsized":"2024-03-08T23:06:41.293922+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:34.885714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.311564+0000","last_change":"2024-03-08T23:04:43.567040+0000","last_active":"2024-03-08T23:06:41.311564+0000","last_peered":"2024-03-08T23:06:41.311564+0000","last_clean":"2024-03-08T23:06:41.311564+0000","last_became_active":"2024-03-08T23:04:43.566474+0000","last_became_peered":"2024-03-08T23:04:43.566474+0000","last_unstale":"2024-03-08T23:06:41.311564+0000","last_undegraded":"2024-03-08T23:06:41.311564+0000","last_fullsized":"2024-03-08T23:06:41.311564+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:32:15.793083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297806+0000","last_change":"2024-03-08T23:04:38.322840+0000","last_active":"2024-03-08T23:06:41.297806+0000","last_peered":"2024-03-08T23:06:41.297806+0000","last_clean":"2024-03-08T23:06:41.297806+0000","last_became_active":"2024-03-08T23:04:38.322475+0000","last_became_peered":"2024-03-08T23:04:38.322475+0000","last_unstale":"2024-03-08T23:06:41.297806+0000","last_undegraded":"2024-03-08T23:06:41.297806+0000","last_fullsized":"2024-03-08T23:06:41.297806+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:54:38.372174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.434958+0000","last_change":"2024-03-08T23:04:39.089964+0000","last_active":"2024-03-08T23:06:41.434958+0000","last_peered":"2024-03-08T23:06:41.434958+0000","last_clean":"2024-03-08T23:06:41.434958+0000","last_became_active":"2024-03-08T23:04:39.089597+0000","last_became_peered":"2024-03-08T23:04:39.089597+0000","last_unstale":"2024-03-08T23:06:41.434958+0000","last_undegraded":"2024-03-08T23:06:41.434958+0000","last_fullsized":"2024-03-08T23:06:41.434958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:03:39.320273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297364+0000","last_change":"2024-03-08T23:04:43.446074+0000","last_active":"2024-03-08T23:06:41.297364+0000","last_peered":"2024-03-08T23:06:41.297364+0000","last_clean":"2024-03-08T23:06:41.297364+0000","last_became_active":"2024-03-08T23:04:43.445568+0000","last_became_peered":"2024-03-08T23:04:43.445568+0000","last_unstale":"2024-03-08T23:06:41.297364+0000","last_undegraded":"2024-03-08T23:06:41.297364+0000","last_fullsized":"2024-03-08T23:06:41.297364+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:59:06.016704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298721+0000","last_change":"2024-03-08T23:04:38.337621+0000","last_active":"2024-03-08T23:06:41.298721+0000","last_peered":"2024-03-08T23:06:41.298721+0000","last_clean":"2024-03-08T23:06:41.298721+0000","last_became_active":"2024-03-08T23:04:38.337248+0000","last_became_peered":"2024-03-08T23:04:38.337248+0000","last_unstale":"2024-03-08T23:06:41.298721+0000","last_undegraded":"2024-03-08T23:06:41.298721+0000","last_fullsized":"2024-03-08T23:06:41.298721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:48:26.983375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435322+0000","last_change":"2024-03-08T23:04:39.076531+0000","last_active":"2024-03-08T23:06:41.435322+0000","last_peered":"2024-03-08T23:06:41.435322+0000","last_clean":"2024-03-08T23:06:41.435322+0000","last_became_active":"2024-03-08T23:04:39.076010+0000","last_became_peered":"2024-03-08T23:04:39.076010+0000","last_unstale":"2024-03-08T23:06:41.435322+0000","last_undegraded":"2024-03-08T23:06:41.435322+0000","last_fullsized":"2024-03-08T23:06:41.435322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:57:30.929168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.559863+0000","last_change":"2024-03-08T23:04:43.448839+0000","last_active":"2024-03-08T23:06:41.559863+0000","last_peered":"2024-03-08T23:06:41.559863+0000","last_clean":"2024-03-08T23:06:41.559863+0000","last_became_active":"2024-03-08T23:04:43.448091+0000","last_became_peered":"2024-03-08T23:04:43.448091+0000","last_unstale":"2024-03-08T23:06:41.559863+0000","last_undegraded":"2024-03-08T23:06:41.559863+0000","last_fullsized":"2024-03-08T23:06:41.559863+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:05:48.281073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293626+0000","last_change":"2024-03-08T23:04:38.336991+0000","last_active":"2024-03-08T23:06:41.293626+0000","last_peered":"2024-03-08T23:06:41.293626+0000","last_clean":"2024-03-08T23:06:41.293626+0000","last_became_active":"2024-03-08T23:04:38.336471+0000","last_became_peered":"2024-03-08T23:04:38.336471+0000","last_unstale":"2024-03-08T23:06:41.293626+0000","last_undegraded":"2024-03-08T23:06:41.293626+0000","last_fullsized":"2024-03-08T23:06:41.293626+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:46:39.797550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435008+0000","last_change":"2024-03-08T23:05:41.148243+0000","last_active":"2024-03-08T23:06:41.435008+0000","last_peered":"2024-03-08T23:06:41.435008+0000","last_clean":"2024-03-08T23:06:41.435008+0000","last_became_active":"2024-03-08T23:05:41.147924+0000","last_became_peered":"2024-03-08T23:05:41.147924+0000","last_unstale":"2024-03-08T23:06:41.435008+0000","last_undegraded":"2024-03-08T23:06:41.435008+0000","last_fullsized":"2024-03-08T23:06:41.435008+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:47.129092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292704+0000","last_change":"2024-03-08T23:04:43.427007+0000","last_active":"2024-03-08T23:06:41.292704+0000","last_peered":"2024-03-08T23:06:41.292704+0000","last_clean":"2024-03-08T23:06:41.292704+0000","last_became_active":"2024-03-08T23:04:43.426581+0000","last_became_peered":"2024-03-08T23:04:43.426581+0000","last_unstale":"2024-03-08T23:06:41.292704+0000","last_undegraded":"2024-03-08T23:06:41.292704+0000","last_fullsized":"2024-03-08T23:06:41.292704+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:42.378500+0000","last_clean_scrub_stamp":"2024-03-08T23:04:42.378500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:08:12.220761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308063+0000","last_change":"2024-03-08T23:05:40.661217+0000","last_active":"2024-03-08T23:06:41.308063+0000","last_peered":"2024-03-08T23:06:41.308063+0000","last_clean":"2024-03-08T23:06:41.308063+0000","last_became_active":"2024-03-08T23:05:40.657165+0000","last_became_peered":"2024-03-08T23:05:40.657165+0000","last_unstale":"2024-03-08T23:06:41.308063+0000","last_undegraded":"2024-03-08T23:06:41.308063+0000","last_fullsized":"2024-03-08T23:06:41.308063+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:37.196977+0000","last_clean_scrub_stamp":"2024-03-08T23:04:37.196977+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:16.125758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.435283+0000","last_change":"2024-03-08T23:04:39.072910+0000","last_active":"2024-03-08T23:06:41.435283+0000","last_peered":"2024-03-08T23:06:41.435283+0000","last_clean":"2024-03-08T23:06:41.435283+0000","last_became_active":"2024-03-08T23:04:39.072556+0000","last_became_peered":"2024-03-08T23:04:39.072556+0000","last_unstale":"2024-03-08T23:06:41.435283+0000","last_undegraded":"2024-03-08T23:06:41.435283+0000","last_fullsized":"2024-03-08T23:06:41.435283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:55:22.182907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295157+0000","last_change":"2024-03-08T23:04:39.074627+0000","last_active":"2024-03-08T23:06:41.295157+0000","last_peered":"2024-03-08T23:06:41.295157+0000","last_clean":"2024-03-08T23:06:41.295157+0000","last_became_active":"2024-03-08T23:04:39.074254+0000","last_became_peered":"2024-03-08T23:04:39.074254+0000","last_unstale":"2024-03-08T23:06:41.295157+0000","last_undegraded":"2024-03-08T23:06:41.295157+0000","last_fullsized":"2024-03-08T23:06:41.295157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:06:31.478654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.299154+0000","last_change":"2024-03-08T23:05:40.651674+0000","last_active":"2024-03-08T23:06:41.299154+0000","last_peered":"2024-03-08T23:06:41.299154+0000","last_clean":"2024-03-08T23:06:41.299154+0000","last_became_active":"2024-03-08T23:05:40.651393+0000","last_became_peered":"2024-03-08T23:05:40.651393+0000","last_unstale":"2024-03-08T23:06:41.299154+0000","last_undegraded":"2024-03-08T23:06:41.299154+0000","last_fullsized":"2024-03-08T23:06:41.299154+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:30:44.691787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,8],"acting":[2,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298225+0000","last_change":"2024-03-08T23:04:39.097991+0000","last_active":"2024-03-08T23:06:41.298225+0000","last_peered":"2024-03-08T23:06:41.298225+0000","last_clean":"2024-03-08T23:06:41.298225+0000","last_became_active":"2024-03-08T23:04:39.097683+0000","last_became_peered":"2024-03-08T23:04:39.097683+0000","last_unstale":"2024-03-08T23:06:41.298225+0000","last_undegraded":"2024-03-08T23:06:41.298225+0000","last_fullsized":"2024-03-08T23:06:41.298225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:05:55.603047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.560171+0000","last_change":"2024-03-08T23:04:39.101194+0000","last_active":"2024-03-08T23:06:41.560171+0000","last_peered":"2024-03-08T23:06:41.560171+0000","last_clean":"2024-03-08T23:06:41.560171+0000","last_became_active":"2024-03-08T23:04:39.097845+0000","last_became_peered":"2024-03-08T23:04:39.097845+0000","last_unstale":"2024-03-08T23:06:41.560171+0000","last_undegraded":"2024-03-08T23:06:41.560171+0000","last_fullsized":"2024-03-08T23:06:41.560171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:12.141069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.308038+0000","last_change":"2024-03-08T23:05:40.993334+0000","last_active":"2024-03-08T23:06:41.308038+0000","last_peered":"2024-03-08T23:06:41.308038+0000","last_clean":"2024-03-08T23:06:41.308038+0000","last_became_active":"2024-03-08T23:05:40.991755+0000","last_became_peered":"2024-03-08T23:05:40.991755+0000","last_unstale":"2024-03-08T23:06:41.308038+0000","last_undegraded":"2024-03-08T23:06:41.308038+0000","last_fullsized":"2024-03-08T23:06:41.308038+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:09:31.085388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.295481+0000","last_change":"2024-03-08T23:04:39.072534+0000","last_active":"2024-03-08T23:06:41.295481+0000","last_peered":"2024-03-08T23:06:41.295481+0000","last_clean":"2024-03-08T23:06:41.295481+0000","last_became_active":"2024-03-08T23:04:39.072193+0000","last_became_peered":"2024-03-08T23:04:39.072193+0000","last_unstale":"2024-03-08T23:06:41.295481+0000","last_undegraded":"2024-03-08T23:06:41.295481+0000","last_fullsized":"2024-03-08T23:06:41.295481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:11:57.257114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.298991+0000","last_change":"2024-03-08T23:04:39.076371+0000","last_active":"2024-03-08T23:06:41.298991+0000","last_peered":"2024-03-08T23:06:41.298991+0000","last_clean":"2024-03-08T23:06:41.298991+0000","last_became_active":"2024-03-08T23:04:39.076050+0000","last_became_peered":"2024-03-08T23:04:39.076050+0000","last_unstale":"2024-03-08T23:06:41.298991+0000","last_undegraded":"2024-03-08T23:06:41.298991+0000","last_fullsized":"2024-03-08T23:06:41.298991+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:55.833577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.294678+0000","last_change":"2024-03-08T23:04:39.089226+0000","last_active":"2024-03-08T23:06:41.294678+0000","last_peered":"2024-03-08T23:06:41.294678+0000","last_clean":"2024-03-08T23:06:41.294678+0000","last_became_active":"2024-03-08T23:04:39.088815+0000","last_became_peered":"2024-03-08T23:04:39.088815+0000","last_unstale":"2024-03-08T23:06:41.294678+0000","last_undegraded":"2024-03-08T23:06:41.294678+0000","last_fullsized":"2024-03-08T23:06:41.294678+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:54:34.254422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.292981+0000","last_change":"2024-03-08T23:04:39.078457+0000","last_active":"2024-03-08T23:06:41.292981+0000","last_peered":"2024-03-08T23:06:41.292981+0000","last_clean":"2024-03-08T23:06:41.292981+0000","last_became_active":"2024-03-08T23:04:39.077985+0000","last_became_peered":"2024-03-08T23:04:39.077985+0000","last_unstale":"2024-03-08T23:06:41.292981+0000","last_undegraded":"2024-03-08T23:06:41.292981+0000","last_fullsized":"2024-03-08T23:06:41.292981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:59:17.000485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.293449+0000","last_change":"2024-03-08T23:04:39.082989+0000","last_active":"2024-03-08T23:06:41.293449+0000","last_peered":"2024-03-08T23:06:41.293449+0000","last_clean":"2024-03-08T23:06:41.293449+0000","last_became_active":"2024-03-08T23:04:39.076808+0000","last_became_peered":"2024-03-08T23:04:39.076808+0000","last_unstale":"2024-03-08T23:06:41.293449+0000","last_undegraded":"2024-03-08T23:06:41.293449+0000","last_fullsized":"2024-03-08T23:06:41.293449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:55:15.476438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.297515+0000","last_change":"2024-03-08T23:04:39.094448+0000","last_active":"2024-03-08T23:06:41.297515+0000","last_peered":"2024-03-08T23:06:41.297515+0000","last_clean":"2024-03-08T23:06:41.297515+0000","last_became_active":"2024-03-08T23:04:39.094072+0000","last_became_peered":"2024-03-08T23:04:39.094072+0000","last_unstale":"2024-03-08T23:06:41.297515+0000","last_undegraded":"2024-03-08T23:06:41.297515+0000","last_fullsized":"2024-03-08T23:06:41.297515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-09T23:53:51.718557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.307876+0000","last_change":"2024-03-08T23:04:39.105582+0000","last_active":"2024-03-08T23:06:41.307876+0000","last_peered":"2024-03-08T23:06:41.307876+0000","last_clean":"2024-03-08T23:06:41.307876+0000","last_became_active":"2024-03-08T23:04:39.104290+0000","last_became_peered":"2024-03-08T23:04:39.104290+0000","last_unstale":"2024-03-08T23:06:41.307876+0000","last_undegraded":"2024-03-08T23:06:41.307876+0000","last_fullsized":"2024-03-08T23:06:41.307876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:11.349129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-08T23:06:41.296016+0000","last_change":"2024-03-08T23:04:39.075721+0000","last_active":"2024-03-08T23:06:41.296016+0000","last_peered":"2024-03-08T23:06:41.296016+0000","last_clean":"2024-03-08T23:06:41.296016+0000","last_became_active":"2024-03-08T23:04:39.075400+0000","last_became_peered":"2024-03-08T23:04:39.075400+0000","last_unstale":"2024-03-08T23:06:41.296016+0000","last_undegraded":"2024-03-08T23:06:41.296016+0000","last_fullsized":"2024-03-08T23:06:41.296016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:04:38.293296+0000","last_clean_scrub_stamp":"2024-03-08T23:04:38.293296+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:59:56.194212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284733,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":72678,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":895,"num_read_kb":526,"num_write":176,"num_write_kb":131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":218034,"data_compressed":5463,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":152,"ondisk_log_size":152,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383728,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":519484,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645365,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":1175288,"data_compressed":64572,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808891,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1040384,"data_stored":523816,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070530,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1036288,"data_stored":519640,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332166,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1085440,"data_stored":1116222,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495692,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":514896,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574952,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1085440,"data_stored":582810,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32196,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712956,"statfs":{"total":95995035648,"available":95962066944,"internally_reserved":0,"allocated":1040384,"data_stored":516284,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36420,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708732,"statfs":{"total":95995035648,"available":95957741568,"internally_reserved":0,"allocated":1110016,"data_stored":584692,"data_compressed":28365,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182492,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515090,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920857,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32188,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712964,"statfs":{"total":95995035648,"available":95962075136,"internally_reserved":0,"allocated":1028096,"data_stored":520078,"data_compressed":26544,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757330,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1093632,"data_stored":1111278,"data_compressed":62751,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34566,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36207,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":68004,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69886,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":70114,"data_compressed":1821,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-08T23:07:33.141 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:33.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:33 smithi046 ceph-mon[31859]: pgmap v409: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:33.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:33 smithi046 ceph-mon[31859]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:33.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:33 smithi062 ceph-mon[35761]: pgmap v409: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:33.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:33 smithi062 ceph-mon[35761]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 -- 172.21.15.46:0/398769342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 msgr2=0x7fd3ec10fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 --2- 172.21.15.46:0/398769342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec10fd70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd3e4009900 tx=0x7fd3e402bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 -- 172.21.15.46:0/398769342 shutdown_connections 2024-03-08T23:07:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 --2- 172.21.15.46:0/398769342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec10fd70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 --2- 172.21.15.46:0/398769342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 0x7fd3ec108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 --2- 172.21.15.46:0/398769342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec107d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 -- 172.21.15.46:0/398769342 >> 172.21.15.46:0/398769342 conn(0x7fd3ec076900 msgr2=0x7fd3ec078d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 -- 172.21.15.46:0/398769342 shutdown_connections 2024-03-08T23:07:33.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.400+0000 7fd3f1000640 1 -- 172.21.15.46:0/398769342 wait complete. 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.401+0000 7fd3f1000640 1 Processor -- start 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.401+0000 7fd3f1000640 1 -- start start 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 0x7fd3ec19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec1a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec113600 con 0x7fd3ec1094e0 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec113480 con 0x7fd3ec107930 2024-03-08T23:07:33.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3f1000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec113780 con 0x7fd3ec108b20 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3ea800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec1a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 0x7fd3ec19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48860/0 (socket says 172.21.15.46:48860) 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 -- 172.21.15.46:0/4182390893 learned_addr learned my addr 172.21.15.46:0/4182390893 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 -- 172.21.15.46:0/4182390893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 msgr2=0x7fd3ec19e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 0x7fd3ec19e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 -- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 msgr2=0x7fd3ec1a5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec1a5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3e9e00640 1 -- 172.21.15.46:0/4182390893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec1a57e0 con 0x7fd3ec107930 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.402+0000 7fd3ea800640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec1a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3e9e00640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ebc0 tx=0x7fd3dc00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:33.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc003b70 con 0x7fd3ec107930 2024-03-08T23:07:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec1a5ad0 con 0x7fd3ec107930 2024-03-08T23:07:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd3dc003d10 con 0x7fd3ec107930 2024-03-08T23:07:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc019980 con 0x7fd3ec107930 2024-03-08T23:07:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.403+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec1a6070 con 0x7fd3ec107930 2024-03-08T23:07:33.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.404+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec10eb80 con 0x7fd3ec107930 2024-03-08T23:07:33.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.407+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd3dc019b20 con 0x7fd3ec107930 2024-03-08T23:07:33.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.408+0000 7fd3e2a00640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.408+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fd3dc0cf6b0 con 0x7fd3ec107930 2024-03-08T23:07:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.408+0000 7fd3e9400640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.409+0000 7fd3e9400640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 0x7fd3d00a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec19f0d0 tx=0x7fd3d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:33.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.410+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd3dc09c400 con 0x7fd3ec107930 2024-03-08T23:07:33.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:33 smithi192 ceph-mon[36700]: pgmap v409: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:33.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:33 smithi192 ceph-mon[36700]: from='client.25448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:33.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.613+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3ec02d040 con 0x7fd3ec107930 2024-03-08T23:07:33.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.615+0000 7fd3e2a00640 1 -- 172.21.15.46:0/4182390893 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7fd3dc09bd00 con 0x7fd3ec107930 2024-03-08T23:07:33.619 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:33.619 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.617+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 msgr2=0x7fd3d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.617+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 0x7fd3d00a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec19f0d0 tx=0x7fd3d8005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 msgr2=0x7fd3ec19dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc00ebc0 tx=0x7fd3dc00c6a0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 shutdown_connections 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fd3d00a2880 0x7fd3d00a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd3ec1094e0 0x7fd3ec1a5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd3ec108b20 0x7fd3ec19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 --2- 172.21.15.46:0/4182390893 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd3ec107930 0x7fd3ec19dbc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 >> 172.21.15.46:0/4182390893 conn(0x7fd3ec076900 msgr2=0x7fd3ec10bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 shutdown_connections 2024-03-08T23:07:33.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.618+0000 7fd3f1000640 1 -- 172.21.15.46:0/4182390893 wait complete. 2024-03-08T23:07:33.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:07:33.642 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 -- 172.21.15.46:0/1733240817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00103510 msgr2=0x7f7e00103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 --2- 172.21.15.46:0/1733240817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00103510 0x7f7e00103930 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009900 tx=0x7f7df002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 -- 172.21.15.46:0/1733240817 shutdown_connections 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 --2- 172.21.15.46:0/1733240817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e001050c0 0x7f7e0010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 --2- 172.21.15.46:0/1733240817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 0x7f7e00104b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 --2- 172.21.15.46:0/1733240817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e00103510 0x7f7e00103930 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 -- 172.21.15.46:0/1733240817 >> 172.21.15.46:0/1733240817 conn(0x7f7e000feaa0 msgr2=0x7f7e00100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:33.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 -- 172.21.15.46:0/1733240817 shutdown_connections 2024-03-08T23:07:33.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.910+0000 7f7e06800640 1 -- 172.21.15.46:0/1733240817 wait complete. 2024-03-08T23:07:33.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 Processor -- start 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 -- start start 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 0x7f7e00199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 0x7f7e001a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0010f1a0 con 0x7f7e00103510 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0010f020 con 0x7f7e00104700 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7e06800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e0010f320 con 0x7f7e001050c0 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.911+0000 7f7dff400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58102/0 (socket says 172.21.15.46:58102) 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 -- 172.21.15.46:0/1463790075 learned_addr learned my addr 172.21.15.46:0/1463790075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:33.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dffe00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 0x7f7e001a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dfea00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 0x7f7e00199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 -- 172.21.15.46:0/1463790075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 msgr2=0x7f7e001a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 0x7f7e001a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 -- 172.21.15.46:0/1463790075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 msgr2=0x7f7e00199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 0x7f7e00199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 -- 172.21.15.46:0/1463790075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e001a1470 con 0x7f7e00103510 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dffe00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 0x7f7e001a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:33.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.912+0000 7f7dff400640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.913+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0041070 con 0x7f7e00103510 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.913+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7df00044d0 con 0x7f7e00103510 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.913+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7df0034440 con 0x7f7e00103510 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.913+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e001a1700 con 0x7f7e00103510 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.913+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e001a1c40 con 0x7f7e00103510 2024-03-08T23:07:33.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.914+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0010a880 con 0x7f7e00103510 2024-03-08T23:07:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.916+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7df0044070 con 0x7f7e00103510 2024-03-08T23:07:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.917+0000 7f7df7e00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:33.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.917+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f7df00f08b0 con 0x7f7e00103510 2024-03-08T23:07:33.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.917+0000 7f7dfea00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:33.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.918+0000 7f7dfea00640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 0x7f7de40a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e0019ad60 tx=0x7f7dec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:33.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:33.921+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7df00bcf20 con 0x7f7e00103510 2024-03-08T23:07:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.126+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7e0002d090 con 0x7f7e00103510 2024-03-08T23:07:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.127+0000 7f7df7e00640 1 -- 172.21.15.46:0/1463790075 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7df00c0fc0 con 0x7f7e00103510 2024-03-08T23:07:34.131 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 msgr2=0x7f7de40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 0x7f7de40a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7e0019ad60 tx=0x7f7dec005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 msgr2=0x7f7e00199850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df0005d50 comp rx=0 tx=0).stop 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 shutdown_connections 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7e001050c0 0x7f7e001a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7e00104700 0x7f7e00199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 --2- 172.21.15.46:0/1463790075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7e00103510 0x7f7e00199850 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 >> 172.21.15.46:0/1463790075 conn(0x7f7e000feaa0 msgr2=0x7f7e00107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 shutdown_connections 2024-03-08T23:07:34.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.130+0000 7f7e06800640 1 -- 172.21.15.46:0/1463790075 wait complete. 2024-03-08T23:07:34.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 -- 172.21.15.46:0/2979899631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 msgr2=0x7f6acc108f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/2979899631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc108f80 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6abc009900 tx=0x7f6abc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 -- 172.21.15.46:0/2979899631 shutdown_connections 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/2979899631 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc1094c0 0x7f6acc10fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/2979899631 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc108f80 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/2979899631 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc107910 0x7f6acc107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.380+0000 7f6ad3a00640 1 -- 172.21.15.46:0/2979899631 >> 172.21.15.46:0/2979899631 conn(0x7f6acc0768b0 msgr2=0x7f6acc078cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.381+0000 7f6ad3a00640 1 -- 172.21.15.46:0/2979899631 shutdown_connections 2024-03-08T23:07:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.381+0000 7f6ad3a00640 1 -- 172.21.15.46:0/2979899631 wait complete. 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 Processor -- start 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 -- start start 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 0x7f6acc19dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc19e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 0x7f6acc1a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc1135e0 con 0x7f6acc1094c0 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc113460 con 0x7f6acc108b00 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.382+0000 7f6ad3a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc113760 con 0x7f6acc107910 2024-03-08T23:07:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6ad1000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 0x7f6acc19dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc19e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6ad1000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 0x7f6acc19dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44846/0 (socket says 172.21.15.46:44846) 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6ad1000640 1 -- 172.21.15.46:0/3272334847 learned_addr learned my addr 172.21.15.46:0/3272334847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6ad1a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 0x7f6acc1a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 -- 172.21.15.46:0/3272334847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 msgr2=0x7f6acc19dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 0x7f6acc19dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 -- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 msgr2=0x7f6acc1a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 0x7f6acc1a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6acbe00640 1 -- 172.21.15.46:0/3272334847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc1a5810 con 0x7f6acc108b00 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.383+0000 7f6ad1a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 0x7f6acc1a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6acbe00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc19e130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6abc036040 tx=0x7f6abc02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc043020 con 0x7f6acc108b00 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc1a5aa0 con 0x7f6acc108b00 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6abc0054e0 con 0x7f6acc108b00 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc003c80 con 0x7f6acc108b00 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.384+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc1a6010 con 0x7f6acc108b00 2024-03-08T23:07:34.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.385+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc10ec30 con 0x7f6acc108b00 2024-03-08T23:07:34.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.387+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6abc046020 con 0x7f6acc108b00 2024-03-08T23:07:34.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:34 smithi062 ceph-mon[35761]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:34 smithi062 ceph-mon[35761]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:34 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4182390893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.388+0000 7f6ac9600640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 0x7f6ab40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.389+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f6abc0f0ea0 con 0x7f6acc108b00 2024-03-08T23:07:34.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.389+0000 7f6ad1000640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 0x7f6ab40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.389+0000 7f6ad1000640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 0x7f6ab40a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000bc80 tx=0x7f6ac000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.393+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6abc04d020 con 0x7f6acc108b00 2024-03-08T23:07:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:34 smithi046 ceph-mon[31859]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:34 smithi046 ceph-mon[31859]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:34 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4182390893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:34 smithi192 ceph-mon[36700]: from='client.15345 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:34 smithi192 ceph-mon[36700]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:07:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:34 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4182390893' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:34.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.597+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6acc061ef0 con 0x7f6acc108b00 2024-03-08T23:07:34.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.598+0000 7f6ac9600640 1 -- 172.21.15.46:0/3272334847 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6abc0bdb60 con 0x7f6acc108b00 2024-03-08T23:07:34.602 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 msgr2=0x7f6ab40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 0x7f6ab40a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000bc80 tx=0x7f6ac000b410 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 msgr2=0x7f6acc19e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc19e130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6abc036040 tx=0x7f6abc02c6c0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 shutdown_connections 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f6ab40a2800 0x7f6ab40a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6acc1094c0 0x7f6acc1a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6acc108b00 0x7f6acc19e130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 --2- 172.21.15.46:0/3272334847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6acc107910 0x7f6acc19dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 >> 172.21.15.46:0/3272334847 conn(0x7f6acc0768b0 msgr2=0x7f6acc10be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:34.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 shutdown_connections 2024-03-08T23:07:34.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.602+0000 7f6ad3a00640 1 -- 172.21.15.46:0/3272334847 wait complete. 2024-03-08T23:07:34.628 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.851+0000 7efd39800640 1 -- 172.21.15.46:0/3040179234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 msgr2=0x7efd34104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.851+0000 7efd39800640 1 --2- 172.21.15.46:0/3040179234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34104b80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009900 tx=0x7efd1c02bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 -- 172.21.15.46:0/3040179234 shutdown_connections 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 --2- 172.21.15.46:0/3040179234 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd3410b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 --2- 172.21.15.46:0/3040179234 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34104b80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 --2- 172.21.15.46:0/3040179234 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 -- 172.21.15.46:0/3040179234 >> 172.21.15.46:0/3040179234 conn(0x7efd340feaa0 msgr2=0x7efd34100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 -- 172.21.15.46:0/3040179234 shutdown_connections 2024-03-08T23:07:34.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.852+0000 7efd39800640 1 -- 172.21.15.46:0/3040179234 wait complete. 2024-03-08T23:07:34.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 Processor -- start 2024-03-08T23:07:34.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 -- start start 2024-03-08T23:07:34.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd341a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3410f1f0 con 0x7efd341050c0 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3410f070 con 0x7efd34104700 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.853+0000 7efd39800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd3410f370 con 0x7efd34103510 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd32a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd32a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44856/0 (socket says 172.21.15.46:44856) 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd32a00640 1 -- 172.21.15.46:0/2798470738 learned_addr learned my addr 172.21.15.46:0/2798470738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd33400640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd341a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 -- 172.21.15.46:0/2798470738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 msgr2=0x7efd34199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 -- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 msgr2=0x7efd341a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd341a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:34.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd2a000640 1 -- 172.21.15.46:0/2798470738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd341a1490 con 0x7efd34104700 2024-03-08T23:07:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd32a00640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.854+0000 7efd33400640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd341a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:34.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd2a000640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34199db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd1c02c180 tx=0x7efd1c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c041070 con 0x7efd34104700 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd341a1720 con 0x7efd34104700 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd1c004450 con 0x7efd34104700 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1c003df0 con 0x7efd34104700 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.855+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd341a1c30 con 0x7efd34104700 2024-03-08T23:07:34.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.856+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd3410a7d0 con 0x7efd34104700 2024-03-08T23:07:34.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.858+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efd1c044070 con 0x7efd34104700 2024-03-08T23:07:34.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.860+0000 7efd2be00640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 0x7efd180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.860+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7efd1c0f0b30 con 0x7efd34104700 2024-03-08T23:07:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.860+0000 7efd32a00640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 0x7efd180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:34.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.861+0000 7efd32a00640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 0x7efd180a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efd240058c0 tx=0x7efd24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:34.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:34.862+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efd1c0bd1a0 con 0x7efd34104700 2024-03-08T23:07:35.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.065+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efd3402d070 con 0x7efd34104700 2024-03-08T23:07:35.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.065+0000 7efd2be00640 1 -- 172.21.15.46:0/2798470738 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7efd1c0c1240 con 0x7efd34104700 2024-03-08T23:07:35.069 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:07:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 msgr2=0x7efd180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 0x7efd180a4cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7efd240058c0 tx=0x7efd24005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 msgr2=0x7efd34199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34199db0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7efd1c02c180 tx=0x7efd1c02c660 comp rx=0 tx=0).stop 2024-03-08T23:07:35.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 shutdown_connections 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7efd180a2800 0x7efd180a4cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7efd341050c0 0x7efd341a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7efd34104700 0x7efd34199db0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 --2- 172.21.15.46:0/2798470738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efd34103510 0x7efd34199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 >> 172.21.15.46:0/2798470738 conn(0x7efd340feaa0 msgr2=0x7efd341079a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 shutdown_connections 2024-03-08T23:07:35.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.068+0000 7efd39800640 1 -- 172.21.15.46:0/2798470738 wait complete. 2024-03-08T23:07:35.092 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:07:35.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 -- 172.21.15.46:0/2382708014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 msgr2=0x7f6894108e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 --2- 172.21.15.46:0/2382708014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f6894108e80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6884009900 tx=0x7f688402bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 -- 172.21.15.46:0/2382708014 shutdown_connections 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 --2- 172.21.15.46:0/2382708014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68941093c0 0x7f689410fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 --2- 172.21.15.46:0/2382708014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f6894108e80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 --2- 172.21.15.46:0/2382708014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6894107810 0x7f6894107c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 -- 172.21.15.46:0/2382708014 >> 172.21.15.46:0/2382708014 conn(0x7f68940767c0 msgr2=0x7f6894078be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.316+0000 7f689b200640 1 -- 172.21.15.46:0/2382708014 shutdown_connections 2024-03-08T23:07:35.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.317+0000 7f689b200640 1 -- 172.21.15.46:0/2382708014 wait complete. 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 Processor -- start 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 -- start start 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f689419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68941093c0 0x7f68941a5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6894113470 con 0x7f6894107810 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941132f0 con 0x7f68941093c0 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.318+0000 7f689b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68941135f0 con 0x7f6894108a00 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58152/0 (socket says 172.21.15.46:58152) 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 -- 172.21.15.46:0/2895541923 learned_addr learned my addr 172.21.15.46:0/2895541923 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899400640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f689419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f689a800640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68941093c0 0x7f68941a5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 -- 172.21.15.46:0/2895541923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 msgr2=0x7f689419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f689419e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 -- 172.21.15.46:0/2895541923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68941093c0 msgr2=0x7f68941a5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68941093c0 0x7f68941a5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.319+0000 7f6899e00640 1 -- 172.21.15.46:0/2895541923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68941a5700 con 0x7f6894107810 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f6899e00640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f688800e9c0 tx=0x7f688800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688800cd80 con 0x7f6894107810 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68941a59f0 con 0x7f6894107810 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68880035f0 con 0x7f6894107810 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6888018630 con 0x7f6894107810 2024-03-08T23:07:35.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.320+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68941a5f60 con 0x7f6894107810 2024-03-08T23:07:35.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.321+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689410eb30 con 0x7f6894107810 2024-03-08T23:07:35.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.323+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f68880037a0 con 0x7f6894107810 2024-03-08T23:07:35.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.325+0000 7f688ea00640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.325+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f68880cf110 con 0x7f6894107810 2024-03-08T23:07:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.325+0000 7f6899400640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.326+0000 7f6899400640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 0x7f687c0a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f688402c7c0 tx=0x7f6884005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:35.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.330+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f688809be10 con 0x7f6894107810 2024-03-08T23:07:35.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:35 smithi062 ceph-mon[35761]: pgmap v410: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-08T23:07:35.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:35 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1463790075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:35 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3272334847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:35 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2798470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:35 smithi046 ceph-mon[31859]: pgmap v410: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-08T23:07:35.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:35 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1463790075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:35 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3272334847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:35 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2798470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.534+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6894061ef0 con 0x7f6894107810 2024-03-08T23:07:35.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.536+0000 7f688ea00640 1 -- 172.21.15.46:0/2895541923 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f688809b710 con 0x7f6894107810 2024-03-08T23:07:35.539 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:07:35.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 msgr2=0x7f687c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 0x7f687c0a4cc0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f688402c7c0 tx=0x7f6884005fd0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 msgr2=0x7f689419dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f688800e9c0 tx=0x7f688800ee80 comp rx=0 tx=0).stop 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 shutdown_connections 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f687c0a2800 0x7f687c0a4cc0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68941093c0 0x7f68941a5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6894108a00 0x7f689419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 --2- 172.21.15.46:0/2895541923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6894107810 0x7f689419dae0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 >> 172.21.15.46:0/2895541923 conn(0x7f68940767c0 msgr2=0x7f689410bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.538+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 shutdown_connections 2024-03-08T23:07:35.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.539+0000 7f689b200640 1 -- 172.21.15.46:0/2895541923 wait complete. 2024-03-08T23:07:35.562 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:07:35.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:35 smithi192 ceph-mon[36700]: pgmap v410: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-08T23:07:35.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1463790075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3272334847' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:35 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2798470738' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 -- 172.21.15.46:0/533243532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 msgr2=0x7f9f20104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 --2- 172.21.15.46:0/533243532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20104ba0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c009900 tx=0x7f9f0c02bc40 comp rx=0 tx=0).stop 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 -- 172.21.15.46:0/533243532 shutdown_connections 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 --2- 172.21.15.46:0/533243532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 0x7f9f2010b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 --2- 172.21.15.46:0/533243532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20104ba0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 --2- 172.21.15.46:0/533243532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 -- 172.21.15.46:0/533243532 >> 172.21.15.46:0/533243532 conn(0x7f9f200feae0 msgr2=0x7f9f20100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 -- 172.21.15.46:0/533243532 shutdown_connections 2024-03-08T23:07:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.785+0000 7f9f26400640 1 -- 172.21.15.46:0/533243532 wait complete. 2024-03-08T23:07:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.786+0000 7f9f26400640 1 Processor -- start 2024-03-08T23:07:35.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 -- start start 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 0x7f9f201a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2010f210 con 0x7f9f20104720 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2010f090 con 0x7f9f201050e0 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f26400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2010f390 con 0x7f9f20103530 2024-03-08T23:07:35.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f1ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f1ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58162/0 (socket says 172.21.15.46:58162) 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f1ea00640 1 -- 172.21.15.46:0/3471713382 learned_addr learned my addr 172.21.15.46:0/3471713382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.787+0000 7f9f1f400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1fe00640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 0x7f9f201a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 -- 172.21.15.46:0/3471713382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 msgr2=0x7f9f20199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 -- 172.21.15.46:0/3471713382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 msgr2=0x7f9f201a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 0x7f9f201a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 -- 172.21.15.46:0/3471713382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f201a1430 con 0x7f9f20104720 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1f400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f1ea00640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c02c8f0 tx=0x7f9f0c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c041070 con 0x7f9f20104720 2024-03-08T23:07:35.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9f0c0054e0 con 0x7f9f20104720 2024-03-08T23:07:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c034440 con 0x7f9f20104720 2024-03-08T23:07:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.788+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f201a16c0 con 0x7f9f20104720 2024-03-08T23:07:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.789+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f201a1b80 con 0x7f9f20104720 2024-03-08T23:07:35.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.790+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f2010a7f0 con 0x7f9f20104720 2024-03-08T23:07:35.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.792+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9f0c044070 con 0x7f9f20104720 2024-03-08T23:07:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.793+0000 7f9f17e00640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 0x7f9f040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.793+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f9f0c0f0b90 con 0x7f9f20104720 2024-03-08T23:07:35.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.794+0000 7f9f1f400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 0x7f9f040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:35.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.794+0000 7f9f1f400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 0x7f9f040a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9f100058c0 tx=0x7f9f10005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:35.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:35.798+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9f0c0bd850 con 0x7f9f20104720 2024-03-08T23:07:36.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.042+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f2002ce70 con 0x7f9f20104720 2024-03-08T23:07:36.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.044+0000 7f9f17e00640 1 -- 172.21.15.46:0/3471713382 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f9f0c0dd0d0 con 0x7f9f20104720 2024-03-08T23:07:36.048 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:36.048 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:07:25.750030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:07:36.048 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.046+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 msgr2=0x7f9f040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 0x7f9f040a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f9f100058c0 tx=0x7f9f10005850 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 msgr2=0x7f9f20199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c02c8f0 tx=0x7f9f0c005e40 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 shutdown_connections 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f9f040a2800 0x7f9f040a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9f201050e0 0x7f9f201a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9f20104720 0x7f9f20199d50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 --2- 172.21.15.46:0/3471713382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9f20103530 0x7f9f20199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 >> 172.21.15.46:0/3471713382 conn(0x7f9f200feae0 msgr2=0x7f9f201079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 shutdown_connections 2024-03-08T23:07:36.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.047+0000 7f9f26400640 1 -- 172.21.15.46:0/3471713382 wait complete. 2024-03-08T23:07:36.072 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 -- 172.21.15.46:0/3712433332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104530 msgr2=0x7ffa4c1049b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3712433332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104530 0x7ffa4c1049b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffa34009860 tx=0x7ffa3402bb70 comp rx=0 tx=0).stop 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 -- 172.21.15.46:0/3712433332 shutdown_connections 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3712433332 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c10b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3712433332 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104530 0x7ffa4c1049b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3712433332 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 0x7ffa4c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 -- 172.21.15.46:0/3712433332 >> 172.21.15.46:0/3712433332 conn(0x7ffa4c0fe8d0 msgr2=0x7ffa4c100cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 -- 172.21.15.46:0/3712433332 shutdown_connections 2024-03-08T23:07:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.284+0000 7ffa50e00640 1 -- 172.21.15.46:0/3712433332 wait complete. 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.285+0000 7ffa50e00640 1 Processor -- start 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.285+0000 7ffa50e00640 1 -- start start 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 0x7ffa4c199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c10efd0 con 0x7ffa4c103340 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c10ee50 con 0x7ffa4c104530 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa50e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4c10f150 con 0x7ffa4c104ef0 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa43e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48964/0 (socket says 172.21.15.46:48964) 2024-03-08T23:07:36.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa4b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 0x7ffa4c199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa43e00640 1 -- 172.21.15.46:0/3621999169 learned_addr learned my addr 172.21.15.46:0/3621999169 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.286+0000 7ffa4be00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 -- 172.21.15.46:0/3621999169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 msgr2=0x7ffa4c1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 -- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 msgr2=0x7ffa4c199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 0x7ffa4c199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 -- 172.21.15.46:0/3621999169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4c1a1230 con 0x7ffa4c104530 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa4be00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa43e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005940 tx=0x7ffa34032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34041020 con 0x7ffa4c104530 2024-03-08T23:07:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa3402c9f0 con 0x7ffa4c104530 2024-03-08T23:07:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa4c1a14c0 con 0x7ffa4c104530 2024-03-08T23:07:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34039040 con 0x7ffa4c104530 2024-03-08T23:07:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.287+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa4c1a19b0 con 0x7ffa4c104530 2024-03-08T23:07:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.288+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4c10a6e0 con 0x7ffa4c104530 2024-03-08T23:07:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.290+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffa34044020 con 0x7ffa4c104530 2024-03-08T23:07:36.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.292+0000 7ffa48c00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 0x7ffa300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.292+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7ffa340f0bf0 con 0x7ffa4c104530 2024-03-08T23:07:36.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.292+0000 7ffa4b400640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 0x7ffa300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.293+0000 7ffa4b400640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 0x7ffa300a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c104390 tx=0x7ffa3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.294+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffa3404b020 con 0x7ffa4c104530 2024-03-08T23:07:36.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:36 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2895541923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:36.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:36 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3471713382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:36.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:36 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2895541923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:36.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:36 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3471713382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:36.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.493+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa4c02d0c0 con 0x7ffa4c104530 2024-03-08T23:07:36.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.496+0000 7ffa48c00640 1 -- 172.21.15.46:0/3621999169 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24696 (secure 0 0 0) 0x7ffa340bd8b0 con 0x7ffa4c104530 2024-03-08T23:07:36.500 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:07:36.500 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":93,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:06:41.288000+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.498+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 msgr2=0x7ffa300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.498+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 0x7ffa300a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c104390 tx=0x7ffa3c009260 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.498+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 msgr2=0x7ffa4c199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.498+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ffa34005940 tx=0x7ffa34032470 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 shutdown_connections 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7ffa300a2800 0x7ffa300a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa4c104ef0 0x7ffa4c1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa4c104530 0x7ffa4c199b50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 --2- 172.21.15.46:0/3621999169 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa4c103340 0x7ffa4c199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 >> 172.21.15.46:0/3621999169 conn(0x7ffa4c0fe8d0 msgr2=0x7ffa4c1078b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 shutdown_connections 2024-03-08T23:07:36.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.499+0000 7ffa50e00640 1 -- 172.21.15.46:0/3621999169 wait complete. 2024-03-08T23:07:36.522 DEBUG:tasks.fwd_scrub:Yielding 2024-03-08T23:07:36.523 INFO:teuthology.run_tasks:Running task exec... 2024-03-08T23:07:36.533 INFO:teuthology.task.exec:Executing custom commands... 2024-03-08T23:07:36.534 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:07:36.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-08T23:07:36.534 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-08T23:07:36.534 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:07:36.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:07:36.534 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:07:36.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:36 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2895541923' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:07:36.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:36 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3471713382' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:07:36.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 -- 172.21.15.46:0/3439853288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20075330 msgr2=0x7f0a20073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3439853288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20075330 0x7f0a20073780 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0a10009930 tx=0x7f0a1002bc20 comp rx=0 tx=0).stop 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 -- 172.21.15.46:0/3439853288 shutdown_connections 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3439853288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a2010d730 0x7f0a2010fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3439853288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20073d50 0x7f0a200741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3439853288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a20075330 0x7f0a20073780 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 -- 172.21.15.46:0/3439853288 >> 172.21.15.46:0/3439853288 conn(0x7f0a200fc6d0 msgr2=0x7f0a200feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 -- 172.21.15.46:0/3439853288 shutdown_connections 2024-03-08T23:07:36.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.730+0000 7f0a1f200640 1 -- 172.21.15.46:0/3439853288 wait complete. 2024-03-08T23:07:36.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.732+0000 7f0a1f200640 1 Processor -- start 2024-03-08T23:07:36.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 -- start start 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20075330 0x7f0a2019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2010d730 0x7f0a201a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20113400 con 0x7f0a2010d730 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20113280 con 0x7f0a20075330 2024-03-08T23:07:36.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a20113580 con 0x7f0a20073d50 2024-03-08T23:07:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.733+0000 7f0a1de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44924/0 (socket says 172.21.15.46:44924) 2024-03-08T23:07:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 -- 172.21.15.46:0/3588930251 learned_addr learned my addr 172.21.15.46:0/3588930251 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1d400640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20075330 0x7f0a2019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1e800640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2010d730 0x7f0a201a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 -- 172.21.15.46:0/3588930251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20075330 msgr2=0x7f0a2019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20075330 0x7f0a2019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 -- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2010d730 msgr2=0x7f0a201a5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2010d730 0x7f0a201a5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 -- 172.21.15.46:0/3588930251 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a201a56c0 con 0x7f0a20073d50 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.734+0000 7f0a1de00640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000c00 tx=0x7f0a10005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.735+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10043020 con 0x7f0a20073d50 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.735+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a100054e0 con 0x7f0a20073d50 2024-03-08T23:07:36.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.735+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a201a5950 con 0x7f0a20073d50 2024-03-08T23:07:36.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.735+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a10003d80 con 0x7f0a20073d50 2024-03-08T23:07:36.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.735+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201a5e70 con 0x7f0a20073d50 2024-03-08T23:07:36.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.736+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a20103900 con 0x7f0a20073d50 2024-03-08T23:07:36.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.738+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0a10046020 con 0x7f0a20073d50 2024-03-08T23:07:36.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.739+0000 7f0a16a00640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 0x7f0a040a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.740+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f0a100f1530 con 0x7f0a20073d50 2024-03-08T23:07:36.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.740+0000 7f0a1d400640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 0x7f0a040a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.741+0000 7f0a1d400640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 0x7f0a040a4cb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0a2019efb0 tx=0x7f0a0c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.742+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0a1004d020 con 0x7f0a20073d50 2024-03-08T23:07:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 -- 172.21.15.46:0/3681842076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01011f0 msgr2=0x7f09e0101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 --2- 172.21.15.46:0/3681842076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01011f0 0x7f09e0101610 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009900 tx=0x7f09d002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:36.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 -- 172.21.15.46:0/3681842076 shutdown_connections 2024-03-08T23:07:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 --2- 172.21.15.46:0/3681842076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e0102da0 0x7f09e0109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 --2- 172.21.15.46:0/3681842076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e01023e0 0x7f09e0102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 --2- 172.21.15.46:0/3681842076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01011f0 0x7f09e0101610 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.746+0000 7f09df200640 1 -- 172.21.15.46:0/3681842076 >> 172.21.15.46:0/3681842076 conn(0x7f09e00fc760 msgr2=0x7f09e00feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.747+0000 7f09df200640 1 -- 172.21.15.46:0/3681842076 shutdown_connections 2024-03-08T23:07:36.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.747+0000 7f09df200640 1 -- 172.21.15.46:0/3681842076 wait complete. 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.747+0000 7f09df200640 1 Processor -- start 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 -- start start 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 0x7f09e00736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e0102da0 0x7f09e0073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e010ced0 con 0x7f09e01011f0 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e010cd50 con 0x7f09e01023e0 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09df200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e010d050 con 0x7f09e0102da0 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09dde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09dde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58216/0 (socket says 172.21.15.46:58216) 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09dde00640 1 -- 172.21.15.46:0/916755263 learned_addr learned my addr 172.21.15.46:0/916755263 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09dd400640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 0x7f09e00736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.748+0000 7f09de800640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e0102da0 0x7f09e0073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 -- 172.21.15.46:0/916755263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e0102da0 msgr2=0x7f09e0073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e0102da0 0x7f09e0073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 -- 172.21.15.46:0/916755263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 msgr2=0x7f09e00736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 0x7f09e00736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 -- 172.21.15.46:0/916755263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00744b0 con 0x7f09e01011f0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dd400640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 0x7f09e00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09dde00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09d0000c00 tx=0x7f09d002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.749+0000 7f09d6a00640 1 -- 172.21.15.46:0/916755263 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0043020 con 0x7f09e01011f0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.750+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e01a1650 con 0x7f09e01011f0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.750+0000 7f09d6a00640 1 -- 172.21.15.46:0/916755263 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d00054e0 con 0x7f09e01011f0 2024-03-08T23:07:36.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.750+0000 7f09d6a00640 1 -- 172.21.15.46:0/916755263 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0003c40 con 0x7f09e01011f0 2024-03-08T23:07:36.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.750+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01a1bf0 con 0x7f09e01011f0 2024-03-08T23:07:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 Processor -- start 2024-03-08T23:07:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 -- start start 2024-03-08T23:07:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09d6a00640 1 -- 172.21.15.46:0/916755263 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f09d0046020 con 0x7f09e01011f0 2024-03-08T23:07:36.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb7320 0x5595d3bb97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3b16a50 con 0x5595d3bb7320 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3abfb00 con 0x5595d3bb17c0 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.753+0000 7f09e7b75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3b12af0 con 0x5595d3bb21b0 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09dde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb7320 0x5595d3bb97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49012/0 (socket says 172.21.15.46:49012) 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09dde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44940/0 (socket says 172.21.15.46:44940) 2024-03-08T23:07:36.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 -- 172.21.15.46:0/1204350685 learned_addr learned my addr 172.21.15.46:0/1204350685 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 -- 172.21.15.46:0/1204350685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 msgr2=0x5595d3bb6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 -- 172.21.15.46:0/1204350685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb7320 msgr2=0x5595d3bb97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb7320 0x5595d3bb97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09de800640 1 -- 172.21.15.46:0/1204350685 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595d3add1e0 con 0x5595d3bb17c0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.754+0000 7f09dde00640 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09de800640 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009fd0 tx=0x7f09d80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=497fc2f49fb2ca2a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09cb400640 1 -- 172.21.15.46:0/1204350685 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8004d20 con 0x5595d3bb17c0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09cb400640 1 -- 172.21.15.46:0/1204350685 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d8004050 con 0x5595d3bb17c0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09cb400640 1 -- 172.21.15.46:0/1204350685 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8005220 con 0x5595d3bb17c0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09d6a00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 0x7f09c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09d6a00640 1 -- 172.21.15.46:0/916755263 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f09d00f0d80 con 0x7f09e01011f0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.755+0000 7f09dde00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 0x7f09c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 -- 172.21.15.46:0/1204350685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 msgr2=0x5595d3bb1be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009fd0 tx=0x7f09d80118d0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 -- 172.21.15.46:0/1204350685 shutdown_connections 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09dde00640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 0x7f09c40a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09d00ee870 tx=0x7f09d00b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb7320 0x5595d3bb97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb21b0 0x5595d3bb6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1204350685 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb17c0 0x5595d3bb1be0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 -- 172.21.15.46:0/1204350685 >> 172.21.15.46:0/1204350685 conn(0x5595d3bacd30 msgr2=0x5595d3bad130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 -- 172.21.15.46:0/1204350685 shutdown_connections 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.756+0000 7f09e7b75740 1 -- 172.21.15.46:0/1204350685 wait complete. 2024-03-08T23:07:36.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.757+0000 7f09e7b75740 1 -- start start 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.757+0000 7f09e7b75740 1 Processor -- start 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.757+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bbbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3bc31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3bba2d0 con 0x5595d3bb21b0 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3ad2f60 con 0x5595d3bb7320 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5595d3bba450 con 0x5595d3bb17c0 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09e7b75740 10 client.0 _ll_register_callbacks cb 0x5595d3abeb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:36.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09de800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bbbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09de800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bbbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44954/0 (socket says 172.21.15.46:44954) 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58232/0 (socket says 172.21.15.46:58232) 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09de800640 1 -- 172.21.15.46:0/88377839 learned_addr learned my addr 172.21.15.46:0/88377839 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09de800640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3bc31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 -- 172.21.15.46:0/88377839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 msgr2=0x5595d3bbbc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bbbc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 -- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 msgr2=0x5595d3bc31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3bc31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.758+0000 7f09dd400640 1 -- 172.21.15.46:0/88377839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595d38f37e0 con 0x5595d3bb21b0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09dd400640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f09e0074e10 tx=0x7f09cc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe0a4d6eb4d83c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc012110 con 0x5595d3bb21b0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f09cc004050 con 0x5595d3bb21b0 2024-03-08T23:07:36.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09d8004050 con 0x5595d3bb21b0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09cc025050 con 0x5595d3bb21b0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09e7b75740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.759+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5595d3953f70 con 0x5595d3bb21b0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d8004050 con 0x5595d3bb21b0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f09cc010430 con 0x5595d3bb21b0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 10 client.15411 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 20 client.15411 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3bc7b60 0x5595d3bc9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 4 client.15411 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5595d3b23f90 con 0x5595d3bc7b60 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.760+0000 7f09de800640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3bc7b60 0x5595d3bc9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.761+0000 7f09de800640 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3bc7b60 0x5595d3bc9f80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=422442aeac9bdc62 server_cookie=8d5a7cfbaa1f2fbc in_seq=0 out_seq=0 2024-03-08T23:07:36.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.761+0000 7f09bfe00640 10 client.15411 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09bfe00640 1 -- 172.21.15.46:0/88377839 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5595d3b23f90 con 0x5595d3bc7b60 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09bfe00640 10 client.15411 handle_command_reply: tid=0 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 client.15411 shutdown 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 msgr2=0x5595d3bbc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f09e0074e10 tx=0x7f09cc00f790 comp rx=0 tx=0).stop 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 shutdown_connections 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3bc7b60 0x5595d3bc9f80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3bc31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bbc160 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 --2- 172.21.15.46:0/88377839 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bbbc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09bfe00640 0 client.15411 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 >> 172.21.15.46:0/88377839 conn(0x5595d3bacd30 msgr2=0x5595d3babf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 shutdown_connections 2024-03-08T23:07:36.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 1 -- 172.21.15.46:0/88377839 wait complete. 2024-03-08T23:07:36.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.762+0000 7f09e7b75740 20 client.15411 trim_cache size 0 max 16384 2024-03-08T23:07:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 Processor -- start 2024-03-08T23:07:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 -- start start 2024-03-08T23:07:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 0x5595d3bc3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb7320 0x5595d3babc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01ae240 con 0x5595d3bb17c0 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01abde0 con 0x5595d3bb21b0 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09e7b75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01a9e80 con 0x5595d3bb7320 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.779+0000 7f09de800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58244/0 (socket says 172.21.15.46:58244) 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 -- 172.21.15.46:0/1625409876 learned_addr learned my addr 172.21.15.46:0/1625409876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09dd400640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 0x5595d3bc3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb7320 0x5595d3babc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 -- 172.21.15.46:0/1625409876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb7320 msgr2=0x5595d3babc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb7320 0x5595d3babc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 -- 172.21.15.46:0/1625409876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 msgr2=0x5595d3bc3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 0x5595d3bc3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 -- 172.21.15.46:0/1625409876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5595d37eb170 con 0x5595d3bb17c0 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09dd400640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 0x5595d3bc3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.780+0000 7f09de800640 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09d800a010 tx=0x7f09d80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bce0395756fba5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.781+0000 7f09caa00640 1 -- 172.21.15.46:0/1625409876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8004250 con 0x5595d3bb17c0 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.781+0000 7f09caa00640 1 -- 172.21.15.46:0/1625409876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d8003da0 con 0x5595d3bb17c0 2024-03-08T23:07:36.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.781+0000 7f09caa00640 1 -- 172.21.15.46:0/1625409876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d800fc30 con 0x5595d3bb17c0 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- 172.21.15.46:0/1625409876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 msgr2=0x5595d3bab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09d800a010 tx=0x7f09d80118d0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- 172.21.15.46:0/1625409876 shutdown_connections 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb7320 0x5595d3babc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb21b0 0x5595d3bc3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 --2- 172.21.15.46:0/1625409876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb17c0 0x5595d3bab6e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- 172.21.15.46:0/1625409876 >> 172.21.15.46:0/1625409876 conn(0x5595d3bacd30 msgr2=0x5595d3bb3630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- 172.21.15.46:0/1625409876 shutdown_connections 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- 172.21.15.46:0/1625409876 wait complete. 2024-03-08T23:07:36.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.782+0000 7f09e7b75740 1 -- start start 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 Processor -- start 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bc2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3c1e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01aa480 con 0x5595d3bb21b0 2024-03-08T23:07:36.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01ab780 con 0x5595d3bb7320 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e01aaac0 con 0x5595d3bb17c0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.783+0000 7f09e7b75740 10 client.0 _ll_register_callbacks cb 0x7f09d8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58254/0 (socket says 172.21.15.46:58254) 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 -- 172.21.15.46:0/3216749654 learned_addr learned my addr 172.21.15.46:0/3216749654 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09de800640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bc2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09de800640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3c1e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 -- 172.21.15.46:0/3216749654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 msgr2=0x5595d3bc2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bc2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 -- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 msgr2=0x5595d3c1e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3c1e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 -- 172.21.15.46:0/3216749654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595d3ae2050 con 0x5595d3bb21b0 2024-03-08T23:07:36.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09de800640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3c1e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09dd400640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f09e0074e10 tx=0x7f09cc00fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=342e32ecb96ab855 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.784+0000 7f09bfe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09cc010430 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f09cc018040 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f09d8003da0 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09cc0105d0 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09e7b75740 10 client.15417 fetch_fsmap learned FSMap version 24 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5595d3c1fe70 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09cc038070 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.785+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f09cc0036f0 con 0x5595d3bb21b0 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 10 client.15417 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 20 client.15417 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:36.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3c20880 0x5595d3c22ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 4 client.15417 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:07:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5595d3bbba10 con 0x5595d3c20880 2024-03-08T23:07:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09de800640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3c20880 0x5595d3c22ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09de800640 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3c20880 0x5595d3c22ca0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e68467446321643e server_cookie=42c6fc24a08cdb25 in_seq=0 out_seq=0 2024-03-08T23:07:36.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.786+0000 7f09bfe00640 10 client.15417 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.787+0000 7f09bfe00640 1 -- 172.21.15.46:0/3216749654 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5595d3bbba10 con 0x5595d3c20880 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.787+0000 7f09bfe00640 10 client.15417 handle_command_reply: tid=0 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.787+0000 7f09e7b75740 1 client.15417 shutdown 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 msgr2=0x5595d3bc2db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f09e0074e10 tx=0x7f09cc00fc90 comp rx=0 tx=0).stop 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 shutdown_connections 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5595d3c20880 0x5595d3c22ca0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5595d3bb7320 0x5595d3c1e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5595d3bb21b0 0x5595d3bc2db0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 --2- 172.21.15.46:0/3216749654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5595d3bb17c0 0x5595d3bc2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09bfe00640 0 client.15417 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 >> 172.21.15.46:0/3216749654 conn(0x5595d3bacd30 msgr2=0x5595d3baff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 shutdown_connections 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 1 -- 172.21.15.46:0/3216749654 wait complete. 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.788+0000 7f09e7b75740 20 client.15417 trim_cache size 0 max 16384 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:07:36.795 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "9f74b3c8-d7c2-4b3a-b3b0-2ae3c6d3cdb1", 2024-03-08T23:07:36.796 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:07:36.796 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:07:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 msgr2=0x7f09c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 0x7f09c40a4cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f09d00ee870 tx=0x7f09d00b9040 comp rx=0 tx=0).stop 2024-03-08T23:07:36.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 msgr2=0x7f09e00750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f09d0000c00 tx=0x7f09d002c650 comp rx=0 tx=0).stop 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 shutdown_connections 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f09c40a2800 0x7f09c40a4cc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f09e0102da0 0x7f09e0073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f09e01023e0 0x7f09e00736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 --2- 172.21.15.46:0/916755263 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f09e01011f0 0x7f09e00750a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 >> 172.21.15.46:0/916755263 conn(0x7f09e00fc760 msgr2=0x7f09e0105730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 shutdown_connections 2024-03-08T23:07:36.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:36.789+0000 7f09df200640 1 -- 172.21.15.46:0/916755263 wait complete. 2024-03-08T23:07:36.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:07:36.812 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:07:37.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.011+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f0a2002ce70 con 0x7f0a20073d50 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 -- 172.21.15.46:0/1380726025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6498102da0 msgr2=0x7f6498109630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 --2- 172.21.15.46:0/1380726025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6498102da0 0x7f6498109630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6490009be0 tx=0x7f649002bef0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 -- 172.21.15.46:0/1380726025 shutdown_connections 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 --2- 172.21.15.46:0/1380726025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6498102da0 0x7f6498109630 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 --2- 172.21.15.46:0/1380726025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64981023e0 0x7f6498102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 --2- 172.21.15.46:0/1380726025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f6498101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 -- 172.21.15.46:0/1380726025 >> 172.21.15.46:0/1380726025 conn(0x7f64980fc760 msgr2=0x7f64980feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 -- 172.21.15.46:0/1380726025 shutdown_connections 2024-03-08T23:07:37.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.012+0000 7f649ca00640 1 -- 172.21.15.46:0/1380726025 wait complete. 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.013+0000 7f649ca00640 1 Processor -- start 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.013+0000 7f649ca00640 1 -- start start 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 0x7f649819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6498102da0 0x7f64981a5060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649810cea0 con 0x7f64981011f0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649810cd20 con 0x7f64981023e0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f649ca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649810d020 con 0x7f6498102da0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58276/0 (socket says 172.21.15.46:58276) 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497400640 1 -- 172.21.15.46:0/1362929103 learned_addr learned my addr 172.21.15.46:0/1362929103 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6496a00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 0x7f649819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497e00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6498102da0 0x7f64981a5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497400640 1 -- 172.21.15.46:0/1362929103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6498102da0 msgr2=0x7f64981a5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.014+0000 7f6497400640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6498102da0 0x7f64981a5060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f6497400640 1 -- 172.21.15.46:0/1362929103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 msgr2=0x7f649819dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f6497400640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 0x7f649819dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f6497400640 1 -- 172.21.15.46:0/1362929103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64981a56d0 con 0x7f64981011f0 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f6496a00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 0x7f649819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:37.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f6497400640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f648800e9f0 tx=0x7f648800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f648fe00640 1 -- 172.21.15.46:0/1362929103 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800ccc0 con 0x7f64981011f0 2024-03-08T23:07:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f648fe00640 1 -- 172.21.15.46:0/1362929103 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64880035f0 con 0x7f64981011f0 2024-03-08T23:07:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64981a59c0 con 0x7f64981011f0 2024-03-08T23:07:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f648fe00640 1 -- 172.21.15.46:0/1362929103 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648800ce60 con 0x7f64981011f0 2024-03-08T23:07:37.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.015+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64981a5f10 con 0x7f64981011f0 2024-03-08T23:07:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f648fe00640 1 -- 172.21.15.46:0/1362929103 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f64880037a0 con 0x7f64981011f0 2024-03-08T23:07:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 Processor -- start 2024-03-08T23:07:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 -- start start 2024-03-08T23:07:37.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e47b0 0x5654b43e4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 0x5654b43e9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b43a7410 con 0x5654b43ea310 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b4349d50 con 0x5654b43e47b0 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.019+0000 7f64a0c60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b42f2b80 con 0x5654b43e51a0 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6497400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 0x5654b43e9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58282/0 (socket says 172.21.15.46:58282) 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 -- 172.21.15.46:0/789313181 learned_addr learned my addr 172.21.15.46:0/789313181 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6497e00640 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e47b0 0x5654b43e4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 -- 172.21.15.46:0/789313181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 msgr2=0x5654b43e9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 0x5654b43e9ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 -- 172.21.15.46:0/789313181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e47b0 msgr2=0x5654b43e4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e47b0 0x5654b43e4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6496a00640 1 -- 172.21.15.46:0/789313181 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654b41287d0 con 0x5654b43ea310 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.020+0000 7f6497400640 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 0x5654b43e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f6496a00640 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f64840114f0 tx=0x7f64840119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376dd92e3be44f19 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f648fe00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f648fe00640 1 -- 172.21.15.46:0/1362929103 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f64880cf0a0 con 0x7f64981011f0 2024-03-08T23:07:37.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f6483e00640 1 -- 172.21.15.46:0/789313181 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648401e070 con 0x5654b43ea310 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f6497e00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f6483e00640 1 -- 172.21.15.46:0/789313181 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64840035f0 con 0x5654b43ea310 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.021+0000 7f6483e00640 1 -- 172.21.15.46:0/789313181 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400f820 con 0x5654b43ea310 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 -- 172.21.15.46:0/789313181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 msgr2=0x5654b43ec7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f6497e00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 0x7f647c0a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6490006010 tx=0x7f6490004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f64840114f0 tx=0x7f64840119b0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 -- 172.21.15.46:0/789313181 shutdown_connections 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43ea310 0x5654b43ec7a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e51a0 0x5654b43e9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 --2- 172.21.15.46:0/789313181 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e47b0 0x5654b43e4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 -- 172.21.15.46:0/789313181 >> 172.21.15.46:0/789313181 conn(0x5654b43dfd20 msgr2=0x5654b43e0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 -- 172.21.15.46:0/789313181 shutdown_connections 2024-03-08T23:07:37.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.022+0000 7f64a0c60740 1 -- 172.21.15.46:0/789313181 wait complete. 2024-03-08T23:07:37.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 -- start start 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 Processor -- start 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e51a0 0x5654b43ef050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 0x5654b43f6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b4305ee0 con 0x5654b43e47b0 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b4345df0 con 0x5654b43e51a0 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654b43ed2c0 con 0x5654b43ea310 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.023+0000 7f64a0c60740 10 client.0 _ll_register_callbacks cb 0x5654b430c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:37.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58284/0 (socket says 172.21.15.46:58284) 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 -- 172.21.15.46:0/4086388521 learned_addr learned my addr 172.21.15.46:0/4086388521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6496a00640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e51a0 0x5654b43ef050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497e00640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 0x5654b43f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 -- 172.21.15.46:0/4086388521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 msgr2=0x5654b43f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 0x5654b43f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 -- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e51a0 msgr2=0x5654b43ef050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e51a0 0x5654b43ef050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 -- 172.21.15.46:0/4086388521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654b405e3f0 con 0x5654b43e47b0 2024-03-08T23:07:37.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497e00640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 0x5654b43f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6497400640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6488097ac0 tx=0x7f6488004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1450f6e4cdbee3b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.024+0000 7f6480c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648809afc0 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f64880d2c70 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64840035f0 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f648809bd60 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 10 client.15429 fetch_fsmap learned FSMap version 24 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5654b401e0f0 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64840035f0 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f64880d10d0 con 0x5654b43e47b0 2024-03-08T23:07:37.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 10 client.15429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 20 client.15429 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b43fa920 0x5654b43fcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 4 client.15429 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.025+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5654b4357290 con 0x5654b43fa920 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.026+0000 7f6496a00640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b43fa920 0x5654b43fcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.026+0000 7f6496a00640 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b43fa920 0x5654b43fcd50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dbea563e7bfc422 server_cookie=ee102a61965feb03 in_seq=0 out_seq=0 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.026+0000 7f6480c00640 10 client.15429 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f6480c00640 1 -- 172.21.15.46:0/4086388521 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5654b4357290 con 0x5654b43fa920 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f6480c00640 10 client.15429 handle_command_reply: tid=0 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 client.15429 shutdown 2024-03-08T23:07:37.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 msgr2=0x5654b43eeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6488097ac0 tx=0x7f6488004750 comp rx=0 tx=0).stop 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 shutdown_connections 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b43fa920 0x5654b43fcd50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43ea310 0x5654b43f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43e51a0 0x5654b43ef050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f6480c00640 0 client.15429 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4086388521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e47b0 0x5654b43eeb10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.027+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 >> 172.21.15.46:0/4086388521 conn(0x5654b43dfd20 msgr2=0x5654b43def40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.028+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 shutdown_connections 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.028+0000 7f64a0c60740 1 -- 172.21.15.46:0/4086388521 wait complete. 2024-03-08T23:07:37.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.028+0000 7f64a0c60740 20 client.15429 trim_cache size 0 max 16384 2024-03-08T23:07:37.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.044+0000 7f64a0c60740 1 Processor -- start 2024-03-08T23:07:37.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.044+0000 7f64a0c60740 1 -- start start 2024-03-08T23:07:37.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b43eee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b43de6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981b2690 con 0x5654b43e51a0 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981b0230 con 0x5654b43ea310 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f64a0c60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981ae2d0 con 0x5654b43e47b0 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58288/0 (socket says 172.21.15.46:58288) 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 -- 172.21.15.46:0/4202682272 learned_addr learned my addr 172.21.15.46:0/4202682272 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6497400640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b43eee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6497e00640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b43de6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 -- 172.21.15.46:0/4202682272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 msgr2=0x5654b43eee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b43eee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 -- 172.21.15.46:0/4202682272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 msgr2=0x5654b43de6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b43de6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6496a00640 1 -- 172.21.15.46:0/4202682272 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654b43fa700 con 0x5654b43e51a0 2024-03-08T23:07:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.045+0000 7f6497e00640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b43de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:37.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.046+0000 7f6496a00640 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f648401c040 tx=0x7f6484003780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9f11ee222d98a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.046+0000 7f6483400640 1 -- 172.21.15.46:0/4202682272 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648401e040 con 0x5654b43e51a0 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.046+0000 7f6483400640 1 -- 172.21.15.46:0/4202682272 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f648401f040 con 0x5654b43e51a0 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.046+0000 7f6483400640 1 -- 172.21.15.46:0/4202682272 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840045f0 con 0x5654b43e51a0 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- 172.21.15.46:0/4202682272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 msgr2=0x5654b43f6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f648401c040 tx=0x7f6484003780 comp rx=0 tx=0).stop 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- 172.21.15.46:0/4202682272 shutdown_connections 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b43de6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b43f6050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 --2- 172.21.15.46:0/4202682272 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b43eee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- 172.21.15.46:0/4202682272 >> 172.21.15.46:0/4202682272 conn(0x5654b43dfd20 msgr2=0x5654b43ee720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- 172.21.15.46:0/4202682272 shutdown_connections 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- 172.21.15.46:0/4202682272 wait complete. 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 -- start start 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.047+0000 7f64a0c60740 1 Processor -- start 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b4450a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981ae8d0 con 0x5654b43e51a0 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981afbd0 con 0x5654b43ea310 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981aef10 con 0x5654b43e47b0 2024-03-08T23:07:37.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f64a0c60740 10 client.0 _ll_register_callbacks cb 0x7f6484009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6497400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6496a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:58302/0 (socket says 172.21.15.46:58302) 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6497400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45032/0 (socket says 172.21.15.46:45032) 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6496a00640 1 -- 172.21.15.46:0/2042468055 learned_addr learned my addr 172.21.15.46:0/2042468055 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.048+0000 7f6497e00640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b4450a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 -- 172.21.15.46:0/2042468055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 msgr2=0x5654b44494d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 -- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 msgr2=0x5654b4450a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b4450a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 -- 172.21.15.46:0/2042468055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654b4451040 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6497400640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6496a00640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6484011eb0 tx=0x7f6484003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb074a414b1dc878 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6480c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6484037060 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f64840291e0 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.049+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f648401f040 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f648403e050 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5654b4452350 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f648401f040 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f64840045f0 con 0x5654b43e51a0 2024-03-08T23:07:37.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 10 client.15435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 20 client.15435 populate_metadata read hostname 'smithi046' 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b4452ab0 0x5654b4454ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 4 client.15435 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5654b43515e0 con 0x5654b4452ab0 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.050+0000 7f6497400640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b4452ab0 0x5654b4454ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f6497400640 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b4452ab0 0x5654b4454ed0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde73dc4b9778086 server_cookie=bb3e1b2ee7e2466 in_seq=0 out_seq=0 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f6480c00640 10 client.15435 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f6480c00640 1 -- 172.21.15.46:0/2042468055 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5654b43515e0 con 0x5654b4452ab0 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f6480c00640 10 client.15435 handle_command_reply: tid=0 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 client.15435 shutdown 2024-03-08T23:07:37.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 msgr2=0x5654b4449a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6484011eb0 tx=0x7f6484003680 comp rx=0 tx=0).stop 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 shutdown_connections 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5654b4452ab0 0x5654b4454ed0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5654b43ea310 0x5654b4450a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5654b43e51a0 0x5654b4449a10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 --2- 172.21.15.46:0/2042468055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5654b43e47b0 0x5654b44494d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f6480c00640 0 client.15435 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.051+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 >> 172.21.15.46:0/2042468055 conn(0x5654b43dfd20 msgr2=0x5654b43e2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 shutdown_connections 2024-03-08T23:07:37.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f64a0c60740 1 -- 172.21.15.46:0/2042468055 wait complete. 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f64a0c60740 20 client.15435 trim_cache size 0 max 16384 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "9f74b3c8-d7c2-4b3a-b3b0-2ae3c6d3cdb1": { 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "9f74b3c8-d7c2-4b3a-b3b0-2ae3c6d3cdb1", 2024-03-08T23:07:37.062 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:07:37.063 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:07:37.063 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:07:37.063 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:07:37.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 msgr2=0x7f647c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 0x7f647c0a4cc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6490006010 tx=0x7f6490004a40 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 msgr2=0x7f649819dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.052+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f648800e9f0 tx=0x7f648800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 shutdown_connections 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f647c0a2800 0x7f647c0a4cc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6498102da0 0x7f64981a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64981023e0 0x7f649819dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 --2- 172.21.15.46:0/1362929103 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64981011f0 0x7f649819dab0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 >> 172.21.15.46:0/1362929103 conn(0x7f64980fc760 msgr2=0x7f6498103600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 shutdown_connections 2024-03-08T23:07:37.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.053+0000 7f649ca00640 1 -- 172.21.15.46:0/1362929103 wait complete. 2024-03-08T23:07:37.073 INFO:tasks.cephfs.filesystem:scrub status for tag:9f74b3c8-d7c2-4b3a-b3b0-2ae3c6d3cdb1 - {'path': '/', 'tag': '9f74b3c8-d7c2-4b3a-b3b0-2ae3c6d3cdb1', 'options': 'recursive,force'} 2024-03-08T23:07:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.123+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f0a100ba160 con 0x7f0a20073d50 2024-03-08T23:07:37.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.124+0000 7f0a16a00640 1 -- 172.21.15.46:0/3588930251 <== mon.1 v2:172.21.15.192:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f0a100f2380 con 0x7f0a20073d50 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 msgr2=0x7f0a040a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 0x7f0a040a4cb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0a2019efb0 tx=0x7f0a0c00b390 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 msgr2=0x7f0a2019daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000c00 tx=0x7f0a10005dc0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 shutdown_connections 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f0a040a27f0 0x7f0a040a4cb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0a2010d730 0x7f0a201a5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0a20075330 0x7f0a2019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 --2- 172.21.15.46:0/3588930251 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0a20073d50 0x7f0a2019daa0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 >> 172.21.15.46:0/3588930251 conn(0x7f0a200fc6d0 msgr2=0x7f0a20100ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 shutdown_connections 2024-03-08T23:07:37.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.128+0000 7f0a1f200640 1 -- 172.21.15.46:0/3588930251 wait complete. 2024-03-08T23:07:37.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-08T23:07:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 -- 172.21.15.46:0/189152328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74104740 msgr2=0x7fab74104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 --2- 172.21.15.46:0/189152328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74104740 0x7fab74104bc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fab5c009900 tx=0x7fab5c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 -- 172.21.15.46:0/189152328 shutdown_connections 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 --2- 172.21.15.46:0/189152328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74105100 0x7fab7410b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 --2- 172.21.15.46:0/189152328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74104740 0x7fab74104bc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 --2- 172.21.15.46:0/189152328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74103550 0x7fab74103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 -- 172.21.15.46:0/189152328 >> 172.21.15.46:0/189152328 conn(0x7fab740feae0 msgr2=0x7fab74100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 -- 172.21.15.46:0/189152328 shutdown_connections 2024-03-08T23:07:37.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.348+0000 7fab79800640 1 -- 172.21.15.46:0/189152328 wait complete. 2024-03-08T23:07:37.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.349+0000 7fab79800640 1 Processor -- start 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 -- start start 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74104740 0x7fab74199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74105100 0x7fab741a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7410f230 con 0x7fab74104740 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7410f0b0 con 0x7fab74105100 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab79800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7410f3b0 con 0x7fab74103550 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45050/0 (socket says 172.21.15.46:45050) 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab72a00640 1 -- 172.21.15.46:0/448284365 learned_addr learned my addr 172.21.15.46:0/448284365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab6be00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74104740 0x7fab74199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.350+0000 7fab73400640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74105100 0x7fab741a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74105100 msgr2=0x7fab741a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74105100 0x7fab741a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74104740 msgr2=0x7fab74199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74104740 0x7fab74199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 -- 172.21.15.46:0/448284365 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab741a1330 con 0x7fab74103550 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab72a00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9c0 tx=0x7fab6400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400cd80 con 0x7fab74103550 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab640035f0 con 0x7fab74103550 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64010690 con 0x7fab74103550 2024-03-08T23:07:37.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.351+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab741a1620 con 0x7fab74103550 2024-03-08T23:07:37.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.352+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab741a1af0 con 0x7fab74103550 2024-03-08T23:07:37.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.353+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7410a7a0 con 0x7fab74103550 2024-03-08T23:07:37.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.355+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fab640037a0 con 0x7fab74103550 2024-03-08T23:07:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.356+0000 7fab6b400640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 0x7fab580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.356+0000 7fab6be00640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 msgr2=0x7fab580a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.356+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fab640cf1d0 con 0x7fab74103550 2024-03-08T23:07:37.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.356+0000 7fab6be00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 0x7fab580a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:07:37.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.360+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fab6409be90 con 0x7fab74103550 2024-03-08T23:07:37.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: pgmap v411: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:37.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3621999169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:37.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:07:37.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3588930251' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: ignoring --setuser ceph since I am not root 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: ignoring --setgroup ceph since I am not root 2024-03-08T23:07:37.392 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-mgr[37159]: -- 172.21.15.62:0/1908009373 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558acd27a5a0 con 0x558acd24ec00 2024-03-08T23:07:37.475 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setuser ceph since I am not root 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: ignoring --setgroup ceph since I am not root 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: pgmap v411: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3621999169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3588930251' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.557+0000 7fab6be00640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 msgr2=0x7fab580a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:37.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.557+0000 7fab6be00640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 0x7fab580a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: ignoring --setuser ceph since I am not root 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: ignoring --setgroup ceph since I am not root 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mgr[38565]: -- 172.21.15.192:0/3170628025 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559cf5d3e5a0 con 0x559cf5d12c00 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: pgmap v411: 129 pgs: 129 active+clean; 648 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 767 B/s wr, 7 op/s 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3621999169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:07:37.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:07:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:07:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:07:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3588930251' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-08T23:07:37.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.567+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fab7402ce70 con 0x7fab74103550 2024-03-08T23:07:37.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.577+0000 7fab6b400640 1 -- 172.21.15.46:0/448284365 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7fab6409b790 con 0x7fab74103550 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 msgr2=0x7fab580a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 0x7fab580a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 msgr2=0x7fab74199710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e9c0 tx=0x7fab6400ee80 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 shutdown_connections 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fab580a2970 0x7fab580a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab74105100 0x7fab741a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab74104740 0x7fab74199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 --2- 172.21.15.46:0/448284365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab74103550 0x7fab74199710 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 >> 172.21.15.46:0/448284365 conn(0x7fab740feae0 msgr2=0x7fab74107970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 shutdown_connections 2024-03-08T23:07:37.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.580+0000 7fab79800640 1 -- 172.21.15.46:0/448284365 wait complete. 2024-03-08T23:07:37.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-08T23:07:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 -- 172.21.15.46:0/2294801018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 msgr2=0x7f8e701041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 --2- 172.21.15.46:0/2294801018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e701041a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009900 tx=0x7f8e6002bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 -- 172.21.15.46:0/2294801018 shutdown_connections 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 --2- 172.21.15.46:0/2294801018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e70109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 --2- 172.21.15.46:0/2294801018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e70106ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 --2- 172.21.15.46:0/2294801018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e701041a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 -- 172.21.15.46:0/2294801018 >> 172.21.15.46:0/2294801018 conn(0x7f8e700fb7a0 msgr2=0x7f8e700fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 -- 172.21.15.46:0/2294801018 shutdown_connections 2024-03-08T23:07:37.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.793+0000 7f8e75800640 1 -- 172.21.15.46:0/2294801018 wait complete. 2024-03-08T23:07:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.794+0000 7f8e75800640 1 Processor -- start 2024-03-08T23:07:37.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 -- start start 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e700ffd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e701002a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010d010 con 0x7f8e70107010 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010ce90 con 0x7f8e70101db0 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e75800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7010d190 con 0x7f8e701046e0 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e6ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e6ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:49134/0 (socket says 172.21.15.46:49134) 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e6ea00640 1 -- 172.21.15.46:0/146393114 learned_addr learned my addr 172.21.15.46:0/146393114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e6e000640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e700ffd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.795+0000 7f8e6f400640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e701002a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 msgr2=0x7f8e700ffd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e700ffd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 msgr2=0x7f8e701002a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e701002a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 -- 172.21.15.46:0/146393114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70100b30 con 0x7f8e70101db0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6e000640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e700ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6f400640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e701002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e6ea00640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8e60000c00 tx=0x7f8e6002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60043020 con 0x7f8e70101db0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e600054e0 con 0x7f8e70101db0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e701a16b0 con 0x7f8e70101db0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e60003c40 con 0x7f8e70101db0 2024-03-08T23:07:37.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.796+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e701a1c50 con 0x7f8e70101db0 2024-03-08T23:07:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.798+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e700fd290 con 0x7f8e70101db0 2024-03-08T23:07:37.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.799+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f8e60046020 con 0x7f8e70101db0 2024-03-08T23:07:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.801+0000 7f8e67400640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:37.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.801+0000 7f8e6e000640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 msgr2=0x7f8e540a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.801+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7f8e600f0fe0 con 0x7f8e70101db0 2024-03-08T23:07:37.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.801+0000 7f8e6e000640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:07:37.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:37.804+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8e6004d020 con 0x7f8e70101db0 2024-03-08T23:07:38.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.002+0000 7f8e6e000640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 msgr2=0x7f8e540a4e30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:38.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.002+0000 7f8e6e000640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:07:38.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.004+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f8e70061ef0 con 0x7f8e70101db0 2024-03-08T23:07:38.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.011+0000 7f8e67400640 1 -- 172.21.15.46:0/146393114 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f8e6003c350 con 0x7f8e70101db0 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.013+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 msgr2=0x7f8e540a4e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.013+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.013+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 msgr2=0x7f8e70101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.013+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8e60000c00 tx=0x7f8e6002c650 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 shutdown_connections 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7f8e540a2970 0x7f8e540a4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8e70107010 0x7f8e701002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8e701046e0 0x7f8e700ffd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 --2- 172.21.15.46:0/146393114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8e70101db0 0x7f8e70101720 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 >> 172.21.15.46:0/146393114 conn(0x7f8e700fb7a0 msgr2=0x7f8e701026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:38.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 shutdown_connections 2024-03-08T23:07:38.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.014+0000 7f8e75800640 1 -- 172.21.15.46:0/146393114 wait complete. 2024-03-08T23:07:38.036 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-08T23:07:38.063 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:37.682+0000 7f4a58473200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:07:38.063 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:37.774+0000 7f4a58473200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:07:38.114 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:37.706+0000 7f3d001a1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:07:38.114 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:37.803+0000 7f3d001a1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.234+0000 7fb91de00640 1 -- 172.21.15.46:0/2824765038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9180ff9b0 msgr2=0x7fb9180ffdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 --2- 172.21.15.46:0/2824765038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9180ff9b0 0x7fb9180ffdd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb908009900 tx=0x7fb90802bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 -- 172.21.15.46:0/2824765038 shutdown_connections 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 --2- 172.21.15.46:0/2824765038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb918100cb0 0x7fb918101130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 --2- 172.21.15.46:0/2824765038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb918100310 0x7fb918100770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 --2- 172.21.15.46:0/2824765038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb9180ff9b0 0x7fb9180ffdd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 -- 172.21.15.46:0/2824765038 >> 172.21.15.46:0/2824765038 conn(0x7fb9180fb5d0 msgr2=0x7fb9180fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 -- 172.21.15.46:0/2824765038 shutdown_connections 2024-03-08T23:07:38.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.235+0000 7fb91de00640 1 -- 172.21.15.46:0/2824765038 wait complete. 2024-03-08T23:07:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 Processor -- start 2024-03-08T23:07:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 -- start start 2024-03-08T23:07:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 0x7fb9181995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb918100310 0x7fb918199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:38.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb918100cb0 0x7fb9181a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91810f030 con 0x7fb918100cb0 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91810eeb0 con 0x7fb918100310 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91de00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb91810f1b0 con 0x7fb9180ff9b0 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91ca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 0x7fb9181995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91ca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 0x7fb9181995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45076/0 (socket says 172.21.15.46:45076) 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb917e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb918100310 0x7fb918199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91ca00640 1 -- 172.21.15.46:0/278364021 learned_addr learned my addr 172.21.15.46:0/278364021 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.236+0000 7fb91d400640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb918100cb0 0x7fb9181a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb918100310 msgr2=0x7fb918199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb918100310 0x7fb918199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb918100cb0 msgr2=0x7fb9181a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:38.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb918100cb0 0x7fb9181a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9181a11f0 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91ca00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 0x7fb9181995d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb908000c00 tx=0x7fb90802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908043020 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb9080054e0 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9181a1480 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.237+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb908003c40 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.238+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9181a19d0 con 0x7fb9180ff9b0 2024-03-08T23:07:38.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.239+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91810a810 con 0x7fb9180ff9b0 2024-03-08T23:07:38.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.241+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fb908046020 con 0x7fb9180ff9b0 2024-03-08T23:07:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.242+0000 7fb915600640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.242+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.242+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9129+0+0 (secure 0 0 0) 0x7fb9080f12a0 con 0x7fb9180ff9b0 2024-03-08T23:07:38.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.242+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:07:38.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.244+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb90804d020 con 0x7fb9180ff9b0 2024-03-08T23:07:38.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-08T23:07:38.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-mon[35761]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-08T23:07:38.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-08T23:07:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-mon[36700]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-08T23:07:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.443+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:38.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.443+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:07:38.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.466+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb918061730 con 0x7fb8fc0a2840 2024-03-08T23:07:38.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-08T23:07:38.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-mon[31859]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-08T23:07:38.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' 2024-03-08T23:07:38.475 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:37.973+0000 7f24544e3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:07:38.475 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:38.105+0000 7f24544e3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:07:38.747 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:38.415+0000 7f4a58473200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:07:38.747 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:38.508+0000 7f4a58473200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:07:38.747 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:38.649+0000 7f4a58473200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:07:38.747 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:38.743+0000 7f4a58473200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:07:38.812 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:38.469+0000 7f3d001a1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:07:38.812 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:38.565+0000 7f3d001a1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:07:38.812 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:38.710+0000 7f3d001a1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:07:38.812 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:38.809+0000 7f3d001a1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:07:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.844+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:38.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:38.844+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-08T23:07:39.063 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:38.826+0000 7f4a58473200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:07:39.141 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:38.895+0000 7f3d001a1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:07:39.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:38 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:38.931+0000 7f24544e3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:07:39.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.057+0000 7f24544e3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:07:39.379 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:39.088+0000 7f4a58473200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:07:39.379 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:39.197+0000 7f4a58473200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:39.380 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:39.282+0000 7f4a58473200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:07:39.380 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:39.376+0000 7f4a58473200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:07:39.471 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:39.166+0000 7f3d001a1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:07:39.472 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:39.279+0000 7f3d001a1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:39.472 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:39.368+0000 7f3d001a1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:07:39.472 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:39.468+0000 7f3d001a1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:07:39.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:39.645+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:39.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:39.645+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-08T23:07:39.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.242+0000 7f24544e3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:07:39.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.372+0000 7f24544e3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:07:39.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.489+0000 7f24544e3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:07:39.813 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:39.555+0000 7f4a58473200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:07:39.891 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:39.649+0000 7f3d001a1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:07:40.108 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.836+0000 7f24544e3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:07:40.108 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:39.984+0000 7f24544e3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:40.108 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:40 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:40.103+0000 7f24544e3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:07:40.463 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:40 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:40.232+0000 7f24544e3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:07:40.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:40 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:40.458+0000 7f24544e3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:07:40.813 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:40.516+0000 7f4a58473200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:07:40.813 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:40.644+0000 7f4a58473200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:07:41.141 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:40.665+0000 7f3d001a1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:07:41.141 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:40.800+0000 7f3d001a1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:07:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:41.247+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:41.247+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-08T23:07:41.263 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:40.977+0000 7f4a58473200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:07:41.263 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:41.066+0000 7f4a58473200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:07:41.441 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:41.143+0000 7f3d001a1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:07:41.442 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:41.237+0000 7f3d001a1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:07:41.563 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:41.260+0000 7f4a58473200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:41.563 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:41.442+0000 7f4a58473200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:07:41.891 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:41.438+0000 7f3d001a1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:41.891 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:41.630+0000 7f3d001a1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:07:41.991 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:41.726+0000 7f4a58473200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:42.195 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:41.925+0000 7f3d001a1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:42.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:41 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:41.792+0000 7f24544e3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:07:42.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:41 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:41.972+0000 7f24544e3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:07:42.313 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:41.987+0000 7f4a58473200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:42.313 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:42.221+0000 7f4a58473200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:07:42.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:42 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:42.192+0000 7f3d001a1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:42.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:42 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:42.438+0000 7f3d001a1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:07:42.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:42.409+0000 7f24544e3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:07:42.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:42.535+0000 7f24544e3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:07:43.068 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:42.805+0000 7f24544e3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:43.313 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:43.018+0000 7f4a58473200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:07:43.470 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:43.064+0000 7f24544e3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:07:43.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:43.274+0000 7f3d001a1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:07:43.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:43.465+0000 7f24544e3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:07:43.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:43.590+0000 7f4a58473200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:07:43.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:43.688+0000 7f4a58473200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:07:43.907 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z[38561]: 2024-03-08T23:07:43.903+0000 7f4a58473200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:07:43.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:43.987+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mgrmap(e 20) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fb9080b9ed0 con 0x7fb9180ff9b0 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:43.794+0000 7f24544e3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: Standby manager daemon z restarted 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: Standby manager daemon z started 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:44.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:07:44.124 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:43 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:43.867+0000 7f3d001a1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:43.969+0000 7f3d001a1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: Standby manager daemon z restarted 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: Standby manager daemon z started 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:07:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:43 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: Standby manager daemon z restarted 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: Standby manager daemon z started 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:07:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:43 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.192:0/217947284' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:44.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:44.451+0000 7fb917e00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.46:6800/757291496 2024-03-08T23:07:44.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:44.451+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-08T23:07:44.474 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:44.118+0000 7f24544e3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:07:44.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y[37155]: 2024-03-08T23:07:44.196+0000 7f3d001a1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:07:45.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:45.022+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 8 ==== mgrmap(e 21) v1 ==== 137307+0+0 (secure 0 0 0) 0x7fb9080eea40 con 0x7fb9180ff9b0 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: Standby manager daemon y restarted 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: Standby manager daemon y started 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:07:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:44 smithi192 ceph-mon[36700]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:45.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-08T23:07:45.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: Standby manager daemon y restarted 2024-03-08T23:07:45.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: Standby manager daemon y started 2024-03-08T23:07:45.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:07:45.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:45.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:07:45.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:44 smithi062 ceph-mon[35761]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:45 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:45.217+0000 7f24544e3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: mgrmap e20: x(active, since 11m), standbys: y, z 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: Standby manager daemon y restarted 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: Standby manager daemon y started 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:07:45.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:44 smithi046 ceph-mon[31859]: from='mgr.? 172.21.15.62:0/1128712452' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:07:46.115 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-mon[31859]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-08T23:07:46.115 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:45 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:45.964+0000 7f24544e3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:07:46.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:46 smithi192 ceph-mon[36700]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-08T23:07:46.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:46 smithi062 ceph-mon[35761]: mgrmap e21: x(active, since 11m), standbys: y, z 2024-03-08T23:07:46.415 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.110+0000 7f24544e3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:07:46.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:46.440+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 9 ==== mgrmap(e 22) v1 ==== 136975+0+0 (secure 0 0 0) 0x7fb908003c20 con 0x7fb9180ff9b0 2024-03-08T23:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:46.441+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 msgr2=0x7fb8fc0a4d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:07:46.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:46.441+0000 7fb915600640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/757291496,v1:172.21.15.46:6801/757291496] conn(0x7fb8fc0a2840 0x7fb8fc0a4d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:46.725 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.411+0000 7f24544e3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:07:47.016 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a8c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.016 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a8c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.016 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a8c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a8c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a8c00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:07:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:07:46.756+0000 7f23a5a00700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: Active manager daemon x restarted 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: Activating manager daemon x 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: osdmap e94: 12 total, 12 up, 12 in 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: mgrmap e22: x(active, starting, since 0.0142535s), standbys: y, z 2024-03-08T23:07:47.017 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:07:47.018 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:07:47.018 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:07:47.018 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:07:47.018 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:07:47.018 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:07:47.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: Active manager daemon x restarted 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: Activating manager daemon x 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: osdmap e94: 12 total, 12 up, 12 in 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: mgrmap e22: x(active, starting, since 0.0142535s), standbys: y, z 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:07:47.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:07:47.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: Manager daemon x is now available 2024-03-08T23:07:47.316 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.317 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:07:47.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: Active manager daemon x restarted 2024-03-08T23:07:47.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: Activating manager daemon x 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: osdmap e94: 12 total, 12 up, 12 in 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: mgrmap e22: x(active, starting, since 0.0142535s), standbys: y, z 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:07:47.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:07:47.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: Manager daemon x is now available 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.394 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.395 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:07:47.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.447+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mon.1 v2:172.21.15.192:3300/0 10 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fb9080ba7a0 con 0x7fb9180ff9b0 2024-03-08T23:07:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.448+0000 7fb915600640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb8fc04d0c0 0x7fb8fc04d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:47.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.449+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9080de360 con 0x7fb8fc04d0c0 2024-03-08T23:07:47.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.454+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb8fc04d0c0 0x7fb8fc04d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:47.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.460+0000 7fb917e00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb8fc04d0c0 0x7fb8fc04d4a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb91819aae0 tx=0x7fb900005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:47.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:07:47.477 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:07:47.478 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: Manager daemon x is now available 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:07:47.479 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:07:47.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.539+0000 7fb915600640 1 -- 172.21.15.46:0/278364021 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb9080de360 con 0x7fb8fc04d0c0 2024-03-08T23:07:47.543 INFO:teuthology.orchestra.run.smithi046.stdout:Schedule set for path / 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb8fc04d0c0 msgr2=0x7fb8fc04d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb8fc04d0c0 0x7fb8fc04d4a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb91819aae0 tx=0x7fb900005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 msgr2=0x7fb9181995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 --2- 172.21.15.46:0/278364021 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb9180ff9b0 0x7fb9181995d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb908000c00 tx=0x7fb90802c650 comp rx=0 tx=0).stop 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91d400640 1 -- 172.21.15.46:0/278364021 reap_dead start 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 shutdown_connections 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 >> 172.21.15.46:0/278364021 conn(0x7fb9180fb5d0 msgr2=0x7fb9180fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 shutdown_connections 2024-03-08T23:07:47.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.542+0000 7fb91de00640 1 -- 172.21.15.46:0/278364021 wait complete. 2024-03-08T23:07:47.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-08T23:07:47.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 -- 172.21.15.46:0/2146223059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 msgr2=0x7ffb3c109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 --2- 172.21.15.46:0/2146223059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c109770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009be0 tx=0x7ffb3402bef0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 -- 172.21.15.46:0/2146223059 shutdown_connections 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 --2- 172.21.15.46:0/2146223059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c109770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 --2- 172.21.15.46:0/2146223059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c102520 0x7ffb3c1029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 --2- 172.21.15.46:0/2146223059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c101330 0x7ffb3c101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 -- 172.21.15.46:0/2146223059 >> 172.21.15.46:0/2146223059 conn(0x7ffb3c0fc8a0 msgr2=0x7ffb3c0fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.759+0000 7ffb41a00640 1 -- 172.21.15.46:0/2146223059 shutdown_connections 2024-03-08T23:07:47.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.760+0000 7ffb41a00640 1 -- 172.21.15.46:0/2146223059 wait complete. 2024-03-08T23:07:47.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.760+0000 7ffb41a00640 1 Processor -- start 2024-03-08T23:07:47.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.760+0000 7ffb41a00640 1 -- start start 2024-03-08T23:07:47.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c101330 0x7ffb3c1997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c10cfc0 con 0x7ffb3c102520 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c10ce40 con 0x7ffb3c101330 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb41a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3c10d140 con 0x7ffb3c102ee0 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb3a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:47.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb3aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c101330 0x7ffb3c1997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb3a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56676/0 (socket says 172.21.15.46:56676) 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb3a000640 1 -- 172.21.15.46:0/3669765133 learned_addr learned my addr 172.21.15.46:0/3669765133 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.761+0000 7ffb3b400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 -- 172.21.15.46:0/3669765133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c101330 msgr2=0x7ffb3c1997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c101330 0x7ffb3c1997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 -- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 msgr2=0x7ffb3c199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:47.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 -- 172.21.15.46:0/3669765133 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c1a13d0 con 0x7ffb3c102ee0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3a000640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb3b400640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c1a0d60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb3402c430 tx=0x7ffb340337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34043020 con 0x7ffb3c102ee0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3c1a1660 con 0x7ffb3c102ee0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3c1a1c00 con 0x7ffb3c102ee0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb3402ca60 con 0x7ffb3c102ee0 2024-03-08T23:07:47.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.762+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb34034690 con 0x7ffb3c102ee0 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.763+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c005290 con 0x7ffb3c102ee0 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.766+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7ffb34046020 con 0x7ffb3c102ee0 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.767+0000 7ffb33400640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 0x7ffb200a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.767+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ffb340f09f0 con 0x7ffb3c102ee0 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.767+0000 7ffb3aa00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 0x7ffb200a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:47.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.768+0000 7ffb3aa00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 0x7ffb200a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009530 tx=0x7ffb2c009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:47.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:47.771+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffb3404d030 con 0x7ffb3c102ee0 2024-03-08T23:07:48.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.016+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb1c000c60 con 0x7ffb200a26b0 2024-03-08T23:07:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.123+0000 7ffb33400640 1 -- 172.21.15.46:0/3669765133 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ffb1c000c60 con 0x7ffb200a26b0 2024-03-08T23:07:48.127 INFO:teuthology.orchestra.run.smithi046.stdout:Retention added to path / 2024-03-08T23:07:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 msgr2=0x7ffb200a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 0x7ffb200a4b70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009530 tx=0x7ffb2c009260 comp rx=0 tx=0).stop 2024-03-08T23:07:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 msgr2=0x7ffb3c1a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c1a0d60 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ffb3402c430 tx=0x7ffb340337b0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 shutdown_connections 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb200a26b0 0x7ffb200a4b70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb3c102ee0 0x7ffb3c1a0d60 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb3c102520 0x7ffb3c199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 --2- 172.21.15.46:0/3669765133 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb3c101330 0x7ffb3c1997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 >> 172.21.15.46:0/3669765133 conn(0x7ffb3c0fc8a0 msgr2=0x7ffb3c073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 shutdown_connections 2024-03-08T23:07:48.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.126+0000 7ffb41a00640 1 -- 172.21.15.46:0/3669765133 wait complete. 2024-03-08T23:07:48.149 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 -- 172.21.15.46:0/165353923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68076ef0 msgr2=0x7f0f68077350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/165353923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68076ef0 0x7f0f68077350 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f0f54009900 tx=0x7f0f5402bc30 comp rx=0 tx=0).stop 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 -- 172.21.15.46:0/165353923 shutdown_connections 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/165353923 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68077890 0x7f0f68078130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/165353923 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68076ef0 0x7f0f68077350 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/165353923 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f680760d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 -- 172.21.15.46:0/165353923 >> 172.21.15.46:0/165353923 conn(0x7f0f680feae0 msgr2=0x7f0f68100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 -- 172.21.15.46:0/165353923 shutdown_connections 2024-03-08T23:07:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.351+0000 7f0f6ec00640 1 -- 172.21.15.46:0/165353923 wait complete. 2024-03-08T23:07:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 Processor -- start 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 -- start start 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f6819dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68076ef0 0x7f0f6819e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f68113620 con 0x7f0f68076ef0 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681134a0 con 0x7f0f68077890 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.352+0000 7f0f6ec00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f681137a0 con 0x7f0f68075cb0 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f67400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68076ef0 0x7f0f6819e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43026/0 (socket says 172.21.15.46:43026) 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 -- 172.21.15.46:0/3717285141 learned_addr learned my addr 172.21.15.46:0/3717285141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f67e00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f6819dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 -- 172.21.15.46:0/3717285141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 msgr2=0x7f0f6819dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f6819dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 -- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68076ef0 msgr2=0x7f0f6819e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68076ef0 0x7f0f6819e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 -- 172.21.15.46:0/3717285141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f681a57b0 con 0x7f0f68077890 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f67e00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f6819dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.353+0000 7f0f6cc00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000eb10 tx=0x7f0f6000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.354+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60004b20 con 0x7f0f68077890 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.354+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f681a5aa0 con 0x7f0f68077890 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.354+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f60004cc0 con 0x7f0f68077890 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.354+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f681a5fc0 con 0x7f0f68077890 2024-03-08T23:07:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.354+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f60010690 con 0x7f0f68077890 2024-03-08T23:07:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.355+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f6804f050 con 0x7f0f68077890 2024-03-08T23:07:48.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.357+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 23) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f0f60005000 con 0x7f0f68077890 2024-03-08T23:07:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.359+0000 7f0f64c00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 0x7f0f4c0a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.359+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0f600cf100 con 0x7f0f68077890 2024-03-08T23:07:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.359+0000 7f0f67e00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 0x7f0f4c0a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.360+0000 7f0f67e00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 0x7f0f4c0a4bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009c20 tx=0x7f0f58005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:48.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.362+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0f6009bfd0 con 0x7f0f68077890 2024-03-08T23:07:48.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.457+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mgrmap(e 24) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f0f60097f40 con 0x7f0f68077890 2024-03-08T23:07:48.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.614+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f68061730 con 0x7f0f4c0a2730 2024-03-08T23:07:48.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.630+0000 7f0f64c00640 1 -- 172.21.15.46:0/3717285141 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f0f68061730 con 0x7f0f4c0a2730 2024-03-08T23:07:48.634 INFO:teuthology.orchestra.run.smithi046.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-08T00:00:00", "created": "2024-03-08T23:07:47", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-08T23:07:48.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 msgr2=0x7f0f4c0a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 0x7f0f4c0a4bf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f58009c20 tx=0x7f0f58005f90 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 msgr2=0x7f0f681a5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0f6000eb10 tx=0x7f0f6000c2d0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 shutdown_connections 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0f4c0a2730 0x7f0f4c0a4bf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.632+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0f68077890 0x7f0f681a5140 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.633+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0f68076ef0 0x7f0f6819e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.633+0000 7f0f6ec00640 1 --2- 172.21.15.46:0/3717285141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0f68075cb0 0x7f0f6819dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.633+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 >> 172.21.15.46:0/3717285141 conn(0x7f0f680feae0 msgr2=0x7f0f681005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.633+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 shutdown_connections 2024-03-08T23:07:48.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.633+0000 7f0f6ec00640 1 -- 172.21.15.46:0/3717285141 wait complete. 2024-03-08T23:07:48.654 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-08T23:07:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: mgrmap e23: x(active, since 1.04155s), standbys: y, z 2024-03-08T23:07:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: [08/Mar/2024:23:07:47] ENGINE Bus STARTING 2024-03-08T23:07:48.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: [08/Mar/2024:23:07:47] ENGINE Serving on http://172.21.15.46:8765 2024-03-08T23:07:48.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: [08/Mar/2024:23:07:47] ENGINE Serving on https://172.21.15.46:7150 2024-03-08T23:07:48.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: [08/Mar/2024:23:07:47] ENGINE Bus STARTED 2024-03-08T23:07:48.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:48 smithi046 ceph-mon[31859]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: mgrmap e23: x(active, since 1.04155s), standbys: y, z 2024-03-08T23:07:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: [08/Mar/2024:23:07:47] ENGINE Bus STARTING 2024-03-08T23:07:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: [08/Mar/2024:23:07:47] ENGINE Serving on http://172.21.15.46:8765 2024-03-08T23:07:48.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: [08/Mar/2024:23:07:47] ENGINE Serving on https://172.21.15.46:7150 2024-03-08T23:07:48.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: [08/Mar/2024:23:07:47] ENGINE Bus STARTED 2024-03-08T23:07:48.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:48 smithi192 ceph-mon[36700]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 -- 172.21.15.46:0/3289944413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c103540 msgr2=0x7f8d5c103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 --2- 172.21.15.46:0/3289944413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c103540 0x7f8d5c103960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8d50009860 tx=0x7f8d5002bb70 comp rx=0 tx=0).stop 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 -- 172.21.15.46:0/3289944413 shutdown_connections 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 --2- 172.21.15.46:0/3289944413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c10b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 --2- 172.21.15.46:0/3289944413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c104730 0x7f8d5c104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 --2- 172.21.15.46:0/3289944413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c103540 0x7f8d5c103960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 -- 172.21.15.46:0/3289944413 >> 172.21.15.46:0/3289944413 conn(0x7f8d5c0feab0 msgr2=0x7f8d5c100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 -- 172.21.15.46:0/3289944413 shutdown_connections 2024-03-08T23:07:48.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.868+0000 7f8d63600640 1 -- 172.21.15.46:0/3289944413 wait complete. 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.869+0000 7f8d63600640 1 Processor -- start 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.869+0000 7f8d63600640 1 -- start start 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c1a0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c10f1d0 con 0x7f8d5c1050f0 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c10f050 con 0x7f8d5c103540 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d63600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c10f350 con 0x7f8d5c104730 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d60c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d60c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43048/0 (socket says 172.21.15.46:43048) 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59330/0 (socket says 172.21.15.46:59330) 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d60c00640 1 -- 172.21.15.46:0/2495595366 learned_addr learned my addr 172.21.15.46:0/2495595366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d61600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c1a0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d60c00640 1 -- 172.21.15.46:0/2495595366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 msgr2=0x7f8d5c199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.870+0000 7f8d60c00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d60c00640 1 -- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 msgr2=0x7f8d5c1a0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d60c00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c1a0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d60c00640 1 -- 172.21.15.46:0/2495595366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c1a13a0 con 0x7f8d5c103540 2024-03-08T23:07:48.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d5be00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d61600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c1a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d60c00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8d500349e0 tx=0x7f8d50032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50041020 con 0x7f8d5c103540 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d5002c9f0 con 0x7f8d5c103540 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50039040 con 0x7f8d5c103540 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.871+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c1a1630 con 0x7f8d5c103540 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.872+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c1a1ad0 con 0x7f8d5c103540 2024-03-08T23:07:48.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.873+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c10a860 con 0x7f8d5c103540 2024-03-08T23:07:48.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.877+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 24) v1 ==== 137410+0+0 (secure 0 0 0) 0x7f8d50044020 con 0x7f8d5c103540 2024-03-08T23:07:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.879+0000 7f8d59600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 0x7f8d440a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:07:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.879+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8d500f0970 con 0x7f8d5c103540 2024-03-08T23:07:48.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.879+0000 7f8d5be00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 0x7f8d440a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:07:48.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.880+0000 7f8d5be00640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 0x7f8d440a4f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d480096c0 tx=0x7f8d48009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:07:48.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:48.883+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8d5004b020 con 0x7f8d5c103540 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: mgrmap e23: x(active, since 1.04155s), standbys: y, z 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: [08/Mar/2024:23:07:47] ENGINE Bus STARTING 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: [08/Mar/2024:23:07:47] ENGINE Serving on http://172.21.15.46:8765 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: [08/Mar/2024:23:07:47] ENGINE Serving on https://172.21.15.46:7150 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: [08/Mar/2024:23:07:47] ENGINE Bus STARTED 2024-03-08T23:07:48.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:48 smithi062 ceph-mon[35761]: from='client.25633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.126+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5c061730 con 0x7f8d440a2a90 2024-03-08T23:07:49.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.135+0000 7f8d59600640 1 -- 172.21.15.46:0/2495595366 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f8d5c061730 con 0x7f8d440a2a90 2024-03-08T23:07:49.139 INFO:teuthology.orchestra.run.smithi046.stdout:/ 1m 6m3h 2024-03-08T23:07:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 msgr2=0x7f8d440a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 0x7f8d440a4f50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8d480096c0 tx=0x7f8d48009260 comp rx=0 tx=0).stop 2024-03-08T23:07:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 msgr2=0x7f8d5c199780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:07:49.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f8d500349e0 tx=0x7f8d50032430 comp rx=0 tx=0).stop 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 shutdown_connections 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8d440a2a90 0x7f8d440a4f50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8d5c1050f0 0x7f8d5c1a0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8d5c104730 0x7f8d5c199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 --2- 172.21.15.46:0/2495595366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8d5c103540 0x7f8d5c199780 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.137+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 >> 172.21.15.46:0/2495595366 conn(0x7f8d5c0feab0 msgr2=0x7f8d5c107a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.138+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 shutdown_connections 2024-03-08T23:07:49.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:07:49.138+0000 7f8d63600640 1 -- 172.21.15.46:0/2495595366 wait complete. 2024-03-08T23:07:49.161 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-08T23:07:49.241 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-08T23:07:49.252 INFO:teuthology.run_tasks:Running task workunit... 2024-03-08T23:07:49.295 INFO:tasks.workunit:Pulling workunits from ref 7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:07:49.295 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-08T23:07:49.295 INFO:tasks.workunit:timeout=3h 2024-03-08T23:07:49.295 INFO:tasks.workunit:cleanup=True 2024-03-08T23:07:49.296 DEBUG:teuthology.orchestra.run.smithi046:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:49.343 INFO:teuthology.orchestra.run.smithi046.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Access: 2024-03-08 23:05:08.174958768 +0000 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Modify: 2024-03-08 23:05:08.174958768 +0000 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout:Change: 2024-03-08 23:07:24.933939138 +0000 2024-03-08T23:07:49.344 INFO:teuthology.orchestra.run.smithi046.stdout: Birth: 2024-03-08 23:05:08.174958768 +0000 2024-03-08T23:07:49.344 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-08T23:07:49.344 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-08T23:07:49.474 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:07:49.520 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:49 smithi046 ceph-mon[31859]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:49 smithi192 ceph-mon[36700]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='client.25625 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:49.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:49 smithi062 ceph-mon[35761]: from='client.25637 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:07:51.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:07:51.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:50 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:07:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:07:51.143 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:50 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:07:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:07:51.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:50 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:52.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:52 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-08T23:07:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:52.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:52 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-03-08T23:07:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:52.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:52 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:53 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:07:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:53 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:07:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:53 smithi062 ceph-mon[35761]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T23:07:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:53 smithi062 ceph-mon[35761]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T23:07:53.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:53 smithi062 ceph-mon[35761]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T23:07:53.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:53 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:07:53.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:53 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:07:53.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:53 smithi046 ceph-mon[31859]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T23:07:53.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:53 smithi046 ceph-mon[31859]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T23:07:53.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:53 smithi046 ceph-mon[31859]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T23:07:53.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:53 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:07:53.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:53 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:07:53.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:53 smithi192 ceph-mon[36700]: Updating smithi046:/etc/ceph/ceph.conf 2024-03-08T23:07:53.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:53 smithi192 ceph-mon[36700]: Updating smithi062:/etc/ceph/ceph.conf 2024-03-08T23:07:53.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:53 smithi192 ceph-mon[36700]: Updating smithi192:/etc/ceph/ceph.conf 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:54 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:54 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-08T23:07:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: Updating smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:54.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:54 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:55.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:55 smithi062 ceph-mon[35761]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:55.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:55 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:55.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:55 smithi046 ceph-mon[31859]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:55.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:55 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:55.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:55 smithi192 ceph-mon[36700]: Updating smithi046:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:07:55.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:55 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:56.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:56.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:56 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:56.465 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:56 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: Updating smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 132 KiB/s wr, 77 op/s 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:07:56.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:07:56.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:56 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:07:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:07:58 smithi062 ceph-mon[35761]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-08T23:07:58.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:07:58 smithi046 ceph-mon[31859]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-08T23:07:58.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:07:58 smithi192 ceph-mon[36700]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 103 KiB/s wr, 60 op/s 2024-03-08T23:08:00.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:00 smithi062 ceph-mon[35761]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-08T23:08:00.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-mon[31859]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-08T23:08:00.475 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:08:00.148+0000 7f2380200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:08:00.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:08:00.148+0000 7f2380200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:08:00.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:08:00.148+0000 7f2380200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:08:00.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:08:00.148+0000 7f2380200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:08:00.476 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:08:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:08:00.148+0000 7f2380200700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-08T23:08:00.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:00 smithi192 ceph-mon[36700]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 51 op/s 2024-03-08T23:08:02.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:02 smithi062 ceph-mon[35761]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-08T23:08:02.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:02 smithi062 ceph-mon[35761]: mgrmap e26: x(active, since 14s), standbys: y, z 2024-03-08T23:08:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:02 smithi046 ceph-mon[31859]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-08T23:08:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:02 smithi046 ceph-mon[31859]: mgrmap e26: x(active, since 14s), standbys: y, z 2024-03-08T23:08:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:02 smithi192 ceph-mon[36700]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-03-08T23:08:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:02 smithi192 ceph-mon[36700]: mgrmap e26: x(active, since 14s), standbys: y, z 2024-03-08T23:08:04.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:04 smithi062 ceph-mon[35761]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 88 KiB/s wr, 49 op/s 2024-03-08T23:08:04.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:04 smithi046 ceph-mon[31859]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 88 KiB/s wr, 49 op/s 2024-03-08T23:08:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:04 smithi192 ceph-mon[36700]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 358 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 88 KiB/s wr, 49 op/s 2024-03-08T23:08:06.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:06 smithi062 ceph-mon[35761]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-03-08T23:08:06.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:06 smithi046 ceph-mon[31859]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-03-08T23:08:06.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:06 smithi192 ceph-mon[36700]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 100 KiB/s wr, 65 op/s 2024-03-08T23:08:07.073 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:08:07.074 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:08:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 -- 172.21.15.46:0/239970883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 msgr2=0x7fd000110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 --2- 172.21.15.46:0/239970883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd000110280 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009ca0 tx=0x7fcff802c0f0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 -- 172.21.15.46:0/239970883 shutdown_connections 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 --2- 172.21.15.46:0/239970883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd000110280 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 --2- 172.21.15.46:0/239970883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd000103f90 0x7fd000107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 --2- 172.21.15.46:0/239970883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001035e0 0x7fd0001039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 -- 172.21.15.46:0/239970883 >> 172.21.15.46:0/239970883 conn(0x7fd0000fee00 msgr2=0x7fd000101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 -- 172.21.15.46:0/239970883 shutdown_connections 2024-03-08T23:08:07.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.308+0000 7fcfff200640 1 -- 172.21.15.46:0/239970883 wait complete. 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.309+0000 7fcfff200640 1 Processor -- start 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 -- start start 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd00019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001999f0 0x7fd000199ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd000113770 con 0x7fd0001035e0 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001135f0 con 0x7fd0001999f0 2024-03-08T23:08:07.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcfff200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001138f0 con 0x7fd0001086b0 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56264/0 (socket says 172.21.15.46:56264) 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcffde00640 1 -- 172.21.15.46:0/3363375421 learned_addr learned my addr 172.21.15.46:0/3363375421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcffe800640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001999f0 0x7fd000199ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.310+0000 7fcff7e00640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd00019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 -- 172.21.15.46:0/3363375421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 msgr2=0x7fd00019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd00019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 -- 172.21.15.46:0/3363375421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001999f0 msgr2=0x7fd000199ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001999f0 0x7fd000199ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 -- 172.21.15.46:0/3363375421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00019a510 con 0x7fd0001035e0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcffde00640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e9f0 tx=0x7fcff000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcff6a00640 1 -- 172.21.15.46:0/3363375421 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff000cd80 con 0x7fd0001035e0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00019a800 con 0x7fd0001035e0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcff6a00640 1 -- 172.21.15.46:0/3363375421 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff0005500 con 0x7fd0001035e0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.311+0000 7fcff6a00640 1 -- 172.21.15.46:0/3363375421 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0018630 con 0x7fd0001035e0 2024-03-08T23:08:07.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.312+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0001a6a50 con 0x7fd0001035e0 2024-03-08T23:08:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fcff6a00640 1 -- 172.21.15.46:0/3363375421 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fcff00046c0 con 0x7fd0001035e0 2024-03-08T23:08:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 Processor -- start 2024-03-08T23:08:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 -- start start 2024-03-08T23:08:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a838fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 0x5624a839b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a8351410 con 0x5624a838f790 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a82f3d50 con 0x5624a8398c90 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fd007fa4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a829cb80 con 0x5624a83900b0 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.315+0000 7fcffde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcff7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 0x5624a839b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50396/0 (socket says 172.21.15.46:50396) 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 -- 172.21.15.46:0/54184996 learned_addr learned my addr 172.21.15.46:0/54184996 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffe800640 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a838fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 -- 172.21.15.46:0/54184996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 msgr2=0x5624a839b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 0x5624a839b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 -- 172.21.15.46:0/54184996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 msgr2=0x5624a838fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a838fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 -- 172.21.15.46:0/54184996 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a80d27d0 con 0x5624a83900b0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcff7e00640 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 0x5624a839b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcffde00640 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcff00ccb90 tx=0x7fcff00d5eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf4972bfe0f63ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.316+0000 7fcff6a00640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 0x7fcfe40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.317+0000 7fcfef400640 1 -- 172.21.15.46:0/54184996 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00df020 con 0x5624a83900b0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.317+0000 7fcffe800640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 0x7fcfe40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.317+0000 7fcfef400640 1 -- 172.21.15.46:0/54184996 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff0087c50 con 0x5624a83900b0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.317+0000 7fcfef400640 1 -- 172.21.15.46:0/54184996 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff009b800 con 0x5624a83900b0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.317+0000 7fcff6a00640 1 -- 172.21.15.46:0/3363375421 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcff00cf9e0 con 0x7fd0001035e0 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- 172.21.15.46:0/54184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 msgr2=0x5624a8398750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcff00ccb90 tx=0x7fcff00d5eb0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fcffe800640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 0x7fcfe40a4f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005e30 tx=0x7fcff8005d80 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- 172.21.15.46:0/54184996 shutdown_connections 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a8398c90 0x5624a839b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8398750 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 --2- 172.21.15.46:0/54184996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a838fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- 172.21.15.46:0/54184996 >> 172.21.15.46:0/54184996 conn(0x5624a8389ec0 msgr2=0x5624a838a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- 172.21.15.46:0/54184996 shutdown_connections 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- 172.21.15.46:0/54184996 wait complete. 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 -- start start 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.318+0000 7fd007fa4740 1 Processor -- start 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83961d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a8396710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a82afee0 con 0x5624a838f790 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a82efdf0 con 0x5624a83900b0 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624a839ba80 con 0x5624a8398c90 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fd007fa4740 10 client.0 _ll_register_callbacks cb 0x5624a82b68d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56274/0 (socket says 172.21.15.46:56274) 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fcffde00640 1 -- 172.21.15.46:0/3446950195 learned_addr learned my addr 172.21.15.46:0/3446950195 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fcff7e00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.319+0000 7fcffe800640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a8396710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 -- 172.21.15.46:0/3446950195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 msgr2=0x5624a8396710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a8396710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 -- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 msgr2=0x5624a83961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 -- 172.21.15.46:0/3446950195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624a80083f0 con 0x5624a838f790 2024-03-08T23:08:07.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcffde00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcff0097e90 tx=0x7fcff0004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4796e09de68f17b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcfe3e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00124e0 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcff0023a90 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcff0087c50 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcff0012a60 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.320+0000 7fd007fa4740 10 client.15537 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5624a7fc80f0 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff0087c50 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcff0012680 con 0x5624a838f790 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 10 client.15537 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 20 client.15537 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a83a8cf0 0x5624a83ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 4 client.15537 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624a8301290 con 0x5624a83a8cf0 2024-03-08T23:08:07.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.321+0000 7fcff7e00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a83a8cf0 0x5624a83ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.322+0000 7fcff7e00640 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a83a8cf0 0x5624a83ab0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=771a56aee7fa5e00 server_cookie=c1802904040c3e21 in_seq=0 out_seq=0 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.322+0000 7fcfe3e00640 10 client.15537 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fcfe3e00640 1 -- 172.21.15.46:0/3446950195 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5624a8301290 con 0x5624a83a8cf0 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fcfe3e00640 10 client.15537 handle_command_reply: tid=0 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 client.15537 shutdown 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 msgr2=0x5624a8395c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcff0097e90 tx=0x7fcff0004ae0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 shutdown_connections 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a83a8cf0 0x5624a83ab0e0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a8396710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 --2- 172.21.15.46:0/3446950195 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8395c90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fcfe3e00640 0 client.15537 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 >> 172.21.15.46:0/3446950195 conn(0x5624a8389ec0 msgr2=0x5624a838bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 shutdown_connections 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 1 -- 172.21.15.46:0/3446950195 wait complete. 2024-03-08T23:08:07.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.323+0000 7fd007fa4740 20 client.15537 trim_cache size 0 max 16384 2024-03-08T23:08:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 Processor -- start 2024-03-08T23:08:07.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 -- start start 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8388dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83ad100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a83ad640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001b2aa0 con 0x5624a838f790 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001b0640 con 0x5624a83900b0 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.340+0000 7fd007fa4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001ae6e0 con 0x5624a8398c90 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8388dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcffe800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a83ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcffde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8388dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56280/0 (socket says 172.21.15.46:56280) 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcffde00640 1 -- 172.21.15.46:0/2768506458 learned_addr learned my addr 172.21.15.46:0/2768506458 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 -- 172.21.15.46:0/2768506458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 msgr2=0x5624a83ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a83ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 -- 172.21.15.46:0/2768506458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 msgr2=0x5624a8388dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8388dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.341+0000 7fcff7e00640 1 -- 172.21.15.46:0/2768506458 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624a839b960 con 0x5624a83900b0 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83ad100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x5624a839bb90 tx=0x7fcfe8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a6a7eb681a25dc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fcfeea00640 1 -- 172.21.15.46:0/2768506458 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8005220 con 0x5624a83900b0 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fcfeea00640 1 -- 172.21.15.46:0/2768506458 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfe8023070 con 0x5624a83900b0 2024-03-08T23:08:07.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fcfeea00640 1 -- 172.21.15.46:0/2768506458 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe801f3d0 con 0x5624a83900b0 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 -- 172.21.15.46:0/2768506458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 msgr2=0x5624a83ad100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83ad100 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x5624a839bb90 tx=0x7fcfe8005f90 comp rx=0 tx=0).stop 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 -- 172.21.15.46:0/2768506458 shutdown_connections 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a8398c90 0x5624a83ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a83900b0 0x5624a83ad100 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2768506458 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a838f790 0x5624a8388dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.342+0000 7fd007fa4740 1 -- 172.21.15.46:0/2768506458 >> 172.21.15.46:0/2768506458 conn(0x5624a8389ec0 msgr2=0x5624a83886b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.343+0000 7fd007fa4740 1 -- 172.21.15.46:0/2768506458 shutdown_connections 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.343+0000 7fd007fa4740 1 -- 172.21.15.46:0/2768506458 wait complete. 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.343+0000 7fd007fa4740 1 -- start start 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.343+0000 7fd007fa4740 1 Processor -- start 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8401530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a8398c90 0x5624a83fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001aece0 con 0x5624a8398c90 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001affe0 con 0x5624a838f790 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0001af320 con 0x5624a83900b0 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fd007fa4740 10 client.0 _ll_register_callbacks cb 0x7fcff00d53a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffe800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a8398c90 0x5624a83fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37552/0 (socket says 172.21.15.46:37552) 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 -- 172.21.15.46:0/2472349935 learned_addr learned my addr 172.21.15.46:0/2472349935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8401530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 -- 172.21.15.46:0/2472349935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 msgr2=0x5624a8401530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8401530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 -- 172.21.15.46:0/2472349935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a8398c90 msgr2=0x5624a83fb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a8398c90 0x5624a83fb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcffde00640 1 -- 172.21.15.46:0/2472349935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624a83fbc50 con 0x5624a838f790 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.344+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8401530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcffde00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcff0099ce0 tx=0x7fcff00d5eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e028c43d2837a8b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff00df070 con 0x5624a838f790 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcff00d5450 con 0x5624a838f790 2024-03-08T23:08:07.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfe8023070 con 0x5624a838f790 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcff0012c60 con 0x5624a838f790 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fd007fa4740 10 client.25661 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5624a83fbc50 con 0x5624a838f790 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfe8023070 con 0x5624a838f790 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.345+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcff00ba180 con 0x5624a838f790 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 10 client.25661 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 10 client.25661 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 20 client.25661 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a8408ab0 0x5624a840aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 4 client.25661 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5624a82fb5e0 con 0x5624a8408ab0 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a8408ab0 0x5624a840aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fcff7e00640 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a8408ab0 0x5624a840aed0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60f6e1de5cf38fe0 server_cookie=e8d99de8aab95fb0 in_seq=0 out_seq=0 2024-03-08T23:08:07.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.346+0000 7fcfe3e00640 10 client.25661 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fcfe3e00640 1 -- 172.21.15.46:0/2472349935 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5624a82fb5e0 con 0x5624a8408ab0 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fcfe3e00640 10 client.25661 handle_command_reply: tid=0 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 client.25661 shutdown 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 msgr2=0x5624a8400ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fcff0099ce0 tx=0x7fcff00d5eb0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 shutdown_connections 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5624a8408ab0 0x5624a840aed0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5624a8398c90 0x5624a83fb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fcfe3e00640 0 client.25661 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5624a83900b0 0x5624a8401530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 --2- 172.21.15.46:0/2472349935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5624a838f790 0x5624a8400ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 >> 172.21.15.46:0/2472349935 conn(0x5624a8389ec0 msgr2=0x5624a83a4560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 shutdown_connections 2024-03-08T23:08:07.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 1 -- 172.21.15.46:0/2472349935 wait complete. 2024-03-08T23:08:07.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.347+0000 7fd007fa4740 20 client.25661 trim_cache size 0 max 16384 2024-03-08T23:08:07.355 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:07.355 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:08:07.355 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:08:07.355 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 msgr2=0x7fcfe40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 0x7fcfe40a4f50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcff8005e30 tx=0x7fcff8005d80 comp rx=0 tx=0).stop 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 msgr2=0x7fd00019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e9f0 tx=0x7fcff000edf0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 shutdown_connections 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfe40a2a90 0x7fcfe40a4f50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd0001999f0 0x7fd000199ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd0001086b0 0x7fd00019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 --2- 172.21.15.46:0/3363375421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0001035e0 0x7fd00019f840 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 >> 172.21.15.46:0/3363375421 conn(0x7fd0000fee00 msgr2=0x7fd0000ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 shutdown_connections 2024-03-08T23:08:07.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.348+0000 7fcfff200640 1 -- 172.21.15.46:0/3363375421 wait complete. 2024-03-08T23:08:07.369 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:08:07.370 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:08:07.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 -- 172.21.15.46:0/2570535502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94103ee0 msgr2=0x7ffa94107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 --2- 172.21.15.46:0/2570535502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94103ee0 0x7ffa94107f20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009900 tx=0x7ffa7c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:08:07.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 -- 172.21.15.46:0/2570535502 shutdown_connections 2024-03-08T23:08:07.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 --2- 172.21.15.46:0/2570535502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94108600 0x7ffa941101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 --2- 172.21.15.46:0/2570535502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94103ee0 0x7ffa94107f20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 --2- 172.21.15.46:0/2570535502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94103530 0x7ffa94103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 -- 172.21.15.46:0/2570535502 >> 172.21.15.46:0/2570535502 conn(0x7ffa940fed90 msgr2=0x7ffa941011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 -- 172.21.15.46:0/2570535502 shutdown_connections 2024-03-08T23:08:07.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.591+0000 7ffa99200640 1 -- 172.21.15.46:0/2570535502 wait complete. 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.592+0000 7ffa99200640 1 Processor -- start 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 -- start start 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94108600 0x7ffa9419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94199a40 0x7ffa94199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa941136e0 con 0x7ffa94199a40 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa94113560 con 0x7ffa94108600 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa99200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa94113860 con 0x7ffa94103530 2024-03-08T23:08:07.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa93400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa93e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94199a40 0x7ffa94199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa93400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50434/0 (socket says 172.21.15.46:50434) 2024-03-08T23:08:07.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa93400640 1 -- 172.21.15.46:0/3117356404 learned_addr learned my addr 172.21.15.46:0/3117356404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:07.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.593+0000 7ffa92a00640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94108600 0x7ffa9419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 -- 172.21.15.46:0/3117356404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94108600 msgr2=0x7ffa9419fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94108600 0x7ffa9419fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 -- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94199a40 msgr2=0x7ffa94199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94199a40 0x7ffa94199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 -- 172.21.15.46:0/3117356404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9419a560 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.594+0000 7ffa93400640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400ea60 tx=0x7ffa8400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.595+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400cde0 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.595+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9419a850 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.595+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa840035f0 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.595+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa84010670 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.595+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa941a6aa0 con 0x7ffa94103530 2024-03-08T23:08:07.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.596+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa94068450 con 0x7ffa94103530 2024-03-08T23:08:07.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.598+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ffa840037a0 con 0x7ffa94103530 2024-03-08T23:08:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.599+0000 7ffa8be00640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 0x7ffa780a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.600+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ffa840cf7e0 con 0x7ffa94103530 2024-03-08T23:08:07.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.600+0000 7ffa92a00640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 0x7ffa780a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:07.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.601+0000 7ffa92a00640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 0x7ffa780a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c02c240 tx=0x7ffa7c02df90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:07.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.604+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffa8409bf30 con 0x7ffa94103530 2024-03-08T23:08:07.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.845+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa9410b860 con 0x7ffa94103530 2024-03-08T23:08:07.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.846+0000 7ffa8be00640 1 -- 172.21.15.46:0/3117356404 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11334 (secure 0 0 0) 0x7ffa8409b830 con 0x7ffa94103530 2024-03-08T23:08:07.849 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:08:07.849 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:07:25.750030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:08:07.849 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 msgr2=0x7ffa780a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 0x7ffa780a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c02c240 tx=0x7ffa7c02df90 comp rx=0 tx=0).stop 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 msgr2=0x7ffa9419f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400ea60 tx=0x7ffa8400ef20 comp rx=0 tx=0).stop 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 shutdown_connections 2024-03-08T23:08:07.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffa780a2a90 0x7ffa780a4f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffa94199a40 0x7ffa94199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffa94108600 0x7ffa9419fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 --2- 172.21.15.46:0/3117356404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffa94103530 0x7ffa9419f890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 >> 172.21.15.46:0/3117356404 conn(0x7ffa940fed90 msgr2=0x7ffa94101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 shutdown_connections 2024-03-08T23:08:07.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:07.849+0000 7ffa99200640 1 -- 172.21.15.46:0/3117356404 wait complete. 2024-03-08T23:08:07.871 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:07.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:08:08.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 -- 172.21.15.46:0/2745486125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1064a0 msgr2=0x7f45dc10e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 --2- 172.21.15.46:0/2745486125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1064a0 0x7f45dc10e070 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009ca0 tx=0x7f45d402c100 comp rx=0 tx=0).stop 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 -- 172.21.15.46:0/2745486125 shutdown_connections 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 --2- 172.21.15.46:0/2745486125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1064a0 0x7f45dc10e070 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 --2- 172.21.15.46:0/2745486125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc101d80 0x7f45dc105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 --2- 172.21.15.46:0/2745486125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1013d0 0x7f45dc1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.092+0000 7f45e2000640 1 -- 172.21.15.46:0/2745486125 >> 172.21.15.46:0/2745486125 conn(0x7f45dc0fcc50 msgr2=0x7f45dc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.093+0000 7f45e2000640 1 -- 172.21.15.46:0/2745486125 shutdown_connections 2024-03-08T23:08:08.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.093+0000 7f45e2000640 1 -- 172.21.15.46:0/2745486125 wait complete. 2024-03-08T23:08:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.094+0000 7f45e2000640 1 Processor -- start 2024-03-08T23:08:08.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.094+0000 7f45e2000640 1 -- start start 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc1013d0 0x7f45dc19f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1999e0 0x7f45dc199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc111540 con 0x7f45dc1999e0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1113c0 con 0x7f45dc1064a0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e2000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1116c0 con 0x7f45dc1013d0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37592/0 (socket says 172.21.15.46:37592) 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 -- 172.21.15.46:0/1660823090 learned_addr learned my addr 172.21.15.46:0/1660823090 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e1600640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1999e0 0x7f45dc199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45e0c00640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc1013d0 0x7f45dc19f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 -- 172.21.15.46:0/1660823090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc1013d0 msgr2=0x7f45dc19f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc1013d0 0x7f45dc19f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 -- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1999e0 msgr2=0x7f45dc199e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1999e0 0x7f45dc199e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.095+0000 7f45dbe00640 1 -- 172.21.15.46:0/1660823090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc19a500 con 0x7f45dc1064a0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e9c0 tx=0x7f45c800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45d9600640 1 -- 172.21.15.46:0/1660823090 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c800cd80 con 0x7f45dc1064a0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc19a7f0 con 0x7f45dc1064a0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45d9600640 1 -- 172.21.15.46:0/1660823090 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45c8005500 con 0x7f45dc1064a0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45d9600640 1 -- 172.21.15.46:0/1660823090 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8010690 con 0x7f45dc1064a0 2024-03-08T23:08:08.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.096+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45dc1a6a90 con 0x7f45dc1064a0 2024-03-08T23:08:08.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.099+0000 7f45e621b740 1 Processor -- start 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 -- start start 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45d9600640 1 -- 172.21.15.46:0/1660823090 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f45c80046c0 con 0x7f45dc1064a0 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb107cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb10727dd0 con 0x55eb107c3530 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb106d0c70 con 0x55eb107cca30 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e621b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb10723e70 con 0x55eb107c3e50 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb107cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56344/0 (socket says 172.21.15.46:56344) 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e1600640 1 -- 172.21.15.46:0/66139614 learned_addr learned my addr 172.21.15.46:0/66139614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.100+0000 7f45e0c00640 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 -- 172.21.15.46:0/66139614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 msgr2=0x55eb107cc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107cc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 -- 172.21.15.46:0/66139614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 msgr2=0x55eb107c3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 -- 172.21.15.46:0/66139614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb1078a2b0 con 0x55eb107cca30 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45e1600640 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45dbe00640 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb107cee20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f45c80cc9a0 tx=0x7f45c80d5b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64e3f7d5d1b993f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45cdc00640 1 -- 172.21.15.46:0/66139614 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c80cfd80 con 0x55eb107cca30 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.101+0000 7f45cdc00640 1 -- 172.21.15.46:0/66139614 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45c80d7d30 con 0x55eb107cca30 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45cdc00640 1 -- 172.21.15.46:0/66139614 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c80da420 con 0x55eb107cca30 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45d9600640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 0x7f45c40a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45d9600640 1 -- 172.21.15.46:0/1660823090 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f45c80cf790 con 0x7f45dc1064a0 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e1600640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 0x7f45c40a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 -- 172.21.15.46:0/66139614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 msgr2=0x55eb107cee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb107cee20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f45c80cc9a0 tx=0x7f45c80d5b20 comp rx=0 tx=0).stop 2024-03-08T23:08:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 -- 172.21.15.46:0/66139614 shutdown_connections 2024-03-08T23:08:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb107cee20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107cc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 --2- 172.21.15.46:0/66139614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107c3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 -- 172.21.15.46:0/66139614 >> 172.21.15.46:0/66139614 conn(0x55eb107bdc60 msgr2=0x55eb107be040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.102+0000 7f45e621b740 1 -- 172.21.15.46:0/66139614 shutdown_connections 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.103+0000 7f45e621b740 1 -- 172.21.15.46:0/66139614 wait complete. 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.103+0000 7f45e1600640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 0x7f45c40a4fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009a60 tx=0x7f45d4005c50 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.103+0000 7f45e621b740 1 -- start start 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.103+0000 7f45e621b740 1 Processor -- start 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 0x55eb107cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 0x55eb107c9770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 0x55eb107c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb107cf820 con 0x55eb107cca30 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb106e40c0 con 0x55eb107c3e50 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb107cf9a0 con 0x55eb107c3530 2024-03-08T23:08:08.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e621b740 10 client.0 _ll_register_callbacks cb 0x55eb1071eec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 0x55eb107c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 0x55eb107c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e1600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 0x55eb107c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56360/0 (socket says 172.21.15.46:56360) 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e0c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 0x55eb107cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e1600640 1 -- 172.21.15.46:0/1745245364 learned_addr learned my addr 172.21.15.46:0/1745245364 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e0c00640 1 -- 172.21.15.46:0/1745245364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 msgr2=0x55eb107c9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.104+0000 7f45e0c00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 0x55eb107c9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45e0c00640 1 -- 172.21.15.46:0/1745245364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 msgr2=0x55eb107c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45e0c00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 0x55eb107c9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45e0c00640 1 -- 172.21.15.46:0/1745245364 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb1043c3f0 con 0x55eb107c3530 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 0x55eb107c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45e0c00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 0x55eb107cc3e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19ab20 tx=0x7f45d0012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f928a8e7d455781 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45c2000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:08:08.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d000f920 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45c80d7d30 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.105+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f45d000fac0 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.106+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45d0025050 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.106+0000 7f45e621b740 10 client.25693 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.106+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55eb105067d0 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.106+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45c80d7d30 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.106+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45d0010430 con 0x55eb107c3530 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 10 client.25693 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 10 client.25693 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 20 client.25693 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb107dcb20 0x55eb107def10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 4 client.25693 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb106fc4f0 con 0x55eb107dcb20 2024-03-08T23:08:08.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb107dcb20 0x55eb107def10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45dbe00640 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb107dcb20 0x55eb107def10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=751e23d9471069c0 server_cookie=742cd9640836c251 in_seq=0 out_seq=0 2024-03-08T23:08:08.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.107+0000 7f45c2000640 10 client.25693 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45c2000640 1 -- 172.21.15.46:0/1745245364 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eb106fc4f0 con 0x55eb107dcb20 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45c2000640 10 client.25693 handle_command_reply: tid=0 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 client.25693 shutdown 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 msgr2=0x55eb107cc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 0x55eb107cc3e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19ab20 tx=0x7f45d0012b60 comp rx=0 tx=0).stop 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 shutdown_connections 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb107dcb20 0x55eb107def10 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107cca30 0x55eb107c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3e50 0x55eb107c9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45c2000640 0 client.25693 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 --2- 172.21.15.46:0/1745245364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3530 0x55eb107cc3e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.108+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 >> 172.21.15.46:0/1745245364 conn(0x55eb107bdc60 msgr2=0x55eb107bf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.109+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 shutdown_connections 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.109+0000 7f45e621b740 1 -- 172.21.15.46:0/1745245364 wait complete. 2024-03-08T23:08:08.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.109+0000 7f45e621b740 20 client.25693 trim_cache size 0 max 16384 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 Processor -- start 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 -- start start 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 0x55eb107cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107cca30 0x55eb107c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1b2aa0 con 0x55eb107c3e50 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1b0640 con 0x55eb107c3530 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e621b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1ae6e0 con 0x55eb107cca30 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56364/0 (socket says 172.21.15.46:56364) 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45dbe00640 1 -- 172.21.15.46:0/3761830556 learned_addr learned my addr 172.21.15.46:0/3761830556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.122+0000 7f45e0c00640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 0x55eb107cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e1600640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107cca30 0x55eb107c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 -- 172.21.15.46:0/3761830556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107cca30 msgr2=0x55eb107c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107cca30 0x55eb107c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 -- 172.21.15.46:0/3761830556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 msgr2=0x55eb107cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 -- 172.21.15.46:0/3761830556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb107c3c80 con 0x55eb107c3530 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45dbe00640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45e0c00640 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 0x55eb107cbea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45d0001eb0 tx=0x7f45d00096c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5854a3fad8b0b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45cd200640 1 -- 172.21.15.46:0/3761830556 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d0022040 con 0x55eb107c3530 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45cd200640 1 -- 172.21.15.46:0/3761830556 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45d000fe00 con 0x55eb107c3530 2024-03-08T23:08:08.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.123+0000 7f45cd200640 1 -- 172.21.15.46:0/3761830556 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d00053b0 con 0x55eb107c3530 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- 172.21.15.46:0/3761830556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 msgr2=0x55eb107cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 0x55eb107cbea0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f45d0001eb0 tx=0x7f45d00096c0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- 172.21.15.46:0/3761830556 shutdown_connections 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107cca30 0x55eb107c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3e50 0x55eb107cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 --2- 172.21.15.46:0/3761830556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107c3530 0x55eb107cbea0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- 172.21.15.46:0/3761830556 >> 172.21.15.46:0/3761830556 conn(0x55eb107bdc60 msgr2=0x55eb107e3850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- 172.21.15.46:0/3761830556 shutdown_connections 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- 172.21.15.46:0/3761830556 wait complete. 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.124+0000 7f45e621b740 1 -- start start 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 Processor -- start 2024-03-08T23:08:08.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107d2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb10832fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1aece0 con 0x55eb107c3530 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1affe0 con 0x55eb107cca30 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45dc1af320 con 0x55eb107c3e50 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e621b740 10 client.0 _ll_register_callbacks cb 0x7f45c80d5400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56376/0 (socket says 172.21.15.46:56376) 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.125+0000 7f45e0c00640 1 -- 172.21.15.46:0/4290333983 learned_addr learned my addr 172.21.15.46:0/4290333983 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45dbe00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107d2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e1600640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb10832fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 -- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 msgr2=0x55eb107d2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107d2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 -- 172.21.15.46:0/4290333983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 msgr2=0x55eb10832fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb10832fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 -- 172.21.15.46:0/4290333983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb107ca080 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45dbe00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107d2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e1600640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb10832fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e0c00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f45d0012ba0 tx=0x7f45d0010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e77e97cab25dca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45c2000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45d000f920 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f45d000fac0 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.126+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45d000fe00 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45d0005070 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 10 client.15567 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55eb10834430 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45d000fe00 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45d00039f0 con 0x55eb107c3530 2024-03-08T23:08:08.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 10 client.15567 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 10 client.15567 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 20 client.15567 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb10834aa0 0x55eb10836ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 4 client.15567 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.127+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55eb10735310 con 0x55eb10834aa0 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45dbe00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb10834aa0 0x55eb10836ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45dbe00640 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb10834aa0 0x55eb10836ec0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ae2394d28b8ad23 server_cookie=62b21ec22d3bd78f in_seq=0 out_seq=0 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45c2000640 10 client.15567 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45c2000640 1 -- 172.21.15.46:0/4290333983 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55eb10735310 con 0x55eb10834aa0 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45c2000640 10 client.15567 handle_command_reply: tid=0 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.128+0000 7f45e621b740 1 client.15567 shutdown 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 msgr2=0x55eb107d28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f45d0012ba0 tx=0x7f45d0010c20 comp rx=0 tx=0).stop 2024-03-08T23:08:08.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 shutdown_connections 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55eb10834aa0 0x55eb10836ec0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eb107cca30 0x55eb10832fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eb107c3e50 0x55eb107d2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 --2- 172.21.15.46:0/4290333983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eb107c3530 0x55eb107d28a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45c2000640 0 client.15567 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 >> 172.21.15.46:0/4290333983 conn(0x55eb107bdc60 msgr2=0x55eb107bc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 shutdown_connections 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 1 -- 172.21.15.46:0/4290333983 wait complete. 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.129+0000 7f45e621b740 20 client.15567 trim_cache size 0 max 16384 2024-03-08T23:08:08.136 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 msgr2=0x7f45c40a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 0x7f45c40a4fd0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009a60 tx=0x7f45d4005c50 comp rx=0 tx=0).stop 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 msgr2=0x7f45dc19fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45c800e9c0 tx=0x7f45c800ee80 comp rx=0 tx=0).stop 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 shutdown_connections 2024-03-08T23:08:08.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45c40a2b10 0x7f45c40a4fd0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45dc1999e0 0x7f45dc199e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45dc1064a0 0x7f45dc19fd70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 --2- 172.21.15.46:0/1660823090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45dc1013d0 0x7f45dc19f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 >> 172.21.15.46:0/1660823090 conn(0x7f45dc0fcc50 msgr2=0x7f45dc10c710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 shutdown_connections 2024-03-08T23:08:08.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.130+0000 7f45e2000640 1 -- 172.21.15.46:0/1660823090 wait complete. 2024-03-08T23:08:08.153 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:08.153 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:08:08.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 -- 172.21.15.46:0/2041525393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 msgr2=0x7fd9581017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 --2- 172.21.15.46:0/2041525393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd9581017b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd948009900 tx=0x7fd94802bc40 comp rx=0 tx=0).stop 2024-03-08T23:08:08.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 -- 172.21.15.46:0/2041525393 shutdown_connections 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 --2- 172.21.15.46:0/2041525393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581064a0 0x7fd95810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 --2- 172.21.15.46:0/2041525393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd958101d80 0x7fd958105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 --2- 172.21.15.46:0/2041525393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd9581017b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.370+0000 7fd957e00640 1 -- 172.21.15.46:0/2041525393 >> 172.21.15.46:0/2041525393 conn(0x7fd9580fcc90 msgr2=0x7fd9580ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.371+0000 7fd957e00640 1 -- 172.21.15.46:0/2041525393 shutdown_connections 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.371+0000 7fd957e00640 1 -- 172.21.15.46:0/2041525393 wait complete. 2024-03-08T23:08:08.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.372+0000 7fd957e00640 1 Processor -- start 2024-03-08T23:08:08.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.372+0000 7fd957e00640 1 -- start start 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9581064a0 0x7fd95819dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581978c0 0x7fd958197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958111580 con 0x7fd9581064a0 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958111400 con 0x7fd9581013d0 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958111700 con 0x7fd9581978c0 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd94e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9581064a0 0x7fd95819dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37638/0 (socket says 172.21.15.46:37638) 2024-03-08T23:08:08.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 -- 172.21.15.46:0/1698774706 learned_addr learned my addr 172.21.15.46:0/1698774706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd957400640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581978c0 0x7fd958197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 -- 172.21.15.46:0/1698774706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581978c0 msgr2=0x7fd958197d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581978c0 0x7fd958197d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 -- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9581064a0 msgr2=0x7fd95819dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.373+0000 7fd956a00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9581064a0 0x7fd95819dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd956a00640 1 -- 172.21.15.46:0/1698774706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9581983e0 con 0x7fd9581013d0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd956a00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd948005ab0 tx=0x7fd9480042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd94fe00640 1 -- 172.21.15.46:0/1698774706 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948041070 con 0x7fd9581013d0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd94fe00640 1 -- 172.21.15.46:0/1698774706 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd94802cca0 con 0x7fd9581013d0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd94fe00640 1 -- 172.21.15.46:0/1698774706 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd948003d90 con 0x7fd9581013d0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd958198670 con 0x7fd9581013d0 2024-03-08T23:08:08.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.374+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9581a47c0 con 0x7fd9581013d0 2024-03-08T23:08:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 Processor -- start 2024-03-08T23:08:08.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 -- start start 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd94fe00640 1 -- 172.21.15.46:0/1698774706 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fd948044070 con 0x7fd9581013d0 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a65720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a75230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed589c9dd0 con 0x55ed58a65340 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed58972c70 con 0x55ed58a72e40 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.379+0000 7fd960609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed589c5e70 con 0x55ed58a65cf0 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd957400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a65720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a75230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd957400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a65720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56400/0 (socket says 172.21.15.46:56400) 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd957400640 1 -- 172.21.15.46:0/2497380429 learned_addr learned my addr 172.21.15.46:0/2497380429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd956a00640 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 -- 172.21.15.46:0/2497380429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 msgr2=0x55ed58a72900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a72900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 -- 172.21.15.46:0/2497380429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 msgr2=0x55ed58a65720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a65720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 -- 172.21.15.46:0/2497380429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed58a2c2b0 con 0x55ed58a72e40 2024-03-08T23:08:08.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.380+0000 7fd94e000640 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a75230 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd958069020 tx=0x7fd940011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2004b521aa48612 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd947e00640 1 -- 172.21.15.46:0/2497380429 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940004050 con 0x55ed58a72e40 2024-03-08T23:08:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd947e00640 1 -- 172.21.15.46:0/2497380429 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd940005500 con 0x55ed58a72e40 2024-03-08T23:08:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd947e00640 1 -- 172.21.15.46:0/2497380429 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940004db0 con 0x55ed58a72e40 2024-03-08T23:08:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd94fe00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 0x7fd93c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd94fe00640 1 -- 172.21.15.46:0/1698774706 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd9480f0e10 con 0x7fd9581013d0 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd957400640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 0x7fd93c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd960609740 1 -- 172.21.15.46:0/2497380429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 msgr2=0x55ed58a75230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.381+0000 7fd960609740 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a75230 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd958069020 tx=0x7fd940011810 comp rx=0 tx=0).stop 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 -- 172.21.15.46:0/2497380429 shutdown_connections 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a75230 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a72900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 --2- 172.21.15.46:0/2497380429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a65720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 -- 172.21.15.46:0/2497380429 >> 172.21.15.46:0/2497380429 conn(0x55ed58a5fc60 msgr2=0x55ed58a60040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 -- 172.21.15.46:0/2497380429 shutdown_connections 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 -- 172.21.15.46:0/2497380429 wait complete. 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd957400640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 0x7fd93c0a4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9500099a0 tx=0x7fd950009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 -- start start 2024-03-08T23:08:08.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.382+0000 7fd960609740 1 Processor -- start 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 0x55ed58a700e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed58a75c00 con 0x55ed58a65340 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed589860c0 con 0x55ed58a65cf0 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed58a75d80 con 0x55ed58a72e40 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd960609740 10 client.0 _ll_register_callbacks cb 0x55ed589c0ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd956a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd957400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 0x55ed58a700e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd94e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd94e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37664/0 (socket says 172.21.15.46:37664) 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd957400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 0x55ed58a700e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50532/0 (socket says 172.21.15.46:50532) 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.383+0000 7fd94e000640 1 -- 172.21.15.46:0/829909571 learned_addr learned my addr 172.21.15.46:0/829909571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 -- 172.21.15.46:0/829909571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 msgr2=0x55ed58a700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 0x55ed58a700e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 -- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 msgr2=0x55ed58a727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 -- 172.21.15.46:0/829909571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed586de3f0 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd94e000640 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd94001b040 tx=0x7fd940009fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0465d9ed67733d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd944c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd940027070 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd940005500 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.384+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd940004f00 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd940033050 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 10 client.25697 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ed587a87d0 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd940005500 con 0x55ed58a65cf0 2024-03-08T23:08:08.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd940003c60 con 0x55ed58a65cf0 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 10 client.25697 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 10 client.25697 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 20 client.25697 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58a82e10 0x55ed58a85200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 4 client.25697 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.385+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed5899e4f0 con 0x55ed58a82e10 2024-03-08T23:08:08.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.386+0000 7fd956a00640 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58a82e10 0x55ed58a85200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:08 smithi062 ceph-mon[35761]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:08.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:08 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3117356404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.386+0000 7fd956a00640 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58a82e10 0x55ed58a85200 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8d6b3c7640f238b6 server_cookie=972d54fee59b7553 in_seq=0 out_seq=0 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.386+0000 7fd944c00640 10 client.25697 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd944c00640 1 -- 172.21.15.46:0/829909571 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed5899e4f0 con 0x55ed58a82e10 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd944c00640 10 client.25697 handle_command_reply: tid=0 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 client.25697 shutdown 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 msgr2=0x55ed58a6fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd94001b040 tx=0x7fd940009fa0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 shutdown_connections 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58a82e10 0x55ed58a85200 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a72e40 0x55ed58a700e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a65cf0 0x55ed58a6fb60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 --2- 172.21.15.46:0/829909571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 >> 172.21.15.46:0/829909571 conn(0x55ed58a5fc60 msgr2=0x55ed58a619b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd944c00640 0 client.25697 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 shutdown_connections 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.387+0000 7fd960609740 1 -- 172.21.15.46:0/829909571 wait complete. 2024-03-08T23:08:08.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.388+0000 7fd960609740 20 client.25697 trim_cache size 0 max 16384 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 Processor -- start 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 -- start start 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 0x55ed58a6eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a727d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581ae5c0 con 0x55ed58a65cf0 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581ac160 con 0x55ed58a72e40 2024-03-08T23:08:08.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd960609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581aa200 con 0x55ed58a65340 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.401+0000 7fd94e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 0x55ed58a6eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd94e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56404/0 (socket says 172.21.15.46:56404) 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd94e000640 1 -- 172.21.15.46:0/357200823 learned_addr learned my addr 172.21.15.46:0/357200823 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd957400640 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a727d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 -- 172.21.15.46:0/357200823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 msgr2=0x55ed58a727d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a727d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 -- 172.21.15.46:0/357200823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 msgr2=0x55ed58a72290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 -- 172.21.15.46:0/357200823 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed58a75ae0 con 0x55ed58a65340 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd94e000640 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.402+0000 7fd956a00640 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 0x55ed58a6eb60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd9480ee130 tx=0x7fd9480dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a116bde033bf352d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd947400640 1 -- 172.21.15.46:0/357200823 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94804b040 con 0x55ed58a65340 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd947400640 1 -- 172.21.15.46:0/357200823 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd948004c80 con 0x55ed58a65340 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd947400640 1 -- 172.21.15.46:0/357200823 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9480bd670 con 0x55ed58a65340 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 -- 172.21.15.46:0/357200823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 msgr2=0x55ed58a6eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 0x55ed58a6eb60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd9480ee130 tx=0x7fd9480dd040 comp rx=0 tx=0).stop 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 -- 172.21.15.46:0/357200823 shutdown_connections 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a727d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65cf0 0x55ed58a72290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 --2- 172.21.15.46:0/357200823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65340 0x55ed58a6eb60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 -- 172.21.15.46:0/357200823 >> 172.21.15.46:0/357200823 conn(0x55ed58a5fc60 msgr2=0x55ed58a6e450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 -- 172.21.15.46:0/357200823 shutdown_connections 2024-03-08T23:08:08.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.403+0000 7fd960609740 1 -- 172.21.15.46:0/357200823 wait complete. 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 -- start start 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 Processor -- start 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a79c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a7a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a78120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581aa800 con 0x55ed58a65340 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581abb00 con 0x55ed58a72e40 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.404+0000 7fd960609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581aae40 con 0x55ed58a65cf0 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd960609740 10 client.0 _ll_register_callbacks cb 0x7fd9400097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a79c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd957400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a78120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd957400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a78120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37674/0 (socket says 172.21.15.46:37674) 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd957400640 1 -- 172.21.15.46:0/3044303117 learned_addr learned my addr 172.21.15.46:0/3044303117 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd94e000640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a7a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 -- 172.21.15.46:0/3044303117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 msgr2=0x55ed58a7a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a7a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 -- 172.21.15.46:0/3044303117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 msgr2=0x55ed58a78120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a78120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd956a00640 1 -- 172.21.15.46:0/3044303117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed58a78890 con 0x55ed58a65340 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.405+0000 7fd94e000640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a7a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd956a00640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a79c60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b9940 tx=0x7fd94802cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e161c0dc2c10fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd944c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94804b070 con 0x55ed58a65340 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd9480b95e0 con 0x55ed58a65340 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd948004c80 con 0x55ed58a65340 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9480bcd40 con 0x55ed58a65340 2024-03-08T23:08:08.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd960609740 10 client.15585 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55ed58ad9bf0 con 0x55ed58a65340 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.406+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd948004c80 con 0x55ed58a65340 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd9480c5280 con 0x55ed58a65340 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 10 client.15585 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 10 client.15585 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 20 client.15585 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58ada1b0 0x55ed58adc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 4 client.15585 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed589d7310 con 0x55ed58ada1b0 2024-03-08T23:08:08.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd94e000640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58ada1b0 0x55ed58adc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.407+0000 7fd94e000640 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58ada1b0 0x55ed58adc5a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7467c4d2bfb89da4 server_cookie=818f2bd1b6d12f7f in_seq=0 out_seq=0 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd944c00640 10 client.15585 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd944c00640 1 -- 172.21.15.46:0/3044303117 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ed589d7310 con 0x55ed58ada1b0 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd944c00640 10 client.15585 handle_command_reply: tid=0 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 client.15585 shutdown 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 msgr2=0x55ed58a79c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a79c60 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd9480b9940 tx=0x7fd94802cf90 comp rx=0 tx=0).stop 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 shutdown_connections 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed58a72e40 0x55ed58a78120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed58ada1b0 0x55ed58adc5a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed58a65cf0 0x55ed58a7a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 --2- 172.21.15.46:0/3044303117 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed58a65340 0x55ed58a79c60 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd944c00640 0 client.15585 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:08.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 >> 172.21.15.46:0/3044303117 conn(0x55ed58a5fc60 msgr2=0x55ed58a6ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 shutdown_connections 2024-03-08T23:08:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 1 -- 172.21.15.46:0/3044303117 wait complete. 2024-03-08T23:08:08.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.408+0000 7fd960609740 20 client.15585 trim_cache size 0 max 16384 2024-03-08T23:08:08.416 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 msgr2=0x7fd93c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 0x7fd93c0a4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd9500099a0 tx=0x7fd950009500 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 msgr2=0x7fd95819d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd948005ab0 tx=0x7fd9480042f0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 shutdown_connections 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd93c0a2a90 0x7fd93c0a4f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd9581978c0 0x7fd958197d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd9581064a0 0x7fd95819dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 --2- 172.21.15.46:0/1698774706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd9581013d0 0x7fd95819d6c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 >> 172.21.15.46:0/1698774706 conn(0x7fd9580fcc90 msgr2=0x7fd9580fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 shutdown_connections 2024-03-08T23:08:08.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.409+0000 7fd957e00640 1 -- 172.21.15.46:0/1698774706 wait complete. 2024-03-08T23:08:08.429 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:08 smithi046 ceph-mon[31859]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:08.429 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:08 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3117356404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:08.431 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:08.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:08:08.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:08 smithi192 ceph-mon[36700]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:08.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:08 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3117356404' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 -- 172.21.15.46:0/1293974221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ff3c0 msgr2=0x7fa5480ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 --2- 172.21.15.46:0/1293974221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ff3c0 0x7fa5480ff840 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fa530009900 tx=0x7fa53002bc30 comp rx=0 tx=0).stop 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 -- 172.21.15.46:0/1293974221 shutdown_connections 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 --2- 172.21.15.46:0/1293974221 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5480ffd80 0x7fa54810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 --2- 172.21.15.46:0/1293974221 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ff3c0 0x7fa5480ff840 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 --2- 172.21.15.46:0/1293974221 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5481073d0 0x7fa5481077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 -- 172.21.15.46:0/1293974221 >> 172.21.15.46:0/1293974221 conn(0x7fa548076340 msgr2=0x7fa5480fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 -- 172.21.15.46:0/1293974221 shutdown_connections 2024-03-08T23:08:08.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.642+0000 7fa54e800640 1 -- 172.21.15.46:0/1293974221 wait complete. 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 Processor -- start 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 -- start start 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 0x7fa548199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ffd80 0x7fa54819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa548111670 con 0x7fa5480ffd80 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481114f0 con 0x7fa5481073d0 2024-03-08T23:08:08.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.643+0000 7fa54e800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481117f0 con 0x7fa5480ff3c0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 0x7fa548199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37692/0 (socket says 172.21.15.46:37692) 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547e00640 1 -- 172.21.15.46:0/1416397511 learned_addr learned my addr 172.21.15.46:0/1416397511 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa53fe00640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ffd80 0x7fa54819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 -- 172.21.15.46:0/1416397511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 msgr2=0x7fa54819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 -- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ffd80 msgr2=0x7fa54819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ffd80 0x7fa54819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547400640 1 -- 172.21.15.46:0/1416397511 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54819ed20 con 0x7fa5480ff3c0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.644+0000 7fa547e00640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa547400640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 0x7fa548199d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e9f0 tx=0x7fa53800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa544c00640 1 -- 172.21.15.46:0/1416397511 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53800ccc0 con 0x7fa5480ff3c0 2024-03-08T23:08:08.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa544c00640 1 -- 172.21.15.46:0/1416397511 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa538010910 con 0x7fa5480ff3c0 2024-03-08T23:08:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54819f010 con 0x7fa5480ff3c0 2024-03-08T23:08:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa544c00640 1 -- 172.21.15.46:0/1416397511 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538018960 con 0x7fa5480ff3c0 2024-03-08T23:08:08.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.645+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5481a68a0 con 0x7fa5480ff3c0 2024-03-08T23:08:08.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 Processor -- start 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa544c00640 1 -- 172.21.15.46:0/1416397511 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fa538010ac0 con 0x7fa5480ff3c0 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 -- start start 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a991730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a99a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a8f5dd0 con 0x55877a991c70 2024-03-08T23:08:08.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a89ec70 con 0x55877a99a850 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa55157a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a8f1e70 con 0x55877a991350 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa53fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa547400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a99a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa53fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37706/0 (socket says 172.21.15.46:37706) 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa53fe00640 1 -- 172.21.15.46:0/2833752917 learned_addr learned my addr 172.21.15.46:0/2833752917 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.649+0000 7fa547e00640 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a991730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 -- 172.21.15.46:0/2833752917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 msgr2=0x55877a991730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a991730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 -- 172.21.15.46:0/2833752917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 msgr2=0x55877a99a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a99a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 -- 172.21.15.46:0/2833752917 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877a9582b0 con 0x55877a99a850 2024-03-08T23:08:08.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa53fe00640 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa530009900 tx=0x7fa530005be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feba07a5d3b3ff68 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.650+0000 7fa544c00640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 0x7fa52c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa53cc00640 1 -- 172.21.15.46:0/2833752917 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530045070 con 0x55877a99a850 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa544c00640 1 -- 172.21.15.46:0/1416397511 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa5380cf8a0 con 0x7fa5480ff3c0 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa53cc00640 1 -- 172.21.15.46:0/2833752917 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa53002c650 con 0x55877a99a850 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa53cc00640 1 -- 172.21.15.46:0/2833752917 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa530004b20 con 0x55877a99a850 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa547e00640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 0x7fa52c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa55157a740 1 -- 172.21.15.46:0/2833752917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 msgr2=0x55877a99cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa55157a740 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa530009900 tx=0x7fa530005be0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.651+0000 7fa547e00640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 0x7fa52c0a4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5400049d0 tx=0x7fa540009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 -- 172.21.15.46:0/2833752917 shutdown_connections 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a99cc40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a99a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 --2- 172.21.15.46:0/2833752917 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a991730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 -- 172.21.15.46:0/2833752917 >> 172.21.15.46:0/2833752917 conn(0x55877a98ba80 msgr2=0x55877a98be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 -- 172.21.15.46:0/2833752917 shutdown_connections 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 -- 172.21.15.46:0/2833752917 wait complete. 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 -- start start 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.652+0000 7fa55157a740 1 Processor -- start 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 0x55877a997470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a9979b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a99d5e0 con 0x55877a991350 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a8b20c0 con 0x55877a99a850 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55877a99d760 con 0x55877a991c70 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa55157a740 10 client.0 _ll_register_callbacks cb 0x55877a8ecec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa547400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa547400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56450/0 (socket says 172.21.15.46:56450) 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa547400640 1 -- 172.21.15.46:0/3825130301 learned_addr learned my addr 172.21.15.46:0/3825130301 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 0x55877a997470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.653+0000 7fa547e00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a9979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 -- 172.21.15.46:0/3825130301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 msgr2=0x55877a997470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 0x55877a997470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 -- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 msgr2=0x55877a9979b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a9979b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 -- 172.21.15.46:0/3825130301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55877a60a3f0 con 0x55877a991350 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 0x55877a997470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547e00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a9979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa547400640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa53809b930 tx=0x7fa538097c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=474f6afa40b42a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa535600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380d2e60 con 0x55877a991350 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa5380862b0 con 0x55877a991350 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa53002c650 con 0x55877a991350 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.654+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa53809b300 con 0x55877a991350 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 10 client.15597 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55877a6d47d0 con 0x55877a991350 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa53002c650 con 0x55877a991350 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa53809adf0 con 0x55877a991350 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 10 client.15597 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 10 client.15597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 20 client.15597 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877a9aa6f0 0x55877a9acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 4 client.15597 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.655+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55877a8ca4f0 con 0x55877a9aa6f0 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.656+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877a9aa6f0 0x55877a9acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.656+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877a9aa6f0 0x55877a9acae0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9688569cc7bdc656 server_cookie=2ba61ebf9fc191ed in_seq=0 out_seq=0 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.656+0000 7fa535600640 10 client.15597 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa535600640 1 -- 172.21.15.46:0/3825130301 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55877a8ca4f0 con 0x55877a9aa6f0 2024-03-08T23:08:08.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa535600640 10 client.15597 handle_command_reply: tid=0 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 client.15597 shutdown 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 msgr2=0x55877a99a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fa53809b930 tx=0x7fa538097c40 comp rx=0 tx=0).stop 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 shutdown_connections 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877a9aa6f0 0x55877a9acae0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a99a850 0x55877a9979b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991c70 0x55877a997470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 --2- 172.21.15.46:0/3825130301 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991350 0x55877a99a0e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa535600640 0 client.15597 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 >> 172.21.15.46:0/3825130301 conn(0x55877a98ba80 msgr2=0x55877a98d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 shutdown_connections 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 1 -- 172.21.15.46:0/3825130301 wait complete. 2024-03-08T23:08:08.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.657+0000 7fa55157a740 20 client.15597 trim_cache size 0 max 16384 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 Processor -- start 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 -- start start 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a996450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991c70 0x55877a999ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 0x55877a99a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481b2950 con 0x55877a99a850 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481b04f0 con 0x55877a991c70 2024-03-08T23:08:08.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa55157a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481ae590 con 0x55877a991350 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa547e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 0x55877a99a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa547400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a996450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa547e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 0x55877a99a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56464/0 (socket says 172.21.15.46:56464) 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa547e00640 1 -- 172.21.15.46:0/3891546317 learned_addr learned my addr 172.21.15.46:0/3891546317 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.671+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991c70 0x55877a999ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 -- 172.21.15.46:0/3891546317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991c70 msgr2=0x55877a999ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991c70 0x55877a999ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 -- 172.21.15.46:0/3891546317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 msgr2=0x55877a99a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 0x55877a99a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 -- 172.21.15.46:0/3891546317 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55877a991aa0 con 0x55877a991350 2024-03-08T23:08:08.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa547400640 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a996450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa538098370 tx=0x7fa53800c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38ab5de19294451c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa537e00640 1 -- 172.21.15.46:0/3891546317 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53809b9a0 con 0x55877a991350 2024-03-08T23:08:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa537e00640 1 -- 172.21.15.46:0/3891546317 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa53809fa40 con 0x55877a991350 2024-03-08T23:08:08.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.672+0000 7fa537e00640 1 -- 172.21.15.46:0/3891546317 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa538086320 con 0x55877a991350 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 -- 172.21.15.46:0/3891546317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 msgr2=0x55877a996450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a996450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa538098370 tx=0x7fa53800c7d0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 -- 172.21.15.46:0/3891546317 shutdown_connections 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a99a850 0x55877a99a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991c70 0x55877a999ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 --2- 172.21.15.46:0/3891546317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a991350 0x55877a996450 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 -- 172.21.15.46:0/3891546317 >> 172.21.15.46:0/3891546317 conn(0x55877a98ba80 msgr2=0x55877a995d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 -- 172.21.15.46:0/3891546317 shutdown_connections 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.673+0000 7fa55157a740 1 -- 172.21.15.46:0/3891546317 wait complete. 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 -- start start 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 Processor -- start 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 0x55877a9a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a9a01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481aeb90 con 0x55877a991c70 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481afe90 con 0x55877a991350 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5481af1d0 con 0x55877a99a850 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.674+0000 7fa55157a740 10 client.0 _ll_register_callbacks cb 0x7fa538005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50612/0 (socket says 172.21.15.46:50612) 2024-03-08T23:08:08.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 0x55877a9a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547e00640 1 -- 172.21.15.46:0/3492701870 learned_addr learned my addr 172.21.15.46:0/3492701870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a9a01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 -- 172.21.15.46:0/3492701870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 msgr2=0x55877a9a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 -- 172.21.15.46:0/3492701870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 msgr2=0x55877a9a01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a9a01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 -- 172.21.15.46:0/3492701870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55877a9a1020 con 0x55877a991350 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a9a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547e00640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa547400640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 0x55877a9a1e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa538000c00 tx=0x7fa53800c840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=398bbf60a554b35e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa535600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.675+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5380b8020 con 0x55877a991350 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa538010630 con 0x55877a991350 2024-03-08T23:08:08.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa53809fa40 con 0x55877a991350 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa53809adf0 con 0x55877a991350 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 10 client.25721 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55877aa01ce0 con 0x55877a991350 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa53809fa40 con 0x55877a991350 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa538097880 con 0x55877a991350 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 10 client.25721 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 20 client.25721 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877aa02110 0x55877aa04500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 4 client.25721 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.676+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55877a903310 con 0x55877aa02110 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877aa02110 0x55877aa04500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa53fe00640 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877aa02110 0x55877aa04500 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb013cff9096238c server_cookie=8670edbea64f8a03 in_seq=0 out_seq=0 2024-03-08T23:08:08.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa535600640 10 client.25721 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa535600640 1 -- 172.21.15.46:0/3492701870 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55877a903310 con 0x55877aa02110 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa535600640 10 client.25721 handle_command_reply: tid=0 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.677+0000 7fa55157a740 1 client.25721 shutdown 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 msgr2=0x55877a9a1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 0x55877a9a1e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa538000c00 tx=0x7fa53800c840 comp rx=0 tx=0).stop 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 shutdown_connections 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55877aa02110 0x55877aa04500 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55877a99a850 0x55877a9a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55877a991c70 0x55877a9a01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 --2- 172.21.15.46:0/3492701870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55877a991350 0x55877a9a1e50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa535600640 0 client.25721 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:08.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 >> 172.21.15.46:0/3492701870 conn(0x55877a98ba80 msgr2=0x55877a9965d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 shutdown_connections 2024-03-08T23:08:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 1 -- 172.21.15.46:0/3492701870 wait complete. 2024-03-08T23:08:08.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.678+0000 7fa55157a740 20 client.25721 trim_cache size 0 max 16384 2024-03-08T23:08:08.686 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 msgr2=0x7fa52c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 0x7fa52c0a4f50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa5400049d0 tx=0x7fa540009500 comp rx=0 tx=0).stop 2024-03-08T23:08:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 msgr2=0x7fa548199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:08.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 0x7fa548199d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa53800e9f0 tx=0x7fa53800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 shutdown_connections 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa52c0a2a90 0x7fa52c0a4f50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa5481073d0 0x7fa54819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa5480ffd80 0x7fa54819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 --2- 172.21.15.46:0/1416397511 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa5480ff3c0 0x7fa548199d30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 >> 172.21.15.46:0/1416397511 conn(0x7fa548076340 msgr2=0x7fa5480fd880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 shutdown_connections 2024-03-08T23:08:08.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:08.679+0000 7fa54e800640 1 -- 172.21.15.46:0/1416397511 wait complete. 2024-03-08T23:08:09.700 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:08:09.700 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:08:09.701 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:08:09.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 -- 172.21.15.46:0/2614323052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41086b0 msgr2=0x7f61e4110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 --2- 172.21.15.46:0/2614323052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41086b0 0x7f61e4110280 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009900 tx=0x7f61dc02bc40 comp rx=0 tx=0).stop 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 -- 172.21.15.46:0/2614323052 shutdown_connections 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 --2- 172.21.15.46:0/2614323052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41086b0 0x7f61e4110280 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 --2- 172.21.15.46:0/2614323052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4103f90 0x7f61e4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 --2- 172.21.15.46:0/2614323052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41035e0 0x7f61e41039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.945+0000 7f61e3e00640 1 -- 172.21.15.46:0/2614323052 >> 172.21.15.46:0/2614323052 conn(0x7f61e40fee20 msgr2=0x7f61e4101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.946+0000 7f61e3e00640 1 -- 172.21.15.46:0/2614323052 shutdown_connections 2024-03-08T23:08:09.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.946+0000 7f61e3e00640 1 -- 172.21.15.46:0/2614323052 wait complete. 2024-03-08T23:08:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.946+0000 7f61e3e00640 1 Processor -- start 2024-03-08T23:08:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 -- start start 2024-03-08T23:08:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41086b0 0x7f61e419fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 0x7f61e4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41137a0 con 0x7f61e41086b0 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4113620 con 0x7f61e4199a60 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e4113920 con 0x7f61e41035e0 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41086b0 0x7f61e419fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50642/0 (socket says 172.21.15.46:50642) 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e2a00640 1 -- 172.21.15.46:0/1635408956 learned_addr learned my addr 172.21.15.46:0/1635408956 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.947+0000 7f61e3400640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 0x7f61e4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 -- 172.21.15.46:0/1635408956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 msgr2=0x7f61e4199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 0x7f61e4199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 -- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41086b0 msgr2=0x7f61e419fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41086b0 0x7f61e419fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 -- 172.21.15.46:0/1635408956 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e419a580 con 0x7f61e41035e0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e3400640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 0x7f61e4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009d50 tx=0x7f61d400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61db400640 1 -- 172.21.15.46:0/1635408956 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4003b70 con 0x7f61e41035e0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61db400640 1 -- 172.21.15.46:0/1635408956 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d4003d10 con 0x7f61e41035e0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61db400640 1 -- 172.21.15.46:0/1635408956 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4010a20 con 0x7f61e41035e0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.948+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e419a870 con 0x7f61e41035e0 2024-03-08T23:08:09.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.949+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e41a6ac0 con 0x7f61e41035e0 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 Processor -- start 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61db400640 1 -- 172.21.15.46:0/1635408956 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f61d4003ec0 con 0x7f61e41035e0 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 -- start start 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f1fba50 con 0x55b11f2977a0 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f1a4b00 con 0x55b11f2980c0 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f1f7af0 con 0x55b11f2a0ca0 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61e3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56492/0 (socket says 172.21.15.46:56492) 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.952+0000 7f61e3400640 1 -- 172.21.15.46:0/616951182 learned_addr learned my addr 172.21.15.46:0/616951182 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2a00640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 -- 172.21.15.46:0/616951182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 msgr2=0x55b11f2a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 -- 172.21.15.46:0/616951182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 msgr2=0x55b11f297b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 -- 172.21.15.46:0/616951182 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b11f1c21e0 con 0x55b11f2a0ca0 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e3400640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.953+0000 7f61e2000640 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a3090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61e419b250 tx=0x7f61cc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd771b19f74f972c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61db400640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 0x7f61c80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61d3e00640 1 -- 172.21.15.46:0/616951182 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc004050 con 0x55b11f2a0ca0 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61d3e00640 1 -- 172.21.15.46:0/616951182 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61cc005500 con 0x55b11f2a0ca0 2024-03-08T23:08:09.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61d3e00640 1 -- 172.21.15.46:0/616951182 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc004d20 con 0x55b11f2a0ca0 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61db400640 1 -- 172.21.15.46:0/1635408956 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f61d40cfc30 con 0x7f61e41035e0 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61e3400640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 0x7f61c80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61ec3f1740 1 -- 172.21.15.46:0/616951182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 msgr2=0x55b11f2a3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.954+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a3090 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f61e419b250 tx=0x7f61cc011810 comp rx=0 tx=0).stop 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 -- 172.21.15.46:0/616951182 shutdown_connections 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a3090 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/616951182 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f297b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 -- 172.21.15.46:0/616951182 >> 172.21.15.46:0/616951182 conn(0x55b11f291ed0 msgr2=0x55b11f2922b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61e3400640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 0x7f61c80a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f61dc02efd0 tx=0x7f61dc02cf90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 -- 172.21.15.46:0/616951182 shutdown_connections 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 -- 172.21.15.46:0/616951182 wait complete. 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 -- start start 2024-03-08T23:08:09.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.955+0000 7f61ec3f1740 1 Processor -- start 2024-03-08T23:08:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29e190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f2a3a90 con 0x55b11f2977a0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f1b7f60 con 0x55b11f2980c0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b11f2a3c10 con 0x55b11f2a0ca0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61ec3f1740 10 client.0 _ll_register_callbacks cb 0x55b11f1a3b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56498/0 (socket says 172.21.15.46:56498) 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61e2a00640 1 -- 172.21.15.46:0/1832380552 learned_addr learned my addr 172.21.15.46:0/1832380552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61e2000640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.956+0000 7f61e3400640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 -- 172.21.15.46:0/1832380552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 msgr2=0x55b11f29e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 -- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 msgr2=0x55b11f29e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 -- 172.21.15.46:0/1832380552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b11efd87e0 con 0x55b11f2977a0 2024-03-08T23:08:09.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2000640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61e2a00640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f61d400cba0 tx=0x7f61d4097800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5927ddc92c07e9b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d409b7c0 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f61d4010bc0 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61cc005500 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d409b5e0 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.957+0000 7f61ec3f1740 10 client.15615 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b11f038f70 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61cc005500 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f61d40d8070 con 0x55b11f2977a0 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 10 client.15615 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 10 client.15615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:09.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 20 client.15615 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f2b1000 0x55b11f2b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 4 client.15615 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.958+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b11f208f90 con 0x55b11f2b1000 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.959+0000 7f61e2000640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f2b1000 0x55b11f2b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.959+0000 7f61e2000640 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f2b1000 0x55b11f2b33f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=511c27945a65c839 server_cookie=70bb7ec08810e106 in_seq=0 out_seq=0 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.959+0000 7f61d0c00640 10 client.15615 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61d0c00640 1 -- 172.21.15.46:0/1832380552 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b11f208f90 con 0x55b11f2b1000 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61d0c00640 10 client.15615 handle_command_reply: tid=0 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 client.15615 shutdown 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 msgr2=0x55b11f29dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f61d400cba0 tx=0x7f61d4097800 comp rx=0 tx=0).stop 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 shutdown_connections 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f2b1000 0x55b11f2b33f0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/1832380552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f29dc50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61d0c00640 0 client.15615 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 >> 172.21.15.46:0/1832380552 conn(0x55b11f291ed0 msgr2=0x55b11f293c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 shutdown_connections 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.960+0000 7f61ec3f1740 1 -- 172.21.15.46:0/1832380552 wait complete. 2024-03-08T23:08:09.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.961+0000 7f61ec3f1740 20 client.15615 trim_cache size 0 max 16384 2024-03-08T23:08:09.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.977+0000 7f61ec3f1740 1 Processor -- start 2024-03-08T23:08:09.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.977+0000 7f61ec3f1740 1 -- start start 2024-03-08T23:08:09.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41b2a80 con 0x55b11f2977a0 2024-03-08T23:08:09.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41b0620 con 0x55b11f2980c0 2024-03-08T23:08:09.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41ae6c0 con 0x55b11f2a0ca0 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37784/0 (socket says 172.21.15.46:37784) 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 -- 172.21.15.46:0/3960477189 learned_addr learned my addr 172.21.15.46:0/3960477189 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2a00640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 -- 172.21.15.46:0/3960477189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 msgr2=0x55b11f29aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e3400640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 -- 172.21.15.46:0/3960477189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 msgr2=0x55b11f2a0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2000640 1 -- 172.21.15.46:0/3960477189 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b11f297ef0 con 0x55b11f2980c0 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e2a00640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.978+0000 7f61e3400640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.979+0000 7f61e2000640 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55b11f293060 tx=0x7f61cc004450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfe973108f61625 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.979+0000 7f61d3400640 1 -- 172.21.15.46:0/3960477189 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc01d040 con 0x55b11f2980c0 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.979+0000 7f61d3400640 1 -- 172.21.15.46:0/3960477189 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61cc02a7d0 con 0x55b11f2980c0 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.979+0000 7f61d3400640 1 -- 172.21.15.46:0/3960477189 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc003970 con 0x55b11f2980c0 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- 172.21.15.46:0/3960477189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 msgr2=0x55b11f29d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55b11f293060 tx=0x7f61cc004450 comp rx=0 tx=0).stop 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- 172.21.15.46:0/3960477189 shutdown_connections 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f29aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f29d530 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/3960477189 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- 172.21.15.46:0/3960477189 >> 172.21.15.46:0/3960477189 conn(0x55b11f291ed0 msgr2=0x55b11f290f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- 172.21.15.46:0/3960477189 shutdown_connections 2024-03-08T23:08:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- 172.21.15.46:0/3960477189 wait complete. 2024-03-08T23:08:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 -- start start 2024-03-08T23:08:09.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.980+0000 7f61ec3f1740 1 Processor -- start 2024-03-08T23:08:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a75e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41aecc0 con 0x55b11f2977a0 2024-03-08T23:08:09.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41affc0 con 0x55b11f2980c0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e41af300 con 0x55b11f2a0ca0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61ec3f1740 10 client.0 _ll_register_callbacks cb 0x7f61cc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56518/0 (socket says 172.21.15.46:56518) 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 -- 172.21.15.46:0/2996989378 learned_addr learned my addr 172.21.15.46:0/2996989378 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e3400640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 -- 172.21.15.46:0/2996989378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 msgr2=0x55b11f2a75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 -- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 msgr2=0x55b11f2a7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e2a00640 1 -- 172.21.15.46:0/2996989378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b11eed0170 con 0x55b11f2977a0 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.981+0000 7f61e3400640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:09.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61e2a00640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f61d409bc10 tx=0x7f61d4005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=463f0b589108dfbc server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40ba530 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f61d40041b0 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61cc02a7d0 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d40ccd80 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61ec3f1740 10 client.15621 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55b11f1c7050 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.982+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61cc02a7d0 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f61d40d8070 con 0x55b11f2977a0 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 10 client.15621 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 10 client.15621 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 20 client.15621 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f311360 0x55b11f313750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:09.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 4 client.15621 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:08:09.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55b11f2032e0 con 0x55b11f311360 2024-03-08T23:08:09.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.983+0000 7f61e2000640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f311360 0x55b11f313750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:09.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.984+0000 7f61e2000640 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f311360 0x55b11f313750 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1048eb020559192 server_cookie=5aacdaa529db47f3 in_seq=0 out_seq=0 2024-03-08T23:08:09.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.984+0000 7f61d0c00640 10 client.15621 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.985+0000 7f61d0c00640 1 -- 172.21.15.46:0/2996989378 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55b11f2032e0 con 0x55b11f311360 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.985+0000 7f61d0c00640 10 client.15621 handle_command_reply: tid=0 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.985+0000 7f61ec3f1740 1 client.15621 shutdown 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 msgr2=0x55b11f2a6b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f61d409bc10 tx=0x7f61d4005e90 comp rx=0 tx=0).stop 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 shutdown_connections 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b11f311360 0x55b11f313750 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b11f2a0ca0 0x55b11f2a75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b11f2980c0 0x55b11f2a7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 --2- 172.21.15.46:0/2996989378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b11f2977a0 0x55b11f2a6b20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61d0c00640 0 client.15621 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 >> 172.21.15.46:0/2996989378 conn(0x55b11f291ed0 msgr2=0x55b11f2906e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 shutdown_connections 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 1 -- 172.21.15.46:0/2996989378 wait complete. 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.986+0000 7f61ec3f1740 20 client.15621 trim_cache size 0 max 16384 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "cd10f75e-ed1c-43dd-bdd1-16800930e3ec", 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:08:09.992 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:09.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 msgr2=0x7f61c80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 0x7f61c80a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f61dc02efd0 tx=0x7f61dc02cf90 comp rx=0 tx=0).stop 2024-03-08T23:08:09.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 msgr2=0x7f61e419f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:09.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f61d4009d50 tx=0x7f61d400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 shutdown_connections 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f61c80a2a90 0x7f61c80a4f50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61e4199a60 0x7f61e4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61e41086b0 0x7f61e419fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 --2- 172.21.15.46:0/1635408956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e41035e0 0x7f61e419f8b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 >> 172.21.15.46:0/1635408956 conn(0x7f61e40fee20 msgr2=0x7f61e40ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 shutdown_connections 2024-03-08T23:08:09.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:09.987+0000 7f61e3e00640 1 -- 172.21.15.46:0/1635408956 wait complete. 2024-03-08T23:08:10.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:08:10.007 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:08:10.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.215+0000 7f9887e00640 1 -- 172.21.15.46:0/3355930436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98881035e0 msgr2=0x7f98881039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.215+0000 7f9887e00640 1 --2- 172.21.15.46:0/3355930436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98881035e0 0x7f98881039c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9878009860 tx=0x7f987802bb70 comp rx=0 tx=0).stop 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 -- 172.21.15.46:0/3355930436 shutdown_connections 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 --2- 172.21.15.46:0/3355930436 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f9888110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 --2- 172.21.15.46:0/3355930436 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9888103f90 0x7f9888107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 --2- 172.21.15.46:0/3355930436 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f98881035e0 0x7f98881039c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 -- 172.21.15.46:0/3355930436 >> 172.21.15.46:0/3355930436 conn(0x7f98880fee20 msgr2=0x7f9888101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 -- 172.21.15.46:0/3355930436 shutdown_connections 2024-03-08T23:08:10.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.216+0000 7f9887e00640 1 -- 172.21.15.46:0/3355930436 wait complete. 2024-03-08T23:08:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.217+0000 7f9887e00640 1 Processor -- start 2024-03-08T23:08:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.217+0000 7f9887e00640 1 -- start start 2024-03-08T23:08:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 0x7f988819fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f988819ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9888113760 con 0x7f98881086b0 2024-03-08T23:08:10.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881135e0 con 0x7f9888199b20 2024-03-08T23:08:10.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881138e0 con 0x7f98881035e0 2024-03-08T23:08:10.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9886a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 0x7f988819fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37806/0 (socket says 172.21.15.46:37806) 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9886a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 0x7f988819fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:50702/0 (socket says 172.21.15.46:50702) 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 -- 172.21.15.46:0/195597907 learned_addr learned my addr 172.21.15.46:0/195597907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f987e000640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f988819ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 -- 172.21.15.46:0/195597907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 msgr2=0x7f988819fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 0x7f988819fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.218+0000 7f9887400640 1 -- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 msgr2=0x7f988819ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f9887400640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f988819ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f9887400640 1 -- 172.21.15.46:0/195597907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988819a720 con 0x7f9888199b20 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f987e000640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f988819ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f9887400640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f988000e970 tx=0x7f988000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f987fe00640 1 -- 172.21.15.46:0/195597907 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988000ccc0 con 0x7f9888199b20 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f987fe00640 1 -- 172.21.15.46:0/195597907 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f988000ce60 con 0x7f9888199b20 2024-03-08T23:08:10.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f987fe00640 1 -- 172.21.15.46:0/195597907 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880010490 con 0x7f9888199b20 2024-03-08T23:08:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98881a6800 con 0x7f9888199b20 2024-03-08T23:08:10.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.219+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98881a6bb0 con 0x7f9888199b20 2024-03-08T23:08:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.222+0000 7f98909b0740 1 Processor -- start 2024-03-08T23:08:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f987fe00640 1 -- 172.21.15.46:0/195597907 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f9880005020 con 0x7f9888199b20 2024-03-08T23:08:10.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 -- start start 2024-03-08T23:08:10.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x55656612c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55656612cc90 0x55656612f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565660e5410 con 0x5565661240b0 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556566087d50 con 0x556566123790 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f98909b0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x556566030b80 con 0x55656612cc90 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f987e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f987e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37822/0 (socket says 172.21.15.46:37822) 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.223+0000 7f987e000640 1 -- 172.21.15.46:0/3014340843 learned_addr learned my addr 172.21.15.46:0/3014340843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55656612cc90 0x55656612f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f9886a00640 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x55656612c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 -- 172.21.15.46:0/3014340843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55656612cc90 msgr2=0x55656612f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55656612cc90 0x55656612f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 -- 172.21.15.46:0/3014340843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 msgr2=0x55656612c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x55656612c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 -- 172.21.15.46:0/3014340843 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556565e667d0 con 0x556566123790 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987e000640 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9870010ba0 tx=0x7f9870009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be1bd7c6b37a04b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f987fe00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 0x7f986c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.224+0000 7f9877e00640 1 -- 172.21.15.46:0/3014340843 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98700100d0 con 0x556566123790 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f9877e00640 1 -- 172.21.15.46:0/3014340843 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9870003e50 con 0x556566123790 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f987fe00640 1 -- 172.21.15.46:0/195597907 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f98800cf900 con 0x7f9888199b20 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f9877e00640 1 -- 172.21.15.46:0/3014340843 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9870004600 con 0x556566123790 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f9886a00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 0x7f986c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 -- 172.21.15.46:0/3014340843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 msgr2=0x556566123b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9870010ba0 tx=0x7f9870009b50 comp rx=0 tx=0).stop 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 -- 172.21.15.46:0/3014340843 shutdown_connections 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55656612cc90 0x55656612f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x55656612c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 --2- 172.21.15.46:0/3014340843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556566123790 0x556566123b70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.225+0000 7f98909b0740 1 -- 172.21.15.46:0/3014340843 >> 172.21.15.46:0/3014340843 conn(0x55656611dec0 msgr2=0x55656611e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.226+0000 7f9886a00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 0x7f986c0a4fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f987802c180 tx=0x7f98780056d0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.226+0000 7f98909b0740 1 -- 172.21.15.46:0/3014340843 shutdown_connections 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.226+0000 7f98909b0740 1 -- 172.21.15.46:0/3014340843 wait complete. 2024-03-08T23:08:10.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.226+0000 7f98909b0740 1 -- start start 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.226+0000 7f98909b0740 1 Processor -- start 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566129d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55656612cc90 0x55656612a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556566043ee0 con 0x55656612cc90 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556566083df0 con 0x5565661240b0 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656612fa20 con 0x556566123790 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f98909b0740 10 client.0 _ll_register_callbacks cb 0x55656604a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37826/0 (socket says 172.21.15.46:37826) 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 -- 172.21.15.46:0/3465710274 learned_addr learned my addr 172.21.15.46:0/3465710274 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f9886a00640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566129d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f9887400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55656612cc90 0x55656612a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 -- 172.21.15.46:0/3465710274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 msgr2=0x556566129d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566129d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 -- 172.21.15.46:0/3465710274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55656612cc90 msgr2=0x55656612a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.227+0000 7f987e000640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55656612cc90 0x55656612a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f987e000640 1 -- 172.21.15.46:0/3465710274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556565d9c3f0 con 0x5565661240b0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f9886a00640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566129d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f987e000640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9870009c30 tx=0x7f987000def0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95dc2009f608c627 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f9874c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98700043c0 con 0x5565661240b0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f9870004560 con 0x5565661240b0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9870003e50 con 0x5565661240b0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9870030050 con 0x5565661240b0 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f98909b0740 10 client.25751 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:10.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.228+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x556565d5c0f0 con 0x5565661240b0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9870003e50 con 0x5565661240b0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f987000b040 con 0x5565661240b0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 10 client.25751 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 10 client.25751 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 20 client.25751 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55656613cda0 0x55656613f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 4 client.25751 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556566095290 con 0x55656613cda0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.229+0000 7f9886a00640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55656613cda0 0x55656613f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.230+0000 7f9886a00640 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55656613cda0 0x55656613f190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48f6a79b4f139f3 server_cookie=61ae238958870d0a in_seq=0 out_seq=0 2024-03-08T23:08:10.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.230+0000 7f9874c00640 10 client.25751 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f9874c00640 1 -- 172.21.15.46:0/3465710274 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556566095290 con 0x55656613cda0 2024-03-08T23:08:10.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f9874c00640 10 client.25751 handle_command_reply: tid=0 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 client.25751 shutdown 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 msgr2=0x55656612a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9870009c30 tx=0x7f987000def0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 shutdown_connections 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55656613cda0 0x55656613f190 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55656612cc90 0x55656612a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5565661240b0 0x55656612a2d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 --2- 172.21.15.46:0/3465710274 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566129d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 >> 172.21.15.46:0/3465710274 conn(0x55656611dec0 msgr2=0x55656611fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f9874c00640 0 client.25751 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 shutdown_connections 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.231+0000 7f98909b0740 1 -- 172.21.15.46:0/3465710274 wait complete. 2024-03-08T23:08:10.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.232+0000 7f98909b0740 20 client.25751 trim_cache size 0 max 16384 2024-03-08T23:08:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.248+0000 7f98909b0740 1 Processor -- start 2024-03-08T23:08:10.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.248+0000 7f98909b0740 1 -- start start 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.248+0000 7f98909b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.248+0000 7f98909b0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 0x556566133c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f98909b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f98909b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881b29c0 con 0x556566123790 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f98909b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881b0560 con 0x55656612cc90 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f98909b0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881ae600 con 0x5565661240b0 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56556/0 (socket says 172.21.15.46:56556) 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 -- 172.21.15.46:0/198227619 learned_addr learned my addr 172.21.15.46:0/198227619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f987e000640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 0x556566133c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9887400640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 -- 172.21.15.46:0/198227619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 msgr2=0x556566133c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 0x556566133c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 -- 172.21.15.46:0/198227619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 msgr2=0x5565661341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661341a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9886a00640 1 -- 172.21.15.46:0/198227619 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556566122ab0 con 0x556566123790 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f987e000640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 0x556566133c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.249+0000 7f9887400640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:10.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f9886a00640 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9878009860 tx=0x7f987802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8892b11a8b0a18cb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f9877400640 1 -- 172.21.15.46:0/198227619 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878055070 con 0x556566123790 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f9877400640 1 -- 172.21.15.46:0/198227619 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9878044040 con 0x556566123790 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f9877400640 1 -- 172.21.15.46:0/198227619 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878043040 con 0x556566123790 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f98909b0740 1 -- 172.21.15.46:0/198227619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 msgr2=0x55656612c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.250+0000 7f98909b0740 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f9878009860 tx=0x7f987802e000 comp rx=0 tx=0).stop 2024-03-08T23:08:10.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 -- 172.21.15.46:0/198227619 shutdown_connections 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5565661240b0 0x556566133c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 --2- 172.21.15.46:0/198227619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556566123790 0x55656612c4c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 -- 172.21.15.46:0/198227619 >> 172.21.15.46:0/198227619 conn(0x55656611dec0 msgr2=0x556566143940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 -- 172.21.15.46:0/198227619 shutdown_connections 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 -- 172.21.15.46:0/198227619 wait complete. 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 -- start start 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.251+0000 7f98909b0740 1 Processor -- start 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566187050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661920d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881aec00 con 0x5565661240b0 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881aff00 con 0x55656612cc90 2024-03-08T23:08:10.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881af240 con 0x556566123790 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f98909b0740 10 client.0 _ll_register_callbacks cb 0x7f9870009da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f987e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f987e000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56560/0 (socket says 172.21.15.46:56560) 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f987e000640 1 -- 172.21.15.46:0/3030611924 learned_addr learned my addr 172.21.15.46:0/3030611924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f9887400640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661920d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.252+0000 7f9886a00640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566187050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 -- 172.21.15.46:0/3030611924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 msgr2=0x556566187050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566187050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 -- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 msgr2=0x5565661920d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661920d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 -- 172.21.15.46:0/3030611924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556566192610 con 0x5565661240b0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f987e000640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55656612b360 tx=0x7f9870009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=857d3f2dad8a5ce server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:10.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f9874c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987002c020 con 0x5565661240b0 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9878044040 con 0x5565661240b0 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.253+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f9870004be0 con 0x5565661240b0 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f987003d050 con 0x5565661240b0 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 10 client.15639 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:10.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5565661937c0 con 0x5565661240b0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9878044040 con 0x5565661240b0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f987000b040 con 0x5565661240b0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 10 client.15639 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 10 client.15639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 20 client.15639 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556566193ec0 0x5565661962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 4 client.15639 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55656608f5e0 con 0x556566193ec0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.254+0000 7f9886a00640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556566193ec0 0x5565661962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f9886a00640 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556566193ec0 0x5565661962e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa095278ebe87d95 server_cookie=8f0520c277415df8 in_seq=0 out_seq=0 2024-03-08T23:08:10.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f9874c00640 10 client.15639 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f9874c00640 1 -- 172.21.15.46:0/3030611924 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55656608f5e0 con 0x556566193ec0 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f9874c00640 10 client.15639 handle_command_reply: tid=0 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f98909b0740 1 client.15639 shutdown 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 msgr2=0x556566187590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.255+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x55656612b360 tx=0x7f9870009630 comp rx=0 tx=0).stop 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 shutdown_connections 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556566193ec0 0x5565661962e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55656612cc90 0x5565661920d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5565661240b0 0x556566187590 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 --2- 172.21.15.46:0/3030611924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556566123790 0x556566187050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f9874c00640 0 client.15639 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 >> 172.21.15.46:0/3030611924 conn(0x55656611dec0 msgr2=0x556566125070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 shutdown_connections 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 1 -- 172.21.15.46:0/3030611924 wait complete. 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f98909b0740 20 client.15639 trim_cache size 0 max 16384 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "cd10f75e-ed1c-43dd-bdd1-16800930e3ec": { 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "cd10f75e-ed1c-43dd-bdd1-16800930e3ec", 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:08:10.263 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:08:10.264 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:08:10.264 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:10.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 msgr2=0x7f986c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 0x7f986c0a4fd0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f987802c180 tx=0x7f98780056d0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 msgr2=0x7f9888199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.256+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f988000e970 tx=0x7f988000ed70 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 shutdown_connections 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f986c0a2b10 0x7f986c0a4fd0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9888199b20 0x7f9888199fa0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f98881086b0 0x7f988819ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 --2- 172.21.15.46:0/195597907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f98881035e0 0x7f988819fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 >> 172.21.15.46:0/195597907 conn(0x7f98880fee20 msgr2=0x7f9888101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 shutdown_connections 2024-03-08T23:08:10.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:10.257+0000 7f9887e00640 1 -- 172.21.15.46:0/195597907 wait complete. 2024-03-08T23:08:10.277 INFO:tasks.cephfs.filesystem:scrub status for tag:cd10f75e-ed1c-43dd-bdd1-16800930e3ec - {'path': '/', 'tag': 'cd10f75e-ed1c-43dd-bdd1-16800930e3ec', 'options': 'recursive,force'} 2024-03-08T23:08:10.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:10 smithi046 ceph-mon[31859]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-08T23:08:10.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:10 smithi192 ceph-mon[36700]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-08T23:08:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:10 smithi062 ceph-mon[35761]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 22 op/s 2024-03-08T23:08:11.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:11 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:08:11.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:11 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:11.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:11 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:08:11.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:11 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:11.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:11 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:08:11.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:11 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:11.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:11 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:08:11.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:11 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:11.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:11 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:08:11.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:11 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:11.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:11 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:08:11.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:11 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:12.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:12 smithi046 ceph-mon[31859]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:12.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:12 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:08:12.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:12 smithi192 ceph-mon[36700]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:12.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:12 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:08:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:12 smithi062 ceph-mon[35761]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-03-08T23:08:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:12 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:08:14.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:14 smithi062 ceph-mon[35761]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-08T23:08:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:14 smithi046 ceph-mon[31859]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-08T23:08:14.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:14 smithi192 ceph-mon[36700]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-03-08T23:08:16.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:16 smithi046 ceph-mon[31859]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-03-08T23:08:16.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:16 smithi192 ceph-mon[36700]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-03-08T23:08:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:16 smithi062 ceph-mon[35761]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 20 op/s 2024-03-08T23:08:18.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:18 smithi046 ceph-mon[31859]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:08:18.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:18 smithi192 ceph-mon[36700]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:08:18.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:18 smithi062 ceph-mon[35761]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:08:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:20 smithi046 ceph-mon[31859]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:08:20.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:20 smithi192 ceph-mon[36700]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:08:20.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:20 smithi062 ceph-mon[35761]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:08:22.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:22 smithi046 ceph-mon[31859]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:22.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:22 smithi192 ceph-mon[36700]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:22 smithi062 ceph-mon[35761]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:24.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:24 smithi046 ceph-mon[31859]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:24 smithi192 ceph-mon[36700]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:24 smithi062 ceph-mon[35761]: pgmap v21: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:26.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:26 smithi046 ceph-mon[31859]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-08T23:08:26.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:26 smithi192 ceph-mon[36700]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-08T23:08:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:26 smithi062 ceph-mon[35761]: pgmap v22: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 6 op/s 2024-03-08T23:08:28.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:28 smithi046 ceph-mon[31859]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:28.559 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:28 smithi192 ceph-mon[36700]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:28.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:28 smithi062 ceph-mon[35761]: pgmap v23: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:30.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:30 smithi046 ceph-mon[31859]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-08T23:08:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:30 smithi192 ceph-mon[36700]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-08T23:08:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:30 smithi062 ceph-mon[35761]: pgmap v24: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 938 B/s wr, 7 op/s 2024-03-08T23:08:32.462 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:32 smithi062 ceph-mon[35761]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-08T23:08:32.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:32 smithi046 ceph-mon[31859]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-08T23:08:32.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:32 smithi192 ceph-mon[36700]: pgmap v25: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 938 B/s wr, 5 op/s 2024-03-08T23:08:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:34 smithi046 ceph-mon[31859]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:34 smithi192 ceph-mon[36700]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:34.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:34 smithi062 ceph-mon[35761]: pgmap v26: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:36.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:36 smithi046 ceph-mon[31859]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:36.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:36 smithi192 ceph-mon[36700]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:36.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:36 smithi062 ceph-mon[35761]: pgmap v27: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:38.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:38 smithi192 ceph-mon[36700]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 937 B/s wr, 5 op/s 2024-03-08T23:08:38.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:38 smithi046 ceph-mon[31859]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 937 B/s wr, 5 op/s 2024-03-08T23:08:38.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:38 smithi062 ceph-mon[35761]: pgmap v28: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 937 B/s wr, 5 op/s 2024-03-08T23:08:40.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:08:40.279 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:08:40.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:40 smithi046 ceph-mon[31859]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 -- 172.21.15.46:0/870189996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01013d0 msgr2=0x7fdbc01017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/870189996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01013d0 0x7fdbc01017b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009860 tx=0x7fdbb002bb70 comp rx=0 tx=0).stop 2024-03-08T23:08:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 -- 172.21.15.46:0/870189996 shutdown_connections 2024-03-08T23:08:40.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/870189996 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc01064a0 0x7fdbc010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/870189996 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc0101d80 0x7fdbc0105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/870189996 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01013d0 0x7fdbc01017b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 -- 172.21.15.46:0/870189996 >> 172.21.15.46:0/870189996 conn(0x7fdbc00fcc90 msgr2=0x7fdbc00ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 -- 172.21.15.46:0/870189996 shutdown_connections 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.476+0000 7fdbc5a00640 1 -- 172.21.15.46:0/870189996 wait complete. 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 Processor -- start 2024-03-08T23:08:40.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 -- start start 2024-03-08T23:08:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 0x7fdbc00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 0x7fdbc0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc0074070 0x7fdbc01a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0111530 con 0x7fdbc01013d0 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01113b0 con 0x7fdbc0074070 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.478+0000 7fdbc5a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01116b0 con 0x7fdbc01064a0 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 0x7fdbc00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 0x7fdbc0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbc5000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc0074070 0x7fdbc01a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 0x7fdbc00735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52966/0 (socket says 172.21.15.46:52966) 2024-03-08T23:08:40.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbfe00640 1 -- 172.21.15.46:0/3789970935 learned_addr learned my addr 172.21.15.46:0/3789970935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 -- 172.21.15.46:0/3789970935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc0074070 msgr2=0x7fdbc01a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc0074070 0x7fdbc01a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 -- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 msgr2=0x7fdbc00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 0x7fdbc00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.479+0000 7fdbbf400640 1 -- 172.21.15.46:0/3789970935 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc01a6300 con 0x7fdbc01064a0 2024-03-08T23:08:40.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbbf400640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 0x7fdbc0073b30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbc006a820 tx=0x7fdbac00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbbcc00640 1 -- 172.21.15.46:0/3789970935 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00ccd0 con 0x7fdbc01064a0 2024-03-08T23:08:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbbcc00640 1 -- 172.21.15.46:0/3789970935 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbac0035f0 con 0x7fdbc01064a0 2024-03-08T23:08:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbbcc00640 1 -- 172.21.15.46:0/3789970935 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac010690 con 0x7fdbc01064a0 2024-03-08T23:08:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc01a65f0 con 0x7fdbc01064a0 2024-03-08T23:08:40.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.480+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc01a6af0 con 0x7fdbc01064a0 2024-03-08T23:08:40.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 Processor -- start 2024-03-08T23:08:40.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 -- start start 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 0x55936b16b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16bf50 0x55936b17cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b12d410 con 0x55936b16b5a0 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b0cfd50 con 0x55936b16bf50 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b078b80 con 0x55936b17d4a0 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.485+0000 7fdbbcc00640 1 -- 172.21.15.46:0/3789970935 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fdbac0037a0 con 0x7fdbc01064a0 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.486+0000 7fdbbf400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.486+0000 7fdbc5000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 0x55936b16b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.486+0000 7fdbbf400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56392/0 (socket says 172.21.15.46:56392) 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.486+0000 7fdbbf400640 1 -- 172.21.15.46:0/958330808 learned_addr learned my addr 172.21.15.46:0/958330808 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.486+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16bf50 0x55936b17cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 -- 172.21.15.46:0/958330808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16bf50 msgr2=0x55936b17cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16bf50 0x55936b17cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 -- 172.21.15.46:0/958330808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 msgr2=0x55936b16b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 0x55936b16b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 -- 172.21.15.46:0/958330808 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55936aeae7d0 con 0x55936b17d4a0 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbc5000640 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 0x55936b16b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbf400640 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0cc8d0 tx=0x7fdbac0d5f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3b6c37db91639b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbcc00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 0x7fdba40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.487+0000 7fdbbcc00640 1 -- 172.21.15.46:0/3789970935 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fdbac0cf6c0 con 0x7fdbc01064a0 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.488+0000 7fdbb5600640 1 -- 172.21.15.46:0/958330808 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac0de070 con 0x55936b17d4a0 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.488+0000 7fdbb5600640 1 -- 172.21.15.46:0/958330808 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbac009510 con 0x55936b17d4a0 2024-03-08T23:08:40.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.488+0000 7fdbb5600640 1 -- 172.21.15.46:0/958330808 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac09b2b0 con 0x55936b17d4a0 2024-03-08T23:08:40.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.488+0000 7fdbc5000640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 0x7fdba40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc5000640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 0x7fdba40a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fd0 tx=0x7fdbb8009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 -- 172.21.15.46:0/958330808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 msgr2=0x55936b17f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdbac0cc8d0 tx=0x7fdbac0d5f10 comp rx=0 tx=0).stop 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 -- 172.21.15.46:0/958330808 shutdown_connections 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b17f890 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16bf50 0x55936b17cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/958330808 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16b5a0 0x55936b16b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 -- 172.21.15.46:0/958330808 >> 172.21.15.46:0/958330808 conn(0x55936b165ec0 msgr2=0x55936b1662a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 -- 172.21.15.46:0/958330808 shutdown_connections 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.489+0000 7fdbc9c25740 1 -- 172.21.15.46:0/958330808 wait complete. 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.490+0000 7fdbc9c25740 1 -- start start 2024-03-08T23:08:40.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.490+0000 7fdbc9c25740 1 Processor -- start 2024-03-08T23:08:40.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b177470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b17d4a0 0x55936b17cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b08bee0 con 0x55936b16bf50 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b0cbdf0 con 0x55936b17d4a0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55936b16caf0 con 0x55936b16b5a0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc9c25740 10 client.0 _ll_register_callbacks cb 0x55936b0928d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbc5000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b17d4a0 0x55936b17cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56396/0 (socket says 172.21.15.46:56396) 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbbfe00640 1 -- 172.21.15.46:0/842271671 learned_addr learned my addr 172.21.15.46:0/842271671 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.491+0000 7fdbbf400640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b177470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 -- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b17d4a0 msgr2=0x55936b17cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b17d4a0 0x55936b17cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 -- 172.21.15.46:0/842271671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 msgr2=0x55936b177470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b177470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 -- 172.21.15.46:0/842271671 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55936ade43f0 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbf400640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b177470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009980 tx=0x7fdbb00095a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caa66103b7c035de server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbaa000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.492+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0043070 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbac009510 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fdbb0004450 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb004e050 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbc9c25740 10 client.25780 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55936ada40f0 con 0x55936b16b5a0 2024-03-08T23:08:40.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.493+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbac009510 con 0x55936b16b5a0 2024-03-08T23:08:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdbb0032640 con 0x55936b16b5a0 2024-03-08T23:08:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 10 client.25780 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 10 client.25780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 20 client.25780 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:40.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b195e30 0x55936b198220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 4 client.25780 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55936b0dd290 con 0x55936b195e30 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.494+0000 7fdbbf400640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b195e30 0x55936b198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.495+0000 7fdbbf400640 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b195e30 0x55936b198220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9adcc292bb63e1e3 server_cookie=86f8e2dd600b00df in_seq=0 out_seq=0 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.495+0000 7fdbaa000640 10 client.25780 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.495+0000 7fdbaa000640 1 -- 172.21.15.46:0/842271671 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55936b0dd290 con 0x55936b195e30 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.495+0000 7fdbaa000640 10 client.25780 handle_command_reply: tid=0 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.495+0000 7fdbc9c25740 1 client.25780 shutdown 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 msgr2=0x55936b176f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009980 tx=0x7fdbb00095a0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 shutdown_connections 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b195e30 0x55936b198220 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b17d4a0 0x55936b17cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b177470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/842271671 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b16b5a0 0x55936b176f30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 >> 172.21.15.46:0/842271671 conn(0x55936b165ec0 msgr2=0x55936b167c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbaa000640 0 client.25780 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 shutdown_connections 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 1 -- 172.21.15.46:0/842271671 wait complete. 2024-03-08T23:08:40.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.496+0000 7fdbc9c25740 20 client.25780 trim_cache size 0 max 16384 2024-03-08T23:08:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.515+0000 7fdbc9c25740 1 Processor -- start 2024-03-08T23:08:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.515+0000 7fdbc9c25740 1 -- start start 2024-03-08T23:08:40.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b176ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b177000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01ae6c0 con 0x55936b16bf50 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01ac260 con 0x55936b16b5a0 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01aa320 con 0x55936b17d4a0 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36264/0 (socket says 172.21.15.46:36264) 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbbfe00640 1 -- 172.21.15.46:0/1829450561 learned_addr learned my addr 172.21.15.46:0/1829450561 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbbf400640 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b176ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.516+0000 7fdbc5000640 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b177000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 -- 172.21.15.46:0/1829450561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 msgr2=0x55936b177000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b177000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 -- 172.21.15.46:0/1829450561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 msgr2=0x55936b176ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b176ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 -- 172.21.15.46:0/1829450561 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55936b16bcf0 con 0x55936b16b5a0 2024-03-08T23:08:40.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00057c0 tx=0x7fdbb0032fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbaa79963b829ad1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbb4c00640 1 -- 172.21.15.46:0/1829450561 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0056020 con 0x55936b16b5a0 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbb4c00640 1 -- 172.21.15.46:0/1829450561 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb0032640 con 0x55936b16b5a0 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.517+0000 7fdbb4c00640 1 -- 172.21.15.46:0/1829450561 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0043040 con 0x55936b16b5a0 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 -- 172.21.15.46:0/1829450561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 msgr2=0x55936b17cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00057c0 tx=0x7fdbb0032fc0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 -- 172.21.15.46:0/1829450561 shutdown_connections 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b177000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b176ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/1829450561 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b17cc20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 -- 172.21.15.46:0/1829450561 >> 172.21.15.46:0/1829450561 conn(0x55936b165ec0 msgr2=0x55936b170c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 -- 172.21.15.46:0/1829450561 shutdown_connections 2024-03-08T23:08:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.518+0000 7fdbc9c25740 1 -- 172.21.15.46:0/1829450561 wait complete. 2024-03-08T23:08:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.519+0000 7fdbc9c25740 1 -- start start 2024-03-08T23:08:40.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.519+0000 7fdbc9c25740 1 Processor -- start 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b1e6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b1e0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01aa920 con 0x55936b16bf50 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01abc00 con 0x55936b16b5a0 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc01aaf40 con 0x55936b17d4a0 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc9c25740 10 client.0 _ll_register_callbacks cb 0x7fdbac0d53e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbbfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36280/0 (socket says 172.21.15.46:36280) 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbbfe00640 1 -- 172.21.15.46:0/2684804370 learned_addr learned my addr 172.21.15.46:0/2684804370 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbc5000640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b1e0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.520+0000 7fdbbf400640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b1e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 -- 172.21.15.46:0/2684804370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 msgr2=0x55936b1e0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b1e0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 -- 172.21.15.46:0/2684804370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 msgr2=0x55936b1e6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b1e6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 -- 172.21.15.46:0/2684804370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55936b1e0a20 con 0x55936b16b5a0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbf400640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b1e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbc5000640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b1e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbbfe00640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009980 tx=0x7fdbb00045f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=399cf8a17a9b4194 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbaa000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.521+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb005f070 con 0x55936b16b5a0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdbb0032640 con 0x55936b16b5a0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fdbb0034640 con 0x55936b16b5a0 2024-03-08T23:08:40.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb006b050 con 0x55936b16b5a0 2024-03-08T23:08:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbc9c25740 10 client.25775 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55936b1ed730 con 0x55936b16b5a0 2024-03-08T23:08:40.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbb0032640 con 0x55936b16b5a0 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.522+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdbb002c8e0 con 0x55936b16b5a0 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 10 client.25775 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 10 client.25775 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 20 client.25775 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b1edc90 0x55936b1f0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 4 client.25775 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55936b0d75e0 con 0x55936b1edc90 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.523+0000 7fdbbf400640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b1edc90 0x55936b1f0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbbf400640 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b1edc90 0x55936b1f0080 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73e05295491c1ee server_cookie=4b72d6ce131ed589 in_seq=0 out_seq=0 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbaa000640 10 client.25775 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:40.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbaa000640 1 -- 172.21.15.46:0/2684804370 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55936b0d75e0 con 0x55936b1edc90 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbaa000640 10 client.25775 handle_command_reply: tid=0 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 client.25775 shutdown 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 msgr2=0x55936b1e5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009980 tx=0x7fdbb00045f0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 shutdown_connections 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55936b1edc90 0x55936b1f0080 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55936b17d4a0 0x55936b1e0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55936b16bf50 0x55936b1e6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 --2- 172.21.15.46:0/2684804370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55936b16b5a0 0x55936b1e5f20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 >> 172.21.15.46:0/2684804370 conn(0x55936b165ec0 msgr2=0x55936b164f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbaa000640 0 client.25775 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 shutdown_connections 2024-03-08T23:08:40.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.524+0000 7fdbc9c25740 1 -- 172.21.15.46:0/2684804370 wait complete. 2024-03-08T23:08:40.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc9c25740 20 client.25775 trim_cache size 0 max 16384 2024-03-08T23:08:40.531 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:40.531 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:08:40.531 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:08:40.531 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 msgr2=0x7fdba40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 0x7fdba40a4f50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdbb8009fd0 tx=0x7fdbb8009500 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 msgr2=0x7fdbc0073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 0x7fdbc0073b30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdbc006a820 tx=0x7fdbac00ee50 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.525+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 shutdown_connections 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdba40a2a90 0x7fdba40a4f50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdbc0074070 0x7fdbc01a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdbc01064a0 0x7fdbc0073b30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 --2- 172.21.15.46:0/3789970935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdbc01013d0 0x7fdbc00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 >> 172.21.15.46:0/3789970935 conn(0x7fdbc00fcc90 msgr2=0x7fdbc00fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 shutdown_connections 2024-03-08T23:08:40.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.526+0000 7fdbc5a00640 1 -- 172.21.15.46:0/3789970935 wait complete. 2024-03-08T23:08:40.552 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:08:40.552 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:08:40.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:40 smithi192 ceph-mon[36700]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:40.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:40 smithi062 ceph-mon[35761]: pgmap v29: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 937 B/s wr, 7 op/s 2024-03-08T23:08:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 -- 172.21.15.46:0/1007874090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 msgr2=0x7faf08103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 --2- 172.21.15.46:0/1007874090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf08103930 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7faef8009900 tx=0x7faef802bc40 comp rx=0 tx=0).stop 2024-03-08T23:08:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 -- 172.21.15.46:0/1007874090 shutdown_connections 2024-03-08T23:08:40.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 --2- 172.21.15.46:0/1007874090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 0x7faf081101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 --2- 172.21.15.46:0/1007874090 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08103f00 0x7faf08107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 --2- 172.21.15.46:0/1007874090 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf08103930 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 -- 172.21.15.46:0/1007874090 >> 172.21.15.46:0/1007874090 conn(0x7faf080fed90 msgr2=0x7faf081011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 -- 172.21.15.46:0/1007874090 shutdown_connections 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.747+0000 7faf07e00640 1 -- 172.21.15.46:0/1007874090 wait complete. 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 Processor -- start 2024-03-08T23:08:40.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 -- start start 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 0x7faf0819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08199a40 0x7faf08199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08113700 con 0x7faf08199a40 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08113580 con 0x7faf08103550 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.749+0000 7faf07e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf08113880 con 0x7faf08108620 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36290/0 (socket says 172.21.15.46:36290) 2024-03-08T23:08:40.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 -- 172.21.15.46:0/2137315911 learned_addr learned my addr 172.21.15.46:0/2137315911 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf07400640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08199a40 0x7faf08199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06000640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 0x7faf0819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 -- 172.21.15.46:0/2137315911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 msgr2=0x7faf0819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 0x7faf0819fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 -- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08199a40 msgr2=0x7faf08199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08199a40 0x7faf08199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 -- 172.21.15.46:0/2137315911 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0819a560 con 0x7faf08103550 2024-03-08T23:08:40.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.750+0000 7faf06a00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faef8005ab0 tx=0x7faef80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.751+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8041070 con 0x7faf08103550 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.751+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef802cca0 con 0x7faf08103550 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.751+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8003d20 con 0x7faf08103550 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.751+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0819a7f0 con 0x7faf08103550 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.751+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf081a6940 con 0x7faf08103550 2024-03-08T23:08:40.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.753+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf08104a20 con 0x7faf08103550 2024-03-08T23:08:40.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.756+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7faef8044070 con 0x7faf08103550 2024-03-08T23:08:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.757+0000 7faeff400640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 0x7faeec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:40.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.757+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7faef80f0e20 con 0x7faf08103550 2024-03-08T23:08:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.757+0000 7faf06000640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 0x7faeec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:40.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.758+0000 7faf06000640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 0x7faeec0a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf08109b70 tx=0x7faef4005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:40.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:40.762+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7faef80bd680 con 0x7faf08103550 2024-03-08T23:08:41.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.079+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf08068000 con 0x7faf08103550 2024-03-08T23:08:41.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.080+0000 7faeff400640 1 -- 172.21.15.46:0/2137315911 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11334 (secure 0 0 0) 0x7faef80bcf80 con 0x7faf08103550 2024-03-08T23:08:41.083 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:08:41.083 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:07:25.750030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:08:41.083 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-08T23:08:41.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 msgr2=0x7faeec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 0x7faeec0a4f50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf08109b70 tx=0x7faef4005f90 comp rx=0 tx=0).stop 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 msgr2=0x7faf0819f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7faef8005ab0 tx=0x7faef80042f0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 shutdown_connections 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faeec0a2a90 0x7faeec0a4f50 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faf08199a40 0x7faf08199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faf08108620 0x7faf0819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 --2- 172.21.15.46:0/2137315911 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faf08103550 0x7faf0819f840 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 >> 172.21.15.46:0/2137315911 conn(0x7faf080fed90 msgr2=0x7faf08101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.083+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 shutdown_connections 2024-03-08T23:08:41.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.084+0000 7faf07e00640 1 -- 172.21.15.46:0/2137315911 wait complete. 2024-03-08T23:08:41.110 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:41.110 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:08:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 -- 172.21.15.46:0/3452906464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43581025a0 msgr2=0x7f4358102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 --2- 172.21.15.46:0/3452906464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43581025a0 0x7f4358102980 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4348009900 tx=0x7f434802bcd0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 -- 172.21.15.46:0/3452906464 shutdown_connections 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 --2- 172.21.15.46:0/3452906464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 0x7f435810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 --2- 172.21.15.46:0/3452906464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4358102ec0 0x7f435810b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 --2- 172.21.15.46:0/3452906464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f43581025a0 0x7f4358102980 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.299+0000 7f435de00640 1 -- 172.21.15.46:0/3452906464 >> 172.21.15.46:0/3452906464 conn(0x7f4358077350 msgr2=0x7f4358077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.300+0000 7f435de00640 1 -- 172.21.15.46:0/3452906464 shutdown_connections 2024-03-08T23:08:41.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.300+0000 7f435de00640 1 -- 172.21.15.46:0/3452906464 wait complete. 2024-03-08T23:08:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 Processor -- start 2024-03-08T23:08:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 -- start start 2024-03-08T23:08:41.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 0x7f4358199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 0x7f435819a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 0x7f435819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358111710 con 0x7f4358102ec0 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358111590 con 0x7f43581025a0 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.301+0000 7f435de00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4358111890 con 0x7f435810b8e0 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 0x7f435819a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 0x7f435819a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53042/0 (socket says 172.21.15.46:53042) 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4357400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 0x7f435819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356000640 1 -- 172.21.15.46:0/4141152403 learned_addr learned my addr 172.21.15.46:0/4141152403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 0x7f4358199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 -- 172.21.15.46:0/4141152403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 msgr2=0x7f435819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 0x7f435819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 -- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 msgr2=0x7f435819a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 0x7f435819a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.302+0000 7f4356a00640 1 -- 172.21.15.46:0/4141152403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435819edf0 con 0x7f43581025a0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f4356a00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 0x7f4358199e50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f434802c210 tx=0x7f4348005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f435ca00640 1 -- 172.21.15.46:0/4141152403 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348042020 con 0x7f43581025a0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f435ca00640 1 -- 172.21.15.46:0/4141152403 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f434802ce00 con 0x7f43581025a0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435819f080 con 0x7f43581025a0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f435ca00640 1 -- 172.21.15.46:0/4141152403 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4348003a50 con 0x7f43581025a0 2024-03-08T23:08:41.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.303+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43581a6970 con 0x7f43581025a0 2024-03-08T23:08:41.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.306+0000 7f435ca00640 1 -- 172.21.15.46:0/4141152403 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f4348045020 con 0x7f43581025a0 2024-03-08T23:08:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.307+0000 7f4360a60740 1 Processor -- start 2024-03-08T23:08:41.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 -- start start 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3aa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 0x561bbf3b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf30edd0 con 0x561bbf3aa350 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf2b7c70 con 0x561bbf3aac70 2024-03-08T23:08:41.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4360a60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf30ae70 con 0x561bbf3b3850 2024-03-08T23:08:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.308+0000 7f4357400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4357400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3aa730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53052/0 (socket says 172.21.15.46:53052) 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 0x561bbf3b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4357400640 1 -- 172.21.15.46:0/3800655886 learned_addr learned my addr 172.21.15.46:0/3800655886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f435ca00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 0x7f433c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356000640 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 -- 172.21.15.46:0/3800655886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 msgr2=0x561bbf3b5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 -- 172.21.15.46:0/3800655886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 msgr2=0x561bbf3aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4357400640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 0x7f433c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f435ca00640 1 -- 172.21.15.46:0/4141152403 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f43480f0e10 con 0x7f43581025a0 2024-03-08T23:08:41.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.309+0000 7f4356a00640 1 -- 172.21.15.46:0/3800655886 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bbf3712b0 con 0x561bbf3aac70 2024-03-08T23:08:41.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.310+0000 7f4356a00640 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 0x561bbf3b3310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f43480ee130 tx=0x7f43480b8c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34ab3c6ed96cfe69 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.310+0000 7f4343e00640 1 -- 172.21.15.46:0/3800655886 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43480bd670 con 0x561bbf3aac70 2024-03-08T23:08:41.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.310+0000 7f4343e00640 1 -- 172.21.15.46:0/3800655886 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43480bcf70 con 0x561bbf3aac70 2024-03-08T23:08:41.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.310+0000 7f4343e00640 1 -- 172.21.15.46:0/3800655886 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43480b95e0 con 0x561bbf3aac70 2024-03-08T23:08:41.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- 172.21.15.46:0/3800655886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 msgr2=0x561bbf3b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4357400640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 0x7f433c0a4fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4350013840 tx=0x7f4350004ab0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 0x561bbf3b3310 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f43480ee130 tx=0x7f43480b8c10 comp rx=0 tx=0).stop 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- 172.21.15.46:0/3800655886 shutdown_connections 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aac70 0x561bbf3b3310 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 --2- 172.21.15.46:0/3800655886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- 172.21.15.46:0/3800655886 >> 172.21.15.46:0/3800655886 conn(0x561bbf3a4a80 msgr2=0x561bbf3a4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- 172.21.15.46:0/3800655886 shutdown_connections 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- 172.21.15.46:0/3800655886 wait complete. 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.311+0000 7f4360a60740 1 -- start start 2024-03-08T23:08:41.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 Processor -- start 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aac70 0x561bbf3b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf2b7c70 con 0x561bbf3aa350 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf2cb0c0 con 0x561bbf3b3850 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561bbf3b6400 con 0x561bbf3aac70 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.312+0000 7f4360a60740 10 client.0 _ll_register_callbacks cb 0x561bbf305ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4356000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aac70 0x561bbf3b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36314/0 (socket says 172.21.15.46:36314) 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 -- 172.21.15.46:0/1535461469 learned_addr learned my addr 172.21.15.46:0/1535461469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4356a00640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 -- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aac70 msgr2=0x561bbf3b04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aac70 0x561bbf3b04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 -- 172.21.15.46:0/1535461469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 msgr2=0x561bbf3b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 -- 172.21.15.46:0/1535461469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561bbf0233f0 con 0x561bbf3b3850 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4356a00640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4357400640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f435000fe30 tx=0x7f4350004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911f1b13dddb702f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4340c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.313+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350004d20 con 0x561bbf3b3850 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43480bcf70 con 0x561bbf3b3850 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4350005020 con 0x561bbf3b3850 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4350010e40 con 0x561bbf3b3850 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 10 client.25793 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561bbf0ed7d0 con 0x561bbf3b3850 2024-03-08T23:08:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43480bcf70 con 0x561bbf3b3850 2024-03-08T23:08:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f435001c030 con 0x561bbf3b3850 2024-03-08T23:08:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 10 client.25793 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 10 client.25793 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 20 client.25793 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf3c3450 0x561bbf3c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 4 client.25793 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.314+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561bbf2e34f0 con 0x561bbf3c3450 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.315+0000 7f4356a00640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf3c3450 0x561bbf3c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4356a00640 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf3c3450 0x561bbf3c5840 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd9d7c2d8614910d server_cookie=4eb2d190c0eedc97 in_seq=0 out_seq=0 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4340c00640 10 client.25793 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4340c00640 1 -- 172.21.15.46:0/1535461469 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561bbf2e34f0 con 0x561bbf3c3450 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4340c00640 10 client.25793 handle_command_reply: tid=0 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4360a60740 1 client.25793 shutdown 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 msgr2=0x561bbf3b0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.316+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f435000fe30 tx=0x7f4350004a40 comp rx=0 tx=0).stop 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 shutdown_connections 2024-03-08T23:08:41.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf3c3450 0x561bbf3c5840 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3b0a00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aac70 0x561bbf3b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 --2- 172.21.15.46:0/1535461469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aa350 0x561bbf3b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 >> 172.21.15.46:0/1535461469 conn(0x561bbf3a4a80 msgr2=0x561bbf3a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4340c00640 0 client.25793 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 shutdown_connections 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 1 -- 172.21.15.46:0/1535461469 wait complete. 2024-03-08T23:08:41.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.317+0000 7f4360a60740 20 client.25793 trim_cache size 0 max 16384 2024-03-08T23:08:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.338+0000 7f4360a60740 1 Processor -- start 2024-03-08T23:08:41.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.338+0000 7f4360a60740 1 -- start start 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aa350 0x561bbf3b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581b2c60 con 0x561bbf3aac70 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581b0800 con 0x561bbf3b3850 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4360a60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581ae8a0 con 0x561bbf3aa350 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53068/0 (socket says 172.21.15.46:53068) 2024-03-08T23:08:41.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4356000640 1 -- 172.21.15.46:0/3133594993 learned_addr learned my addr 172.21.15.46:0/3133594993 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4357400640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.339+0000 7f4356a00640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aa350 0x561bbf3b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 -- 172.21.15.46:0/3133594993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aa350 msgr2=0x561bbf3b2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aa350 0x561bbf3b2bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 -- 172.21.15.46:0/3133594993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 msgr2=0x561bbf3b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 -- 172.21.15.46:0/3133594993 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561bbf3a5650 con 0x561bbf3b3850 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4356000640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4357400640 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3ad740 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f435000f790 tx=0x7f4350010db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b2c0c2d6534b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4343400640 1 -- 172.21.15.46:0/3133594993 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f435000fbf0 con 0x561bbf3b3850 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4343400640 1 -- 172.21.15.46:0/3133594993 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4350024070 con 0x561bbf3b3850 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.340+0000 7f4343400640 1 -- 172.21.15.46:0/3133594993 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4350020780 con 0x561bbf3b3850 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 -- 172.21.15.46:0/3133594993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 msgr2=0x561bbf3ad740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3ad740 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f435000f790 tx=0x7f4350010db0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 -- 172.21.15.46:0/3133594993 shutdown_connections 2024-03-08T23:08:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3b3850 0x561bbf3ad740 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 --2- 172.21.15.46:0/3133594993 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3aa350 0x561bbf3b2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.341+0000 7f4360a60740 1 -- 172.21.15.46:0/3133594993 >> 172.21.15.46:0/3133594993 conn(0x561bbf3a4a80 msgr2=0x561bbf3aed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.342+0000 7f4360a60740 1 -- 172.21.15.46:0/3133594993 shutdown_connections 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.342+0000 7f4360a60740 1 -- 172.21.15.46:0/3133594993 wait complete. 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.342+0000 7f4360a60740 1 -- start start 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.342+0000 7f4360a60740 1 Processor -- start 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aa350 0x561bbf3ba860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581aeea0 con 0x561bbf3aac70 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581b01a0 con 0x561bbf3aa350 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43581af4e0 con 0x561bbf3b3850 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4360a60740 10 client.0 _ll_register_callbacks cb 0x7f43480b8ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4356000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53084/0 (socket says 172.21.15.46:53084) 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4356000640 1 -- 172.21.15.46:0/280191692 learned_addr learned my addr 172.21.15.46:0/280191692 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4357400640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.343+0000 7f4356a00640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aa350 0x561bbf3ba860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 -- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 msgr2=0x561bbf3b91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 -- 172.21.15.46:0/280191692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aa350 msgr2=0x561bbf3ba860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aa350 0x561bbf3ba860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 -- 172.21.15.46:0/280191692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561bbf3b98b0 con 0x561bbf3aac70 2024-03-08T23:08:41.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4357400640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4356000640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4344012cb0 tx=0x7f434400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1656c444dbb6e85 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4340c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.344+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4344022020 con 0x561bbf3aac70 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4350024070 con 0x561bbf3aac70 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4344005020 con 0x561bbf3aac70 2024-03-08T23:08:41.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f434402a050 con 0x561bbf3aac70 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4360a60740 10 client.15678 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561bbf41a980 con 0x561bbf3aac70 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.345+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4350024070 con 0x561bbf3aac70 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f434401e440 con 0x561bbf3aac70 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 10 client.15678 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 10 client.15678 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 20 client.15678 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf41b0b0 0x561bbf41d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 4 client.15678 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.346+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561bbf31c310 con 0x561bbf41b0b0 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4356a00640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf41b0b0 0x561bbf41d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4356a00640 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf41b0b0 0x561bbf41d4d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0a0bb08ae2f2281 server_cookie=7b96367f1750c65f in_seq=0 out_seq=0 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4340c00640 10 client.15678 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:41.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4340c00640 1 -- 172.21.15.46:0/280191692 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561bbf31c310 con 0x561bbf41b0b0 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4340c00640 10 client.15678 handle_command_reply: tid=0 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.347+0000 7f4360a60740 1 client.15678 shutdown 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 msgr2=0x561bbf3b8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4344012cb0 tx=0x7f434400f790 comp rx=0 tx=0).stop 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 shutdown_connections 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561bbf41b0b0 0x561bbf41d4d0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561bbf3b3850 0x561bbf3b91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561bbf3aac70 0x561bbf3b8c80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 --2- 172.21.15.46:0/280191692 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561bbf3aa350 0x561bbf3ba860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 >> 172.21.15.46:0/280191692 conn(0x561bbf3a4a80 msgr2=0x561bbf3a3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4340c00640 0 client.15678 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 shutdown_connections 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 1 -- 172.21.15.46:0/280191692 wait complete. 2024-03-08T23:08:41.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.348+0000 7f4360a60740 20 client.15678 trim_cache size 0 max 16384 2024-03-08T23:08:41.354 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:41.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 msgr2=0x7f433c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 0x7f433c0a4fd0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4350013840 tx=0x7f4350004ab0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 msgr2=0x7f4358199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 0x7f4358199e50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f434802c210 tx=0x7f4348005e80 comp rx=0 tx=0).stop 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 shutdown_connections 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f433c0a2b10 0x7f433c0a4fd0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f435810b8e0 0x7f435819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4358102ec0 0x7f435819a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 --2- 172.21.15.46:0/4141152403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f43581025a0 0x7f4358199e50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 >> 172.21.15.46:0/4141152403 conn(0x7f4358077350 msgr2=0x7f43580fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 shutdown_connections 2024-03-08T23:08:41.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.349+0000 7f435de00640 1 -- 172.21.15.46:0/4141152403 wait complete. 2024-03-08T23:08:41.367 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:41 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2137315911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:41.371 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:41.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 -- 172.21.15.46:0/3717019508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 msgr2=0x7f174c109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 --2- 172.21.15.46:0/3717019508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c109950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f173c009900 tx=0x7f173c02bc50 comp rx=0 tx=0).stop 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 -- 172.21.15.46:0/3717019508 shutdown_connections 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 --2- 172.21.15.46:0/3717019508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c103e70 0x7f174c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 --2- 172.21.15.46:0/3717019508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c1034b0 0x7f174c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 --2- 172.21.15.46:0/3717019508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c109950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 -- 172.21.15.46:0/3717019508 >> 172.21.15.46:0/3717019508 conn(0x7f174c0fefa0 msgr2=0x7f174c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 -- 172.21.15.46:0/3717019508 shutdown_connections 2024-03-08T23:08:41.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.556+0000 7f1750c00640 1 -- 172.21.15.46:0/3717019508 wait complete. 2024-03-08T23:08:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 Processor -- start 2024-03-08T23:08:41.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 -- start start 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 0x7f174c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c103e70 0x7f174c19ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c113850 con 0x7f174c109570 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1136d0 con 0x7f174c1034b0 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.557+0000 7f1750c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1139d0 con 0x7f174c103e70 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f1743e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c103e70 0x7f174c19ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53108/0 (socket says 172.21.15.46:53108) 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 -- 172.21.15.46:0/367398598 learned_addr learned my addr 172.21.15.46:0/367398598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f1749e00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 0x7f174c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 -- 172.21.15.46:0/367398598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c103e70 msgr2=0x7f174c19ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c103e70 0x7f174c19ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 -- 172.21.15.46:0/367398598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 msgr2=0x7f174c19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 0x7f174c19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f174a800640 1 -- 172.21.15.46:0/367398598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f174c19a3d0 con 0x7f174c109570 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.558+0000 7f1749e00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 0x7f174c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f174a800640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f174400eb10 tx=0x7f174400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f1742a00640 1 -- 172.21.15.46:0/367398598 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744004b20 con 0x7f174c109570 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f1742a00640 1 -- 172.21.15.46:0/367398598 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1744004cc0 con 0x7f174c109570 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f174c19a6c0 con 0x7f174c109570 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f1742a00640 1 -- 172.21.15.46:0/367398598 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1744010760 con 0x7f174c109570 2024-03-08T23:08:41.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.559+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f174c1a69d0 con 0x7f174c109570 2024-03-08T23:08:41.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:41 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2137315911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:41.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.562+0000 7f1742a00640 1 -- 172.21.15.46:0/367398598 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f1744005020 con 0x7f174c109570 2024-03-08T23:08:41.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.562+0000 7f1753974740 1 Processor -- start 2024-03-08T23:08:41.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.562+0000 7f1753974740 1 -- start start 2024-03-08T23:08:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf6693730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 0x55fcf669ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf65f7dd0 con 0x55fcf669c850 2024-03-08T23:08:41.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf65a0c70 con 0x55fcf6693c70 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1753974740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf65f3e70 con 0x55fcf6693350 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36366/0 (socket says 172.21.15.46:36366) 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 -- 172.21.15.46:0/780714382 learned_addr learned my addr 172.21.15.46:0/780714382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1743e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 0x55fcf669ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1743e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf6693730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 -- 172.21.15.46:0/780714382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 msgr2=0x55fcf6693730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf6693730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 -- 172.21.15.46:0/780714382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 msgr2=0x55fcf669ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 0x55fcf669ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1749e00640 1 -- 172.21.15.46:0/780714382 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcf665a2b0 con 0x55fcf6693c70 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.563+0000 7f1743e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 0x55fcf669ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f1742a00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 0x7f17300a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f1749e00640 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f173c02c190 tx=0x7f173c009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ab851af8b15cee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f1742a00640 1 -- 172.21.15.46:0/367398598 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f17440cfa60 con 0x7f174c109570 2024-03-08T23:08:41.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f1743e00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 0x7f17300a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f173b000640 1 -- 172.21.15.46:0/780714382 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c042070 con 0x55fcf6693c70 2024-03-08T23:08:41.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f173b000640 1 -- 172.21.15.46:0/780714382 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f173c004c70 con 0x55fcf6693c70 2024-03-08T23:08:41.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.564+0000 7f173b000640 1 -- 172.21.15.46:0/780714382 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c005210 con 0x55fcf6693c70 2024-03-08T23:08:41.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 -- 172.21.15.46:0/780714382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 msgr2=0x55fcf669c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1743e00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 0x7f17300a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x55fcf668ec10 tx=0x7f1734014040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f173c02c190 tx=0x7f173c009610 comp rx=0 tx=0).stop 2024-03-08T23:08:41.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 -- 172.21.15.46:0/780714382 shutdown_connections 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf669c850 0x55fcf669ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669c310 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 --2- 172.21.15.46:0/780714382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf6693730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 -- 172.21.15.46:0/780714382 >> 172.21.15.46:0/780714382 conn(0x55fcf668da80 msgr2=0x55fcf668de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 -- 172.21.15.46:0/780714382 shutdown_connections 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.565+0000 7f1753974740 1 -- 172.21.15.46:0/780714382 wait complete. 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 -- start start 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 Processor -- start 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 0x55fcf66995c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf6699bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf669f5e0 con 0x55fcf6693c70 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf65b40c0 con 0x55fcf669c850 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcf669f760 con 0x55fcf6693350 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.566+0000 7f1753974740 10 client.0 _ll_register_callbacks cb 0x55fcf65eeec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1743e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 0x55fcf66995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56536/0 (socket says 172.21.15.46:56536) 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1743e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 0x55fcf66995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53136/0 (socket says 172.21.15.46:53136) 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 -- 172.21.15.46:0/3526588211 learned_addr learned my addr 172.21.15.46:0/3526588211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f174a800640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf6699bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 -- 172.21.15.46:0/3526588211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 msgr2=0x55fcf6699bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf6699bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 -- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 msgr2=0x55fcf66995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 0x55fcf66995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 -- 172.21.15.46:0/3526588211 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fcf630c3f0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f1749e00640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f173c036040 tx=0x7f173c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d97273aa6c65120 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.567+0000 7f172f400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c042070 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fcf65cc4f0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f173c0054d0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f173c059050 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f1753974740 10 client.25822 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fcf63d67d0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fcf65cc4f0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.568+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f173c0506b0 con 0x55fcf6693350 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 10 client.25822 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 10 client.25822 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 20 client.25822 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf66ac780 0x55fcf66aeb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 4 client.25822 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcf6605310 con 0x55fcf66ac780 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1743e00640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf66ac780 0x55fcf66aeb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f1743e00640 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf66ac780 0x55fcf66aeb70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=585cb076f91948a2 server_cookie=daf78e0b794262eb in_seq=0 out_seq=0 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.569+0000 7f172f400640 10 client.25822 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f172f400640 1 -- 172.21.15.46:0/3526588211 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fcf6605310 con 0x55fcf66ac780 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f172f400640 10 client.25822 handle_command_reply: tid=0 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 client.25822 shutdown 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 msgr2=0x55fcf669c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f173c036040 tx=0x7f173c009680 comp rx=0 tx=0).stop 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 shutdown_connections 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf6699bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693c70 0x55fcf66995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf66ac780 0x55fcf66aeb70 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 --2- 172.21.15.46:0/3526588211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693350 0x55fcf669c230 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 >> 172.21.15.46:0/3526588211 conn(0x55fcf668da80 msgr2=0x55fcf668f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.570+0000 7f172f400640 0 client.25822 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.571+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 shutdown_connections 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.571+0000 7f1753974740 1 -- 172.21.15.46:0/3526588211 wait complete. 2024-03-08T23:08:41.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.571+0000 7f1753974740 20 client.25822 trim_cache size 0 max 16384 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.583+0000 7f1753974740 1 Processor -- start 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 -- start start 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 0x55fcf669c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1b2ac0 con 0x55fcf6693350 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1b0660 con 0x55fcf6693c70 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1753974740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1ae700 con 0x55fcf669c850 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1749e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1743e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1749e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53152/0 (socket says 172.21.15.46:53152) 2024-03-08T23:08:41.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f1749e00640 1 -- 172.21.15.46:0/3005273528 learned_addr learned my addr 172.21.15.46:0/3005273528 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.584+0000 7f174a800640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 0x55fcf669c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 -- 172.21.15.46:0/3005273528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 msgr2=0x55fcf669c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 0x55fcf669c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 -- 172.21.15.46:0/3005273528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 msgr2=0x55fcf6698760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 -- 172.21.15.46:0/3005273528 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcf6699f40 con 0x55fcf6693c70 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f174a800640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 0x55fcf669c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1749e00640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f1743e00640 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669bcf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55fcf6688070 tx=0x7f17340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c64bc57e1802a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f173a600640 1 -- 172.21.15.46:0/3005273528 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734018030 con 0x55fcf6693c70 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f173a600640 1 -- 172.21.15.46:0/3005273528 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f173402f070 con 0x55fcf6693c70 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.585+0000 7f173a600640 1 -- 172.21.15.46:0/3005273528 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1734029760 con 0x55fcf6693c70 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 -- 172.21.15.46:0/3005273528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 msgr2=0x55fcf669bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669bcf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55fcf6688070 tx=0x7f17340046c0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 -- 172.21.15.46:0/3005273528 shutdown_connections 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf669c850 0x55fcf669c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf6693c70 0x55fcf669bcf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 --2- 172.21.15.46:0/3005273528 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf6698760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 -- 172.21.15.46:0/3005273528 >> 172.21.15.46:0/3005273528 conn(0x55fcf668da80 msgr2=0x55fcf668de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 -- 172.21.15.46:0/3005273528 shutdown_connections 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.586+0000 7f1753974740 1 -- 172.21.15.46:0/3005273528 wait complete. 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 -- start start 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 Processor -- start 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693c70 0x55fcf66a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf66a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1aed00 con 0x55fcf6693350 2024-03-08T23:08:41.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1b0000 con 0x55fcf669c850 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f174c1af340 con 0x55fcf6693c70 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.587+0000 7f1753974740 10 client.0 _ll_register_callbacks cb 0x7f173c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 -- 172.21.15.46:0/2832443014 learned_addr learned my addr 172.21.15.46:0/2832443014 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1743e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693c70 0x55fcf66a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f174a800640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf66a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 -- 172.21.15.46:0/2832443014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693c70 msgr2=0x55fcf66a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693c70 0x55fcf66a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 -- 172.21.15.46:0/2832443014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 msgr2=0x55fcf66a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf66a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 -- 172.21.15.46:0/2832443014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fcf66a2e20 con 0x55fcf6693350 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f1749e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f173c005700 tx=0x7f173c005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c74394932e98a96 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f172f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173c042040 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.588+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f173c02cbe0 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f173402f070 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f173c050450 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 10 client.15696 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fcf6703930 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f173402f070 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f173c0035f0 con 0x55fcf6693350 2024-03-08T23:08:41.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 10 client.15696 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 10 client.15696 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 20 client.15696 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf6703e00 0x55fcf67061f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 4 client.15696 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.589+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fcf65ff660 con 0x55fcf6703e00 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f1743e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf6703e00 0x55fcf67061f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f1743e00640 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf6703e00 0x55fcf67061f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=66e559521b02797e server_cookie=9289e2226a956754 in_seq=0 out_seq=0 2024-03-08T23:08:41.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f172f400640 10 client.15696 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f172f400640 1 -- 172.21.15.46:0/2832443014 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fcf65ff660 con 0x55fcf6703e00 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f172f400640 10 client.15696 handle_command_reply: tid=0 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f1753974740 1 client.15696 shutdown 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 msgr2=0x55fcf66a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.590+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f173c005700 tx=0x7f173c005730 comp rx=0 tx=0).stop 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 shutdown_connections 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fcf669c850 0x55fcf66a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55fcf6703e00 0x55fcf67061f0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fcf6693c70 0x55fcf66a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 --2- 172.21.15.46:0/2832443014 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fcf6693350 0x55fcf66a3d10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f172f400640 0 client.15696 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:08:41.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 >> 172.21.15.46:0/2832443014 conn(0x55fcf668da80 msgr2=0x55fcf668c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 shutdown_connections 2024-03-08T23:08:41.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 1 -- 172.21.15.46:0/2832443014 wait complete. 2024-03-08T23:08:41.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1753974740 20 client.15696 trim_cache size 0 max 16384 2024-03-08T23:08:41.599 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 msgr2=0x7f17300a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 0x7f17300a4f50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x55fcf668ec10 tx=0x7f1734014040 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.591+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 msgr2=0x7f174c199b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f174400eb10 tx=0x7f174400c2d0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 shutdown_connections 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f17300a2a90 0x7f17300a4f50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f174c109570 0x7f174c199b10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f174c103e70 0x7f174c19ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 --2- 172.21.15.46:0/367398598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f174c1034b0 0x7f174c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 >> 172.21.15.46:0/367398598 conn(0x7f174c0fefa0 msgr2=0x7f174c100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 shutdown_connections 2024-03-08T23:08:41.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.592+0000 7f1750c00640 1 -- 172.21.15.46:0/367398598 wait complete. 2024-03-08T23:08:41.610 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:08:41.610 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:08:41.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:41 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2137315911' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 -- 172.21.15.46:0/3282673183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 msgr2=0x7f3080109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 --2- 172.21.15.46:0/3282673183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f3080109950 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3070009930 tx=0x7f307002bc30 comp rx=0 tx=0).stop 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 -- 172.21.15.46:0/3282673183 shutdown_connections 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 --2- 172.21.15.46:0/3282673183 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f3080110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 --2- 172.21.15.46:0/3282673183 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 0x7f3080103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 --2- 172.21.15.46:0/3282673183 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f3080109950 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 -- 172.21.15.46:0/3282673183 >> 172.21.15.46:0/3282673183 conn(0x7f30800fef60 msgr2=0x7f3080101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 -- 172.21.15.46:0/3282673183 shutdown_connections 2024-03-08T23:08:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.803+0000 7f3085400640 1 -- 172.21.15.46:0/3282673183 wait complete. 2024-03-08T23:08:41.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.804+0000 7f3085400640 1 Processor -- start 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.804+0000 7f3085400640 1 -- start start 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 0x7f3080199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f308019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080113880 con 0x7f3080103e70 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080113700 con 0x7f3080109570 2024-03-08T23:08:41.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f3085400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080113a00 con 0x7f30801034b0 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f307f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f308019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f307e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 0x7f3080199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f307f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36400/0 (socket says 172.21.15.46:36400) 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.805+0000 7f307f200640 1 -- 172.21.15.46:0/654019501 learned_addr learned my addr 172.21.15.46:0/654019501 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 -- 172.21.15.46:0/654019501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 msgr2=0x7f3080199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 0x7f3080199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 -- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 msgr2=0x7f308019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f308019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 -- 172.21.15.46:0/654019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f308019ed50 con 0x7f3080109570 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307de00640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f308019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.806+0000 7f307f200640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f307800eb10 tx=0x7f307800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.807+0000 7f3077400640 1 -- 172.21.15.46:0/654019501 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078004b20 con 0x7f3080109570 2024-03-08T23:08:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.807+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308019f040 con 0x7f3080109570 2024-03-08T23:08:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.807+0000 7f3077400640 1 -- 172.21.15.46:0/654019501 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3078004cc0 con 0x7f3080109570 2024-03-08T23:08:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.807+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30801a68d0 con 0x7f3080109570 2024-03-08T23:08:41.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.807+0000 7f3077400640 1 -- 172.21.15.46:0/654019501 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078010760 con 0x7f3080109570 2024-03-08T23:08:41.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.811+0000 7f3088214740 1 Processor -- start 2024-03-08T23:08:41.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3077400640 1 -- 172.21.15.46:0/654019501 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f3078005020 con 0x7f3080109570 2024-03-08T23:08:41.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 -- start start 2024-03-08T23:08:41.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efc9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7ef24dd0 con 0x560c7efc9850 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7eecdc70 con 0x560c7efc0350 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.812+0000 7f3088214740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7ef20e70 con 0x560c7efc0c70 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efc9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53180/0 (socket says 172.21.15.46:53180) 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 -- 172.21.15.46:0/2576865658 learned_addr learned my addr 172.21.15.46:0/2576865658 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 -- 172.21.15.46:0/2576865658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 msgr2=0x560c7efc9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efc9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 -- 172.21.15.46:0/2576865658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 msgr2=0x560c7efc0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.813+0000 7f307de00640 1 -- 172.21.15.46:0/2576865658 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c7ef872b0 con 0x560c7efc9850 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.814+0000 7f3077400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 0x7f30640a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.814+0000 7f3077400640 1 -- 172.21.15.46:0/654019501 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f30780cf820 con 0x7f3080109570 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.814+0000 7f307de00640 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f306c013bf0 tx=0x7f306c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420c86b0008c4939 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.814+0000 7f307e800640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 0x7f30640a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.814+0000 7f306b400640 1 -- 172.21.15.46:0/2576865658 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c0130d0 con 0x560c7efc9850 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f306b400640 1 -- 172.21.15.46:0/2576865658 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306c013270 con 0x560c7efc9850 2024-03-08T23:08:41.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f306b400640 1 -- 172.21.15.46:0/2576865658 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c004a90 con 0x560c7efc9850 2024-03-08T23:08:41.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 -- 172.21.15.46:0/2576865658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 msgr2=0x560c7efcbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f306c013bf0 tx=0x7f306c0096c0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 -- 172.21.15.46:0/2576865658 shutdown_connections 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc9850 0x560c7efcbc40 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f307e800640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 0x7f30640a4fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3070009900 tx=0x7f307002df90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efc9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 --2- 172.21.15.46:0/2576865658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 -- 172.21.15.46:0/2576865658 >> 172.21.15.46:0/2576865658 conn(0x560c7efbaa80 msgr2=0x560c7efbae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 -- 172.21.15.46:0/2576865658 shutdown_connections 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.815+0000 7f3088214740 1 -- 172.21.15.46:0/2576865658 wait complete. 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.816+0000 7f3088214740 1 -- start start 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.816+0000 7f3088214740 1 Processor -- start 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0350 0x560c7efc9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efc6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7efcc5e0 con 0x560c7efc0c70 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7eee10c0 con 0x560c7efc9850 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c7efcc760 con 0x560c7efc0350 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f3088214740 10 client.0 _ll_register_callbacks cb 0x560c7ef1bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f307e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0350 0x560c7efc9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.817+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53188/0 (socket says 172.21.15.46:53188) 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 -- 172.21.15.46:0/403580448 learned_addr learned my addr 172.21.15.46:0/403580448 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307f200640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efc6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 -- 172.21.15.46:0/403580448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0350 msgr2=0x560c7efc9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0350 0x560c7efc9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 -- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 msgr2=0x560c7efc6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efc6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 -- 172.21.15.46:0/403580448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c7ec393f0 con 0x560c7efc0c70 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307f200640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efc6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f307de00640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f306c01d040 tx=0x7f306c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f31f15faaf3f556d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.818+0000 7f305fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c0130d0 con 0x560c7efc0c70 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f306c005430 con 0x560c7efc0c70 2024-03-08T23:08:41.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f306c013270 con 0x560c7efc0c70 2024-03-08T23:08:41.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f306c032050 con 0x560c7efc0c70 2024-03-08T23:08:41.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f3088214740 10 client.15708 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.819+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560c7ed037d0 con 0x560c7efc0c70 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306c013270 con 0x560c7efc0c70 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f306c003850 con 0x560c7efc0c70 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 10 client.15708 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 10 client.15708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 20 client.15708 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7efd9790 0x560c7efdbb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 4 client.15708 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c7eef94f0 con 0x560c7efd9790 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.820+0000 7f307e800640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7efd9790 0x560c7efdbb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.821+0000 7f307e800640 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7efd9790 0x560c7efdbb80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbd3f58941ed241a server_cookie=e45e3b76bbae9023 in_seq=0 out_seq=0 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.821+0000 7f305fe00640 10 client.15708 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:41.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f305fe00640 1 -- 172.21.15.46:0/403580448 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560c7eef94f0 con 0x560c7efd9790 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f305fe00640 10 client.15708 handle_command_reply: tid=0 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 client.15708 shutdown 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 msgr2=0x560c7efc65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f306c01d040 tx=0x7f306c005f90 comp rx=0 tx=0).stop 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 shutdown_connections 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7efd9790 0x560c7efdbb80 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efc6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc65b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 --2- 172.21.15.46:0/403580448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0350 0x560c7efc9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 >> 172.21.15.46:0/403580448 conn(0x560c7efbaa80 msgr2=0x560c7efbc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.822+0000 7f305fe00640 0 client.15708 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.823+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 shutdown_connections 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.823+0000 7f3088214740 1 -- 172.21.15.46:0/403580448 wait complete. 2024-03-08T23:08:41.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.823+0000 7f3088214740 20 client.15708 trim_cache size 0 max 16384 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 Processor -- start 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 -- start start 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 0x560c7efd0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3060012140 con 0x560c7efc0c70 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000fce0 con 0x560c7efc0350 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.842+0000 7f3088214740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000dd80 con 0x560c7efc9850 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53192/0 (socket says 172.21.15.46:53192) 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 -- 172.21.15.46:0/3622913438 learned_addr learned my addr 172.21.15.46:0/3622913438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307f200640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 0x560c7efd0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307e800640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 -- 172.21.15.46:0/3622913438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 msgr2=0x560c7efd0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 0x560c7efd0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 -- 172.21.15.46:0/3622913438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 msgr2=0x560c7efc8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307de00640 1 -- 172.21.15.46:0/3622913438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c7efc71c0 con 0x560c7efc0c70 2024-03-08T23:08:41.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.843+0000 7f307f200640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 0x560c7efd0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:41.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.844+0000 7f307de00640 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f306c0114d0 tx=0x7f306c0137f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24101cd2ab91fe1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.844+0000 7f306aa00640 1 -- 172.21.15.46:0/3622913438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c017070 con 0x560c7efc0c70 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.844+0000 7f306aa00640 1 -- 172.21.15.46:0/3622913438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306c005d30 con 0x560c7efc0c70 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.844+0000 7f306aa00640 1 -- 172.21.15.46:0/3622913438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c005490 con 0x560c7efc0c70 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 -- 172.21.15.46:0/3622913438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 msgr2=0x560c7efc9220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f306c0114d0 tx=0x7f306c0137f0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 -- 172.21.15.46:0/3622913438 shutdown_connections 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc9850 0x560c7efd0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0c70 0x560c7efc9220 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 --2- 172.21.15.46:0/3622913438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc0350 0x560c7efc8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 -- 172.21.15.46:0/3622913438 >> 172.21.15.46:0/3622913438 conn(0x560c7efbaa80 msgr2=0x560c7efe03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 -- 172.21.15.46:0/3622913438 shutdown_connections 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.845+0000 7f3088214740 1 -- 172.21.15.46:0/3622913438 wait complete. 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 -- start start 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 Processor -- start 2024-03-08T23:08:41.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efcef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efcf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000e380 con 0x560c7efc0350 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000f680 con 0x560c7efc9850 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306000e9c0 con 0x560c7efc0c70 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f3088214740 10 client.0 _ll_register_callbacks cb 0x7f306c009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.846+0000 7f307e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:53206/0 (socket says 172.21.15.46:53206) 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 -- 172.21.15.46:0/629313625 learned_addr learned my addr 172.21.15.46:0/629313625 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307f200640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efcf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307de00640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efcef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 -- 172.21.15.46:0/629313625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 msgr2=0x560c7efcef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efcef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 -- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 msgr2=0x560c7efcf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efcf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 -- 172.21.15.46:0/629313625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c7f03fe00 con 0x560c7efc0350 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307de00640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efcef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f307e800640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3070009930 tx=0x7f3070009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3076014164c19671 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:41.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.847+0000 7f305fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070064060 con 0x560c7efc0350 2024-03-08T23:08:41.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f306c005d30 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f307002cc10 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f307006b050 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f3088214740 10 client.15714 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.848+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560c7f041160 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f306c005d30 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f307005e030 con 0x560c7efc0350 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 10 client.15714 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:41.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 20 client.15714 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7f041810 0x560c7f043c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 4 client.15714 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.849+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560c7ef32310 con 0x560c7f041810 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f307de00640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7f041810 0x560c7f043c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f307de00640 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7f041810 0x560c7f043c30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d538b8cba91eaae server_cookie=d60a1bce8b2a213b in_seq=0 out_seq=0 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f305fe00640 10 client.15714 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f305fe00640 1 -- 172.21.15.46:0/629313625 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560c7ef32310 con 0x560c7f041810 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f305fe00640 10 client.15714 handle_command_reply: tid=0 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 client.15714 shutdown 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 msgr2=0x560c7efce9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3070009930 tx=0x7f3070009260 comp rx=0 tx=0).stop 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 shutdown_connections 2024-03-08T23:08:41.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560c7f041810 0x560c7f043c30 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.850+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c7efc9850 0x560c7efcf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c7efc0c70 0x560c7efcef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 1 --2- 172.21.15.46:0/629313625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c7efc0350 0x560c7efce9e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f305fe00640 0 client.15714 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 >> 172.21.15.46:0/629313625 conn(0x560c7efbaa80 msgr2=0x560c7efdb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 shutdown_connections 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 1 -- 172.21.15.46:0/629313625 wait complete. 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.851+0000 7f3088214740 20 client.15714 trim_cache size 0 max 16384 2024-03-08T23:08:41.858 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:08:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 msgr2=0x7f30640a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 0x7f30640a4fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3070009900 tx=0x7f307002df90 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 msgr2=0x7f308019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f307800eb10 tx=0x7f307800c2d0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 shutdown_connections 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f30640a2b10 0x7f30640a4fd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3080109570 0x7f308019e620 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3080103e70 0x7f308019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 --2- 172.21.15.46:0/654019501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f30801034b0 0x7f3080199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 >> 172.21.15.46:0/654019501 conn(0x7f30800fef60 msgr2=0x7f30801005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 shutdown_connections 2024-03-08T23:08:41.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:41.852+0000 7f3085400640 1 -- 172.21.15.46:0/654019501 wait complete. 2024-03-08T23:08:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:42 smithi192 ceph-mon[36700]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:42 smithi062 ceph-mon[35761]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:42.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:42 smithi046 ceph-mon[31859]: pgmap v30: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:42.877 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:08:42.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:08:42.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:08:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 -- 172.21.15.46:0/977983070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 msgr2=0x7f0ca81004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 --2- 172.21.15.46:0/977983070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca81004f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0c98009900 tx=0x7f0c9802bd10 comp rx=0 tx=0).stop 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 -- 172.21.15.46:0/977983070 shutdown_connections 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 --2- 172.21.15.46:0/977983070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8105510 0x7f0ca81059a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 --2- 172.21.15.46:0/977983070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8100a30 0x7f0ca8104ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 --2- 172.21.15.46:0/977983070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca81004f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.095+0000 7f0cad000640 1 -- 172.21.15.46:0/977983070 >> 172.21.15.46:0/977983070 conn(0x7f0ca80fbb40 msgr2=0x7f0ca80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.096+0000 7f0cad000640 1 -- 172.21.15.46:0/977983070 shutdown_connections 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.096+0000 7f0cad000640 1 -- 172.21.15.46:0/977983070 wait complete. 2024-03-08T23:08:43.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.097+0000 7f0cad000640 1 Processor -- start 2024-03-08T23:08:43.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.097+0000 7f0cad000640 1 -- start start 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8100a30 0x7f0ca819e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8105510 0x7f0ca81a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8117ba0 con 0x7f0ca8100110 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8117a20 con 0x7f0ca8100a30 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0cad000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca8117d20 con 0x7f0ca8105510 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0ca7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0ca7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56044/0 (socket says 172.21.15.46:56044) 2024-03-08T23:08:43.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0ca7400640 1 -- 172.21.15.46:0/2046621531 learned_addr learned my addr 172.21.15.46:0/2046621531 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8100a30 0x7f0ca819e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.098+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8105510 0x7f0ca81a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 -- 172.21.15.46:0/2046621531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8105510 msgr2=0x7f0ca81a2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8105510 0x7f0ca81a2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 -- 172.21.15.46:0/2046621531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8100a30 msgr2=0x7f0ca819e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8100a30 0x7f0ca819e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 -- 172.21.15.46:0/2046621531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca81a3050 con 0x7f0ca8100110 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0ca7400640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000c00 tx=0x7f0c980043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0c9fe00640 1 -- 172.21.15.46:0/2046621531 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98042020 con 0x7f0ca8100110 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.099+0000 7f0c9fe00640 1 -- 172.21.15.46:0/2046621531 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c98005000 con 0x7f0ca8100110 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.100+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca81a32e0 con 0x7f0ca8100110 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.100+0000 7f0c9fe00640 1 -- 172.21.15.46:0/2046621531 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c98003b20 con 0x7f0ca8100110 2024-03-08T23:08:43.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.100+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca81aabd0 con 0x7f0ca8100110 2024-03-08T23:08:43.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.103+0000 7f0cb1431740 1 Processor -- start 2024-03-08T23:08:43.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 -- start start 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3fa760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e355a50 con 0x55987e3faca0 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e2feb00 con 0x55987e3f20c0 2024-03-08T23:08:43.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0cb1431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e351af0 con 0x55987e3f17a0 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0ca6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0ca6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56058/0 (socket says 172.21.15.46:56058) 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0ca6a00640 1 -- 172.21.15.46:0/830939667 learned_addr learned my addr 172.21.15.46:0/830939667 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.104+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 -- 172.21.15.46:0/830939667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 msgr2=0x55987e3f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 -- 172.21.15.46:0/830939667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 msgr2=0x55987e3fa760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3fa760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 -- 172.21.15.46:0/830939667 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55987e31c1e0 con 0x55987e3faca0 2024-03-08T23:08:43.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0c9fe00640 1 -- 172.21.15.46:0/2046621531 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f0c98045020 con 0x7f0ca8100110 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.105+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b730 tx=0x7f0c9400bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ad98a14f4a20a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.106+0000 7f0c9cc00640 1 -- 172.21.15.46:0/830939667 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94003dd0 con 0x55987e3faca0 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.106+0000 7f0c9cc00640 1 -- 172.21.15.46:0/830939667 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c94015070 con 0x55987e3faca0 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.106+0000 7f0c9cc00640 1 -- 172.21.15.46:0/830939667 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94004de0 con 0x55987e3faca0 2024-03-08T23:08:43.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 -- 172.21.15.46:0/830939667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 msgr2=0x55987e3fd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400b730 tx=0x7f0c9400bbf0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 -- 172.21.15.46:0/830939667 shutdown_connections 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0c9fe00640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 0x7f0c8c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3fd090 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3fa760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 --2- 172.21.15.46:0/830939667 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.107+0000 7f0cb1431740 1 -- 172.21.15.46:0/830939667 >> 172.21.15.46:0/830939667 conn(0x55987e3ebed0 msgr2=0x55987e3ec2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.108+0000 7f0c9fe00640 1 -- 172.21.15.46:0/2046621531 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0c980f0fa0 con 0x7f0ca8100110 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.108+0000 7f0cb1431740 1 -- 172.21.15.46:0/830939667 shutdown_connections 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.108+0000 7f0cb1431740 1 -- 172.21.15.46:0/830939667 wait complete. 2024-03-08T23:08:43.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.108+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 0x7f0c8c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.108+0000 7f0cb1431740 1 -- start start 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 Processor -- start 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 0x7f0c8c0a4fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0005fd0 tx=0x7f0ca0009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e3fda90 con 0x55987e3faca0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e311f60 con 0x55987e3f20c0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.109+0000 7f0cb1431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55987e3fdc10 con 0x55987e3f17a0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0cb1431740 10 client.0 _ll_register_callbacks cb 0x55987e2fdb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3f87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56066/0 (socket says 172.21.15.46:56066) 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7e00640 1 -- 172.21.15.46:0/4263398069 learned_addr learned my addr 172.21.15.46:0/4263398069 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 -- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 msgr2=0x55987e3f8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 -- 172.21.15.46:0/4263398069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 msgr2=0x55987e3f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3f87c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca7400640 1 -- 172.21.15.46:0/4263398069 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55987e1327e0 con 0x55987e3f17a0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.110+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0ca7400640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f7d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c980ee2c0 tx=0x7f0c980bcb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dcdb533512fde100 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0c91600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980c1320 con 0x55987e3f17a0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0c9802cb10 con 0x55987e3f17a0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c94015070 con 0x55987e3f17a0 2024-03-08T23:08:43.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c980c5970 con 0x55987e3f17a0 2024-03-08T23:08:43.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0cb1431740 10 client.25858 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:43.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.111+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55987e192f70 con 0x55987e3f17a0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c94015070 con 0x55987e3f17a0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c98005500 con 0x55987e3f17a0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 10 client.25858 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 10 client.25858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 20 client.25858 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e40afb0 0x55987e40d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 4 client.25858 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55987e362f90 con 0x55987e40afb0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.112+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e40afb0 0x55987e40d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.113+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e40afb0 0x55987e40d400 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6907194efe380a6 server_cookie=8f2ee89ee49af960 in_seq=0 out_seq=0 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.113+0000 7f0c91600640 10 client.25858 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0c91600640 1 -- 172.21.15.46:0/4263398069 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55987e362f90 con 0x55987e40afb0 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0c91600640 10 client.25858 handle_command_reply: tid=0 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 client.25858 shutdown 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 msgr2=0x55987e3f7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f7d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0c980ee2c0 tx=0x7f0c980bcb30 comp rx=0 tx=0).stop 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 shutdown_connections 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e40afb0 0x55987e40d400 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3faca0 0x55987e3f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 --2- 172.21.15.46:0/4263398069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e3f7d40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0c91600640 0 client.25858 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.114+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 >> 172.21.15.46:0/4263398069 conn(0x55987e3ebed0 msgr2=0x55987e3edc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.115+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 shutdown_connections 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.115+0000 7f0cb1431740 1 -- 172.21.15.46:0/4263398069 wait complete. 2024-03-08T23:08:43.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.115+0000 7f0cb1431740 20 client.25858 trim_cache size 0 max 16384 2024-03-08T23:08:43.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.145+0000 7f0cb1431740 1 Processor -- start 2024-03-08T23:08:43.148 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 -- start start 2024-03-08T23:08:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3faca0 0x55987e3f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b6d40 con 0x55987e3f17a0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b48e0 con 0x55987e3f20c0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0cb1431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b2980 con 0x55987e3faca0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0ca7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0ca7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56074/0 (socket says 172.21.15.46:56074) 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0ca7400640 1 -- 172.21.15.46:0/1132980000 learned_addr learned my addr 172.21.15.46:0/1132980000 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.146+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3faca0 0x55987e3f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 -- 172.21.15.46:0/1132980000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3faca0 msgr2=0x55987e3f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3faca0 0x55987e3f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 -- 172.21.15.46:0/1132980000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 msgr2=0x55987e3f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 -- 172.21.15.46:0/1132980000 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55987e3f1ef0 con 0x55987e3f17a0 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0ca7400640 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0c98006010 tx=0x7f0c9802bd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ccb0177f4736f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.147+0000 7f0c93e00640 1 -- 172.21.15.46:0/1132980000 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c980bd6c0 con 0x55987e3f17a0 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.148+0000 7f0c93e00640 1 -- 172.21.15.46:0/1132980000 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c980c1320 con 0x55987e3f17a0 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.148+0000 7f0c93e00640 1 -- 172.21.15.46:0/1132980000 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c9802cb10 con 0x55987e3f17a0 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.148+0000 7f0cb1431740 1 -- 172.21.15.46:0/1132980000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 msgr2=0x55987e3fa470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.148+0000 7f0cb1431740 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f0c98006010 tx=0x7f0c9802bd10 comp rx=0 tx=0).stop 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 -- 172.21.15.46:0/1132980000 shutdown_connections 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3faca0 0x55987e3f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3f20c0 0x55987e3f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 --2- 172.21.15.46:0/1132980000 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f17a0 0x55987e3fa470 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 -- 172.21.15.46:0/1132980000 >> 172.21.15.46:0/1132980000 conn(0x55987e3ebed0 msgr2=0x55987e3f6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 -- 172.21.15.46:0/1132980000 shutdown_connections 2024-03-08T23:08:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 -- 172.21.15.46:0/1132980000 wait complete. 2024-03-08T23:08:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.149+0000 7f0cb1431740 1 -- start start 2024-03-08T23:08:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 Processor -- start 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e400eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3faca0 0x55987e4614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b2f80 con 0x55987e3f20c0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b4280 con 0x55987e3faca0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca81b35c0 con 0x55987e3f17a0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0cb1431740 10 client.0 _ll_register_callbacks cb 0x7f0c94004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.150+0000 7f0ca6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:56084/0 (socket says 172.21.15.46:56084) 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 -- 172.21.15.46:0/3069437252 learned_addr learned my addr 172.21.15.46:0/3069437252 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca7e00640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3faca0 0x55987e4614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca7400640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e400eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 -- 172.21.15.46:0/3069437252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 msgr2=0x55987e400eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e400eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 -- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3faca0 msgr2=0x55987e4614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3faca0 0x55987e4614c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 -- 172.21.15.46:0/3069437252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55987e02a170 con 0x55987e3f20c0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0ca6a00640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006010 tx=0x7f0c94004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f97b30f227cce2b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.151+0000 7f0c91600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94003bd0 con 0x55987e3f20c0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0c94021070 con 0x55987e3f20c0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c980c1320 con 0x55987e3f20c0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c94037050 con 0x55987e3f20c0 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0cb1431740 10 client.15732 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:43.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.152+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55987e321050 con 0x55987e3f20c0 2024-03-08T23:08:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c980c1320 con 0x55987e3f20c0 2024-03-08T23:08:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c94011920 con 0x55987e3f20c0 2024-03-08T23:08:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 10 client.15732 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 10 client.15732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 20 client.15732 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e462fd0 0x55987e4653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 4 client.15732 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55987e35d2e0 con 0x55987e462fd0 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.153+0000 7f0ca7400640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e462fd0 0x55987e4653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.154+0000 7f0c91600640 10 client.15732 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.154+0000 7f0ca7400640 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e462fd0 0x55987e4653f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a823342d1de2d5aa server_cookie=8b6a849f4eaa0d82 in_seq=0 out_seq=0 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0c91600640 1 -- 172.21.15.46:0/3069437252 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55987e35d2e0 con 0x55987e462fd0 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0c91600640 10 client.15732 handle_command_reply: tid=0 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 client.15732 shutdown 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 msgr2=0x55987e4013f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0c94006010 tx=0x7f0c94004740 comp rx=0 tx=0).stop 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 shutdown_connections 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55987e462fd0 0x55987e4653f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55987e3faca0 0x55987e4614c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55987e3f20c0 0x55987e4013f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 --2- 172.21.15.46:0/3069437252 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55987e3f17a0 0x55987e400eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0c91600640 0 client.15732 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 >> 172.21.15.46:0/3069437252 conn(0x55987e3ebed0 msgr2=0x55987e3f8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 shutdown_connections 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 1 -- 172.21.15.46:0/3069437252 wait complete. 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.156+0000 7f0cb1431740 20 client.15732 trim_cache size 0 max 16384 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:43.160 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:08:43.161 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "f979d169-cc4a-4eb0-b988-af8cafdea722", 2024-03-08T23:08:43.161 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:08:43.161 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:43.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.157+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 msgr2=0x7f0c8c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.157+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 0x7f0c8c0a4fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0005fd0 tx=0x7f0ca0009500 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.157+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 msgr2=0x7f0ca819e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.157+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0c98000c00 tx=0x7f0c980043c0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 shutdown_connections 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0c8c0a2b10 0x7f0c8c0a4fd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0ca8105510 0x7f0ca81a2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0ca8100a30 0x7f0ca819e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 --2- 172.21.15.46:0/2046621531 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0ca8100110 0x7f0ca819e060 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 >> 172.21.15.46:0/2046621531 conn(0x7f0ca80fbb40 msgr2=0x7f0ca8102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 shutdown_connections 2024-03-08T23:08:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.158+0000 7f0cad000640 1 -- 172.21.15.46:0/2046621531 wait complete. 2024-03-08T23:08:43.183 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:08:43.183 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 -- 172.21.15.46:0/504417284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c81034b0 msgr2=0x7ff4c8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 --2- 172.21.15.46:0/504417284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c81034b0 0x7ff4c8103930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4009900 tx=0x7ff4b402bc30 comp rx=0 tx=0).stop 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 -- 172.21.15.46:0/504417284 shutdown_connections 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 --2- 172.21.15.46:0/504417284 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c8103e70 0x7ff4c8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 --2- 172.21.15.46:0/504417284 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c81034b0 0x7ff4c8103930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 --2- 172.21.15.46:0/504417284 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 0x7ff4c8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 -- 172.21.15.46:0/504417284 >> 172.21.15.46:0/504417284 conn(0x7ff4c80fef60 msgr2=0x7ff4c8101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 -- 172.21.15.46:0/504417284 shutdown_connections 2024-03-08T23:08:43.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.376+0000 7ff4cd000640 1 -- 172.21.15.46:0/504417284 wait complete. 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.377+0000 7ff4cd000640 1 Processor -- start 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.377+0000 7ff4cd000640 1 -- start start 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.377+0000 7ff4cd000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4cd000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 0x7ff4c819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4cd000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 0x7ff4c819e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4cd000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8113880 con 0x7ff4c8103e70 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4cd000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8113700 con 0x7ff4c81034b0 2024-03-08T23:08:43.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4cd000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8113a00 con 0x7ff4c8109570 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48914/0 (socket says 172.21.15.46:48914) 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 0x7ff4c819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3906464041 learned_addr learned my addr 172.21.15.46:0/3906464041 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c6800640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 0x7ff4c819e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3906464041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 msgr2=0x7ff4c819e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 0x7ff4c819e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 msgr2=0x7ff4c819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 0x7ff4c819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3906464041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c819ed10 con 0x7ff4c81034b0 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.378+0000 7ff4c5400640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 0x7ff4c819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:43.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9f0 tx=0x7ff4b800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4bea00640 1 -- 172.21.15.46:0/3906464041 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800ccc0 con 0x7ff4c81034b0 2024-03-08T23:08:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c819f000 con 0x7ff4c81034b0 2024-03-08T23:08:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4bea00640 1 -- 172.21.15.46:0/3906464041 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b8010910 con 0x7ff4c81034b0 2024-03-08T23:08:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4bea00640 1 -- 172.21.15.46:0/3906464041 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80188a0 con 0x7ff4c81034b0 2024-03-08T23:08:43.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.379+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c81a6890 con 0x7ff4c81034b0 2024-03-08T23:08:43.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.382+0000 7ff4cfdba740 1 Processor -- start 2024-03-08T23:08:43.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.382+0000 7ff4cfdba740 1 -- start start 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa152540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa154e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa10b410 con 0x5588fa149ea0 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa0add50 con 0x5588fa149580 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa056b80 con 0x5588fa152a80 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48926/0 (socket says 172.21.15.46:48926) 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 -- 172.21.15.46:0/426684876 learned_addr learned my addr 172.21.15.46:0/426684876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c5400640 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 -- 172.21.15.46:0/426684876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 msgr2=0x5588fa154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 -- 172.21.15.46:0/426684876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 msgr2=0x5588fa152540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:08:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa152540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.383+0000 7ff4c6800640 1 -- 172.21.15.46:0/426684876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588f9e8c7d0 con 0x5588fa149580 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.384+0000 7ff4c6800640 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011440 tx=0x7ff4c0011900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a36e97d355ab90d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.384+0000 7ff4bea00640 1 -- 172.21.15.46:0/3906464041 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ff4b8010ac0 con 0x7ff4c81034b0 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.384+0000 7ff4b3400640 1 -- 172.21.15.46:0/426684876 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0003c30 con 0x5588fa149580 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.384+0000 7ff4b3400640 1 -- 172.21.15.46:0/426684876 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0003dd0 con 0x5588fa149580 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.384+0000 7ff4b3400640 1 -- 172.21.15.46:0/426684876 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000fae0 con 0x5588fa149580 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.385+0000 7ff4cfdba740 1 -- 172.21.15.46:0/426684876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 msgr2=0x5588fa149960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.385+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0011440 tx=0x7ff4c0011900 comp rx=0 tx=0).stop 2024-03-08T23:08:43.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.385+0000 7ff4bea00640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 0x7ff4ac0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.385+0000 7ff4bea00640 1 -- 172.21.15.46:0/3906464041 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4b80cf980 con 0x7ff4c81034b0 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 -- 172.21.15.46:0/426684876 shutdown_connections 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/426684876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa149960 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 -- 172.21.15.46:0/426684876 >> 172.21.15.46:0/426684876 conn(0x5588fa143cb0 msgr2=0x5588fa144090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4c6800640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 0x7ff4ac0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 -- 172.21.15.46:0/426684876 shutdown_connections 2024-03-08T23:08:43.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.386+0000 7ff4cfdba740 1 -- 172.21.15.46:0/426684876 wait complete. 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 -- start start 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4c6800640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 0x7ff4ac0a4f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000f780 tx=0x7ff4c000f540 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 Processor -- start 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149ea0 0x5588fa150050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa152a80 0x5588fa150590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa069ee0 con 0x5588fa152a80 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa0a9df0 con 0x5588fa149580 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588fa155870 con 0x5588fa149ea0 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.387+0000 7ff4cfdba740 10 client.0 _ll_register_callbacks cb 0x5588fa0708d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48928/0 (socket says 172.21.15.46:48928) 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1395165543 learned_addr learned my addr 172.21.15.46:0/1395165543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c6800640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa152a80 0x5588fa150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149ea0 0x5588fa150050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1395165543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149ea0 msgr2=0x5588fa150050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149ea0 0x5588fa150050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1395165543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa152a80 msgr2=0x5588fa150590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa152a80 0x5588fa150590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.388+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1395165543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588f9dc23f0 con 0x5588fa149580 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80cd370 tx=0x7ff4b809c1b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45621e37c3d4704f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80a41d0 con 0x5588fa149580 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4b809ae80 con 0x5588fa149580 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4c0003dd0 con 0x5588fa149580 2024-03-08T23:08:43.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4b809b080 con 0x5588fa149580 2024-03-08T23:08:43.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4cfdba740 10 client.25862 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:43.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.389+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5588f9d820f0 con 0x5588fa149580 2024-03-08T23:08:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0003dd0 con 0x5588fa149580 2024-03-08T23:08:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff4b80862b0 con 0x5588fa149580 2024-03-08T23:08:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 10 client.25862 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:43.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 10 client.25862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 20 client.25862 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa16b440 0x5588fa16d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 4 client.25862 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588fa0bb290 con 0x5588fa16b440 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.390+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa16b440 0x5588fa16d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.391+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa16b440 0x5588fa16d830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1d2dc418dc90e2 server_cookie=a42d980d49ff505a in_seq=0 out_seq=0 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.391+0000 7ff4a7e00640 10 client.25862 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.391+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1395165543 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588fa0bb290 con 0x5588fa16b440 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.391+0000 7ff4a7e00640 10 client.25862 handle_command_reply: tid=0 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 client.25862 shutdown 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 msgr2=0x5588fa14fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff4b80cd370 tx=0x7ff4b809c1b0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 shutdown_connections 2024-03-08T23:08:43.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa16b440 0x5588fa16d830 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa152a80 0x5588fa150590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149ea0 0x5588fa150050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1395165543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14fb10 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4a7e00640 0 client.25862 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 >> 172.21.15.46:0/1395165543 conn(0x5588fa143cb0 msgr2=0x5588fa145a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 shutdown_connections 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1395165543 wait complete. 2024-03-08T23:08:43.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.392+0000 7ff4cfdba740 20 client.25862 trim_cache size 0 max 16384 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.423+0000 7ff4cfdba740 1 Processor -- start 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.423+0000 7ff4cfdba740 1 -- start start 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1503b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa142d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81ae840 con 0x5588fa149ea0 2024-03-08T23:08:43.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81ac3e0 con 0x5588fa149580 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81aa480 con 0x5588fa152a80 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:48940/0 (socket says 172.21.15.46:48940) 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3756134907 learned_addr learned my addr 172.21.15.46:0/3756134907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4c5400640 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1503b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.424+0000 7ff4c6800640 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3756134907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 msgr2=0x5588fa142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3756134907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 msgr2=0x5588fa1503b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1503b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 -- 172.21.15.46:0/3756134907 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588fa1551e0 con 0x5588fa149580 2024-03-08T23:08:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4b809b730 tx=0x7ff4b809fd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50eec1b71225fd83 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4b2a00640 1 -- 172.21.15.46:0/3756134907 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8097b00 con 0x5588fa149580 2024-03-08T23:08:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4b2a00640 1 -- 172.21.15.46:0/3756134907 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b80ccb30 con 0x5588fa149580 2024-03-08T23:08:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.425+0000 7ff4b2a00640 1 -- 172.21.15.46:0/3756134907 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8097cd0 con 0x5588fa149580 2024-03-08T23:08:43.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- 172.21.15.46:0/3756134907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 msgr2=0x5588fa14ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff4b809b730 tx=0x7ff4b809fd00 comp rx=0 tx=0).stop 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- 172.21.15.46:0/3756134907 shutdown_connections 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa152a80 0x5588fa142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1503b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/3756134907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa149580 0x5588fa14ec90 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- 172.21.15.46:0/3756134907 >> 172.21.15.46:0/3756134907 conn(0x5588fa143cb0 msgr2=0x5588fa142680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- 172.21.15.46:0/3756134907 shutdown_connections 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- 172.21.15.46:0/3756134907 wait complete. 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.426+0000 7ff4cfdba740 1 -- start start 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 Processor -- start 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa152a80 0x5588fa1b5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81aaa80 con 0x5588fa149ea0 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81abd80 con 0x5588fa152a80 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c81ab0c0 con 0x5588fa149580 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4cfdba740 10 client.0 _ll_register_callbacks cb 0x7ff4c0009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39428/0 (socket says 172.21.15.46:39428) 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1169266847 learned_addr learned my addr 172.21.15.46:0/1169266847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.427+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c6800640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa152a80 0x5588fa1b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa152a80 msgr2=0x5588fa1b5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa152a80 0x5588fa1b5860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1169266847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 msgr2=0x5588fa1bb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1bb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 -- 172.21.15.46:0/1169266847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588fa1b6060 con 0x5588fa149580 2024-03-08T23:08:43.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4c5e00640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4b809c2e0 tx=0x7ff4b800ce60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e38d7267a871c3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b809b390 con 0x5588fa149580 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4b8010430 con 0x5588fa149580 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.428+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4b80ccb30 con 0x5588fa149580 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4b809ba30 con 0x5588fa149580 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 10 client.25882 fetch_fsmap learned FSMap version 24 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5588fa1c2c00 con 0x5588fa149580 2024-03-08T23:08:43.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b80ccb30 con 0x5588fa149580 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff4b80a0aa0 con 0x5588fa149580 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 10 client.25882 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 10 client.25882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 20 client.25882 populate_metadata read hostname 'smithi046' 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa1c30d0 0x5588fa1c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 4 client.25882 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.429+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5588fa0b55e0 con 0x5588fa1c30d0 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa1c30d0 0x5588fa1c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4c5400640 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa1c30d0 0x5588fa1c54c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d64eb2823862d7 server_cookie=dd061b5303c2d232 in_seq=0 out_seq=0 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4a7e00640 10 client.25882 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4a7e00640 1 -- 172.21.15.46:0/1169266847 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5588fa0b55e0 con 0x5588fa1c30d0 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4a7e00640 10 client.25882 handle_command_reply: tid=0 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.430+0000 7ff4cfdba740 1 client.25882 shutdown 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 msgr2=0x5588fa1bb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff4b809c2e0 tx=0x7ff4b800ce60 comp rx=0 tx=0).stop 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 shutdown_connections 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588fa1c30d0 0x5588fa1c54c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588fa152a80 0x5588fa1b5860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5588fa149ea0 0x5588fa1bb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 --2- 172.21.15.46:0/1169266847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588fa149580 0x5588fa1bb470 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 >> 172.21.15.46:0/1169266847 conn(0x5588fa143cb0 msgr2=0x5588fa14eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4a7e00640 0 client.25882 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 shutdown_connections 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 1 -- 172.21.15.46:0/1169266847 wait complete. 2024-03-08T23:08:43.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.431+0000 7ff4cfdba740 20 client.25882 trim_cache size 0 max 16384 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "f979d169-cc4a-4eb0-b988-af8cafdea722": { 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "f979d169-cc4a-4eb0-b988-af8cafdea722", 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:08:43.438 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 msgr2=0x7ff4ac0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 0x7ff4ac0a4f50 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000f780 tx=0x7ff4c000f540 comp rx=0 tx=0).stop 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 msgr2=0x7ff4c8199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800e9f0 tx=0x7ff4b800edf0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 shutdown_connections 2024-03-08T23:08:43.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4ac0a2a90 0x7ff4ac0a4f50 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4c8109570 0x7ff4c819e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4c8103e70 0x7ff4c819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 --2- 172.21.15.46:0/3906464041 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4c81034b0 0x7ff4c8199d20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 >> 172.21.15.46:0/3906464041 conn(0x7ff4c80fef60 msgr2=0x7ff4c81006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 shutdown_connections 2024-03-08T23:08:43.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:08:43.432+0000 7ff4cd000640 1 -- 172.21.15.46:0/3906464041 wait complete. 2024-03-08T23:08:43.452 INFO:tasks.cephfs.filesystem:scrub status for tag:f979d169-cc4a-4eb0-b988-af8cafdea722 - {'path': '/', 'tag': 'f979d169-cc4a-4eb0-b988-af8cafdea722', 'options': 'recursive,force'} 2024-03-08T23:08:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:44 smithi192 ceph-mon[36700]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:08:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:44 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:08:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:44 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:44 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:08:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:44 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:44.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:44 smithi046 ceph-mon[31859]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:08:44.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:44 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:08:44.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:44 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:44.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:44 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:08:44.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:44 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:44 smithi062 ceph-mon[35761]: pgmap v31: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:08:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:44 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:08:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:44 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:44 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:08:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:44 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:08:46.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:46 smithi192 ceph-mon[36700]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:46.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:46 smithi062 ceph-mon[35761]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:46.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:46 smithi046 ceph-mon[31859]: pgmap v32: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:47.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:08:47.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:08:47.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:08:48.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:48 smithi192 ceph-mon[36700]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:48.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:48 smithi062 ceph-mon[35761]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:48 smithi046 ceph-mon[31859]: pgmap v33: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:08:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:50 smithi192 ceph-mon[36700]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:50 smithi062 ceph-mon[35761]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:50 smithi046 ceph-mon[31859]: pgmap v34: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:52.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:52 smithi192 ceph-mon[36700]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:52.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:52 smithi062 ceph-mon[35761]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:52.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:52 smithi046 ceph-mon[31859]: pgmap v35: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:54 smithi192 ceph-mon[36700]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:54.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:54 smithi062 ceph-mon[35761]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:54.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:54 smithi046 ceph-mon[31859]: pgmap v36: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:08:56.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:56 smithi192 ceph-mon[36700]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:56.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:56 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:08:56.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:56 smithi062 ceph-mon[35761]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:56.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:56 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:08:56.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:56 smithi046 ceph-mon[31859]: pgmap v37: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:56.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:56 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:08:57.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:57 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:08:57.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:57 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:08:57.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:57 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:08:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:57 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:08:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:57 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:08:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:57 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:08:57.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:57 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:08:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:57 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:08:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:57 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:08:58.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:08:58 smithi192 ceph-mon[36700]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:58.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:08:58 smithi062 ceph-mon[35761]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:08:58.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:08:58 smithi046 ceph-mon[31859]: pgmap v38: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:09:00.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:00 smithi192 ceph-mon[36700]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:09:00.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:00 smithi062 ceph-mon[35761]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:09:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:00 smithi046 ceph-mon[31859]: pgmap v39: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:09:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:02 smithi192 ceph-mon[36700]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:09:02.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:02 smithi062 ceph-mon[35761]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:09:02.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:02 smithi046 ceph-mon[31859]: pgmap v40: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:09:03.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:03 smithi192 ceph-mon[36700]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:03.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:03 smithi062 ceph-mon[35761]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:03.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:03 smithi046 ceph-mon[31859]: pgmap v41: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:05.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:05 smithi192 ceph-mon[36700]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:05.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:05 smithi062 ceph-mon[35761]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:05.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:05 smithi046 ceph-mon[31859]: pgmap v42: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:07.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:07 smithi192 ceph-mon[36700]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:07.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:07 smithi062 ceph-mon[35761]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:07.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:07 smithi046 ceph-mon[31859]: pgmap v43: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:09.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:09 smithi192 ceph-mon[36700]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:09.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:09 smithi062 ceph-mon[35761]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:09.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:09 smithi046 ceph-mon[31859]: pgmap v44: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:11.665 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 59% (7361/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 99% (12171/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-08T23:09:11.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:11 smithi192 ceph-mon[36700]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:11.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:11 smithi062 ceph-mon[35761]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:11.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:11 smithi046 ceph-mon[31859]: pgmap v45: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '7d63482aa89628d733e298984a762389b625c761'. 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-08T23:09:12.334 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:12.335 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 7d63482aa89 Merge remote-tracking branch 'rishabh/fsshell-deprecation' into rishabh-2024mar8 2024-03-08T23:09:12.342 DEBUG:teuthology.orchestra.run.smithi046:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-08T23:09:12.401 INFO:tasks.workunit.client.0.smithi046.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-08T23:09:12.408 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-08T23:09:12.408 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-08T23:09:13.329 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-08T23:09:13.454 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:09:13.454 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:09:13.461 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-08T23:09:13.548 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-08T23:09:13.553 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-08T23:09:13.553 INFO:tasks.workunit.client.0.smithi046.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-08T23:09:13.632 INFO:tasks.workunit.client.0.smithi046.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-08T23:09:13.643 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:09:13.643 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-08T23:09:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.670+0000 7fc998000640 1 -- 172.21.15.46:0/2981555688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc990105550 msgr2=0x7fc99010d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.670+0000 7fc998000640 1 --2- 172.21.15.46:0/2981555688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc990105550 0x7fc99010d120 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc988009ca0 tx=0x7fc98802c100 comp rx=0 tx=0).stop 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 -- 172.21.15.46:0/2981555688 shutdown_connections 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 --2- 172.21.15.46:0/2981555688 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc990105550 0x7fc99010d120 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 --2- 172.21.15.46:0/2981555688 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100e30 0x7fc990104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 --2- 172.21.15.46:0/2981555688 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990100480 0x7fc990100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 -- 172.21.15.46:0/2981555688 >> 172.21.15.46:0/2981555688 conn(0x7fc990077350 msgr2=0x7fc990077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 -- 172.21.15.46:0/2981555688 shutdown_connections 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.671+0000 7fc998000640 1 -- 172.21.15.46:0/2981555688 wait complete. 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.672+0000 7fc998000640 1 Processor -- start 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.672+0000 7fc998000640 1 -- start start 2024-03-08T23:09:13.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.672+0000 7fc998000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.672+0000 7fc998000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990105550 0x7fc99019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc998000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 0x7fc990197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc998000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc990110610 con 0x7fc990100480 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc998000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc990110490 con 0x7fc9901979d0 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc998000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc990110790 con 0x7fc990105550 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42222/0 (socket says 172.21.15.46:42222) 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc995600640 1 -- 172.21.15.46:0/4261545391 learned_addr learned my addr 172.21.15.46:0/4261545391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 0x7fc990197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc994c00640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990105550 0x7fc99019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 -- 172.21.15.46:0/4261545391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990105550 msgr2=0x7fc99019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990105550 0x7fc99019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 -- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 msgr2=0x7fc99019d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc996000640 1 -- 172.21.15.46:0/4261545391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9901984f0 con 0x7fc9901979d0 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.673+0000 7fc995600640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:13.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc996000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 0x7fc990197e80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc988009bf0 tx=0x7fc988035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc98e000640 1 -- 172.21.15.46:0/4261545391 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc988039040 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc990198780 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc98e000640 1 -- 172.21.15.46:0/4261545391 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc988044070 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc98e000640 1 -- 172.21.15.46:0/4261545391 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9880042a0 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.674+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9901a49d0 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.677+0000 7fc98e000640 1 -- 172.21.15.46:0/4261545391 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fc988004440 con 0x7fc9901979d0 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 Processor -- start 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 -- start start 2024-03-08T23:09:13.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0398be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb0394d410 con 0x55fb0398b9a0 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb038efd50 con 0x55fb0398aff0 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.678+0000 7fc99ac72740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb03898b80 con 0x55fb0399d1b0 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0398be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0398be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42232/0 (socket says 172.21.15.46:42232) 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc995600640 1 -- 172.21.15.46:0/1029408610 learned_addr learned my addr 172.21.15.46:0/1029408610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 -- 172.21.15.46:0/1029408610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 msgr2=0x55fb0399f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 -- 172.21.15.46:0/1029408610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 msgr2=0x55fb0398be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0398be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 -- 172.21.15.46:0/1029408610 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb036ce7d0 con 0x55fb0398aff0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc98e000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 0x7fc9780a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc994c00640 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398b3d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9901991c0 tx=0x7fc97c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeac8ffacb92d683 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.679+0000 7fc98e000640 1 -- 172.21.15.46:0/4261545391 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc9880f17e0 con 0x7fc9901979d0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.680+0000 7fc995600640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 0x7fc9780a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.680+0000 7fc982600640 1 -- 172.21.15.46:0/1029408610 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c011050 con 0x55fb0398aff0 2024-03-08T23:09:13.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.680+0000 7fc982600640 1 -- 172.21.15.46:0/1029408610 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c0111f0 con 0x55fb0398aff0 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.680+0000 7fc982600640 1 -- 172.21.15.46:0/1029408610 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c010430 con 0x55fb0398aff0 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- 172.21.15.46:0/1029408610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 msgr2=0x55fb0398b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398b3d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc9901991c0 tx=0x7fc97c004730 comp rx=0 tx=0).stop 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc995600640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 0x7fc9780a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x55fb0398c490 tx=0x7fc984005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- 172.21.15.46:0/1029408610 shutdown_connections 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0398be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1029408610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398b3d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- 172.21.15.46:0/1029408610 >> 172.21.15.46:0/1029408610 conn(0x55fb03985cb0 msgr2=0x55fb03986090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- 172.21.15.46:0/1029408610 shutdown_connections 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- 172.21.15.46:0/1029408610 wait complete. 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.681+0000 7fc99ac72740 1 -- start start 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 Processor -- start 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0398b9a0 0x55fb0398dac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 0x55fb0398e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb038abee0 con 0x55fb0399d1b0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb038ebdf0 con 0x55fb0398aff0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb039a0000 con 0x55fb0398b9a0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc99ac72740 10 client.0 _ll_register_callbacks cb 0x55fb038b28d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc996000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 0x55fb0398e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc995600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc994c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0398b9a0 0x55fb0398dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc996000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 0x55fb0398e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42240/0 (socket says 172.21.15.46:42240) 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.682+0000 7fc996000640 1 -- 172.21.15.46:0/1827329899 learned_addr learned my addr 172.21.15.46:0/1827329899 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 -- 172.21.15.46:0/1827329899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0398b9a0 msgr2=0x55fb0398dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0398b9a0 0x55fb0398dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 -- 172.21.15.46:0/1827329899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 msgr2=0x55fb0398e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 0x55fb0398e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 -- 172.21.15.46:0/1827329899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb036043f0 con 0x55fb0398aff0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc995600640 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398d580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc984012580 tx=0x7fc984004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=648933f64541967a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.683+0000 7fc976a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98400fe60 con 0x55fb0398aff0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc97c0111f0 con 0x55fb0398aff0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc984011b50 con 0x55fb0398aff0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc984011d50 con 0x55fb0398aff0 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc99ac72740 10 client.25880 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:13.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fb035c40f0 con 0x55fb0398aff0 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.684+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c0111f0 con 0x55fb0398aff0 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc984027070 con 0x55fb0398aff0 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 10 client.25880 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 10 client.25880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 20 client.25880 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb039ad480 0x55fb039af8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 4 client.25880 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb038fd290 con 0x55fb039ad480 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.685+0000 7fc994c00640 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb039ad480 0x55fb039af8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.686+0000 7fc994c00640 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb039ad480 0x55fb039af8c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970f952123d3a6ae server_cookie=e7760b464fe7d3ca in_seq=0 out_seq=0 2024-03-08T23:09:13.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.686+0000 7fc976a00640 10 client.25880 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc976a00640 1 -- 172.21.15.46:0/1827329899 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb038fd290 con 0x55fb039ad480 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc976a00640 10 client.25880 handle_command_reply: tid=0 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 client.25880 shutdown 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 msgr2=0x55fb0398d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398d580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc984012580 tx=0x7fc984004730 comp rx=0 tx=0).stop 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 shutdown_connections 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb039ad480 0x55fb039af8c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0399d1b0 0x55fb0398e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0398b9a0 0x55fb0398dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 --2- 172.21.15.46:0/1827329899 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398d580 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 >> 172.21.15.46:0/1827329899 conn(0x55fb03985cb0 msgr2=0x55fb03987a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc976a00640 0 client.25880 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 shutdown_connections 2024-03-08T23:09:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 1 -- 172.21.15.46:0/1827329899 wait complete. 2024-03-08T23:09:13.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.687+0000 7fc99ac72740 20 client.25880 trim_cache size 0 max 16384 2024-03-08T23:09:13.702 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-08T23:09:13.703 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-08T23:09:13.703 DEBUG:teuthology.orchestra.run.smithi046:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-08T23:09:13.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.704+0000 7fc99ac72740 1 Processor -- start 2024-03-08T23:09:13.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.704+0000 7fc99ac72740 1 -- start start 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901ae700 con 0x55fb0398b9a0 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901ac2a0 con 0x55fb0398aff0 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc99ac72740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901aa340 con 0x55fb0399d1b0 2024-03-08T23:09:13.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc994c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc994c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42254/0 (socket says 172.21.15.46:42254) 2024-03-08T23:09:13.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc994c00640 1 -- 172.21.15.46:0/3391524978 learned_addr learned my addr 172.21.15.46:0/3391524978 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc996000640 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.705+0000 7fc995600640 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 -- 172.21.15.46:0/3391524978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 msgr2=0x55fb0399eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 -- 172.21.15.46:0/3391524978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 msgr2=0x55fb0398fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 -- 172.21.15.46:0/3391524978 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb0398a670 con 0x55fb0398b9a0 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc994c00640 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc97c004a60 tx=0x7fc97c004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351f86747ceb34f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc981c00640 1 -- 172.21.15.46:0/3391524978 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c004040 con 0x55fb0398b9a0 2024-03-08T23:09:13.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc981c00640 1 -- 172.21.15.46:0/3391524978 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c02e020 con 0x55fb0398b9a0 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.706+0000 7fc981c00640 1 -- 172.21.15.46:0/3391524978 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c003850 con 0x55fb0398b9a0 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 -- 172.21.15.46:0/3391524978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 msgr2=0x55fb0399e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc97c004a60 tx=0x7fc97c004a90 comp rx=0 tx=0).stop 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 -- 172.21.15.46:0/3391524978 shutdown_connections 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb0399eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398b9a0 0x55fb0399e5a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 --2- 172.21.15.46:0/3391524978 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398aff0 0x55fb0398fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 -- 172.21.15.46:0/3391524978 >> 172.21.15.46:0/3391524978 conn(0x55fb03985cb0 msgr2=0x55fb0399f410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 -- 172.21.15.46:0/3391524978 shutdown_connections 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.707+0000 7fc99ac72740 1 -- 172.21.15.46:0/3391524978 wait complete. 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 -- start start 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 Processor -- start 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398b9a0 0x55fb039a33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb039a3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901aa940 con 0x55fb0398aff0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901abc40 con 0x55fb0398b9a0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9901aaf80 con 0x55fb0399d1b0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.708+0000 7fc99ac72740 10 client.0 _ll_register_callbacks cb 0x7fc97c009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42264/0 (socket says 172.21.15.46:42264) 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 -- 172.21.15.46:0/859236449 learned_addr learned my addr 172.21.15.46:0/859236449 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc996000640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb039a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc994c00640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398b9a0 0x55fb039a33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 -- 172.21.15.46:0/859236449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 msgr2=0x55fb039a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb039a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 -- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398b9a0 msgr2=0x55fb039a33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398b9a0 0x55fb039a33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 -- 172.21.15.46:0/859236449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb03a03c00 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc995600640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc984009640 tx=0x7fc984029c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549fea09cafe224b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.709+0000 7fc976a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98403c060 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc98401f8d0 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc97c02e020 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc984043050 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc99ac72740 10 client.15765 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55fb03a04ef0 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c02e020 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc984027040 con 0x55fb0398aff0 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc99ac72740 10 client.15765 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.710+0000 7fc99ac72740 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:13.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc99ac72740 20 client.15765 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:13.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb03a05590 0x55fb03a079b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc99ac72740 4 client.15765 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:09:13.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fb038f75e0 con 0x55fb03a05590 2024-03-08T23:09:13.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc994c00640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb03a05590 0x55fb03a079b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc994c00640 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb03a05590 0x55fb03a079b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6583d879cfe46ffe server_cookie=d48378c5b4d1a5b6 in_seq=0 out_seq=0 2024-03-08T23:09:13.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.711+0000 7fc976a00640 10 client.15765 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:13.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc976a00640 1 -- 172.21.15.46:0/859236449 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fb038f75e0 con 0x55fb03a05590 2024-03-08T23:09:13.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc976a00640 10 client.15765 handle_command_reply: tid=0 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 client.15765 shutdown 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 msgr2=0x55fb039a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc984009640 tx=0x7fc984029c00 comp rx=0 tx=0).stop 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 shutdown_connections 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fb03a05590 0x55fb03a079b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fb0399d1b0 0x55fb039a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fb0398b9a0 0x55fb039a33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 --2- 172.21.15.46:0/859236449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fb0398aff0 0x55fb039a2e80 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 >> 172.21.15.46:0/859236449 conn(0x55fb03985cb0 msgr2=0x55fb03984f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc976a00640 0 client.15765 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 shutdown_connections 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 1 -- 172.21.15.46:0/859236449 wait complete. 2024-03-08T23:09:13.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.712+0000 7fc99ac72740 20 client.15765 trim_cache size 0 max 16384 2024-03-08T23:09:13.721 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:09:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:09:13.721 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 msgr2=0x7fc9780a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 0x7fc9780a4fd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x55fb0398c490 tx=0x7fc984005f90 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 msgr2=0x7fc990197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 0x7fc990197e80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc988009bf0 tx=0x7fc988035d10 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 shutdown_connections 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9780a2b10 0x7fc9780a4fd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9901979d0 0x7fc990197e80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc990105550 0x7fc99019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 --2- 172.21.15.46:0/4261545391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc990100480 0x7fc99019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 >> 172.21.15.46:0/4261545391 conn(0x7fc990077350 msgr2=0x7fc990069b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 shutdown_connections 2024-03-08T23:09:13.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.713+0000 7fc998000640 1 -- 172.21.15.46:0/4261545391 wait complete. 2024-03-08T23:09:13.735 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:09:13.735 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:09:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ set -e 2024-03-08T23:09:13.761 INFO:tasks.workunit.client.0.smithi046.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-08T23:09:13.767 INFO:tasks.workunit.client.0.smithi046.stderr:Cloning into 'xfstests-dev'... 2024-03-08T23:09:13.829 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:13 smithi046 ceph-mon[31859]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:13.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:13 smithi062 ceph-mon[35761]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:13.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.961+0000 7f2792e00640 1 -- 172.21.15.46:0/403197782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c103f00 msgr2=0x7f278c107f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.961+0000 7f2792e00640 1 --2- 172.21.15.46:0/403197782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c103f00 0x7f278c107f40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2778009900 tx=0x7f277802bc40 comp rx=0 tx=0).stop 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 -- 172.21.15.46:0/403197782 shutdown_connections 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 --2- 172.21.15.46:0/403197782 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 0x7f278c1101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 --2- 172.21.15.46:0/403197782 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c103f00 0x7f278c107f40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 --2- 172.21.15.46:0/403197782 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 -- 172.21.15.46:0/403197782 >> 172.21.15.46:0/403197782 conn(0x7f278c0fed90 msgr2=0x7f278c1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 -- 172.21.15.46:0/403197782 shutdown_connections 2024-03-08T23:09:13.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.962+0000 7f2792e00640 1 -- 172.21.15.46:0/403197782 wait complete. 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 Processor -- start 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 -- start start 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 0x7f278c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c1136e0 con 0x7f278c108620 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c113560 con 0x7f278c199a40 2024-03-08T23:09:13.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.963+0000 7f2792e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278c113860 con 0x7f278c103550 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2791a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2791000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 0x7f278c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40962/0 (socket says 172.21.15.46:40962) 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 -- 172.21.15.46:0/255940706 learned_addr learned my addr 172.21.15.46:0/255940706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2791a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c19f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59870/0 (socket says 172.21.15.46:59870) 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 -- 172.21.15.46:0/255940706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 msgr2=0x7f278c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 -- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 msgr2=0x7f278c19fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 0x7f278c19fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.964+0000 7f2792400640 1 -- 172.21.15.46:0/255940706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c19a560 con 0x7f278c199a40 2024-03-08T23:09:13.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f2792400640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f278400eb10 tx=0x7f278400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784004b20 con 0x7f278c199a40 2024-03-08T23:09:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c19a850 con 0x7f278c199a40 2024-03-08T23:09:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2784004cc0 con 0x7f278c199a40 2024-03-08T23:09:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2784010760 con 0x7f278c199a40 2024-03-08T23:09:13.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.965+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c1a6aa0 con 0x7f278c199a40 2024-03-08T23:09:13.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.966+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278c068000 con 0x7f278c199a40 2024-03-08T23:09:13.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.968+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f2784005020 con 0x7f278c199a40 2024-03-08T23:09:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.970+0000 7f278a000640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 0x7f27740a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.970+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f27840cf990 con 0x7f278c199a40 2024-03-08T23:09:13.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.970+0000 7f2791a00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 0x7f27740a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:13.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.971+0000 7f2791a00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 0x7f27740a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f278c19ab80 tx=0x7f2780005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:13.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:13.974+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f278409c0e0 con 0x7f278c199a40 2024-03-08T23:09:14.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:13 smithi192 ceph-mon[36700]: pgmap v46: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:09:14.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.229+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f278c103f00 con 0x7f278c199a40 2024-03-08T23:09:14.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.230+0000 7f278a000640 1 -- 172.21.15.46:0/255940706 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f278409b9e0 con 0x7f278c199a40 2024-03-08T23:09:14.234 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:09:14.234 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:07:25.750030+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:09:14.235 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 24 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 msgr2=0x7f27740a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 0x7f27740a4fd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f278c19ab80 tx=0x7f2780005f90 comp rx=0 tx=0).stop 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 msgr2=0x7f278c199ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f278400eb10 tx=0x7f278400c2d0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 shutdown_connections 2024-03-08T23:09:14.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f27740a2b10 0x7f27740a4fd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f278c199a40 0x7f278c199ef0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f278c108620 0x7f278c19fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 --2- 172.21.15.46:0/255940706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f278c103550 0x7f278c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.233+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 >> 172.21.15.46:0/255940706 conn(0x7f278c0fed90 msgr2=0x7f278c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.234+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 shutdown_connections 2024-03-08T23:09:14.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.234+0000 7f2792e00640 1 -- 172.21.15.46:0/255940706 wait complete. 2024-03-08T23:09:14.256 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:14.256 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 -- 172.21.15.46:0/3533905559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e010b930 msgr2=0x7f39e010dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 --2- 172.21.15.46:0/3533905559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e010b930 0x7f39e010dda0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009900 tx=0x7f39d802bcc0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 -- 172.21.15.46:0/3533905559 shutdown_connections 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 --2- 172.21.15.46:0/3533905559 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e010b930 0x7f39e010dda0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 --2- 172.21.15.46:0/3533905559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e0102e80 0x7f39e010b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 --2- 172.21.15.46:0/3533905559 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e0102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 -- 172.21.15.46:0/3533905559 >> 172.21.15.46:0/3533905559 conn(0x7f39e0077350 msgr2=0x7f39e0077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 -- 172.21.15.46:0/3533905559 shutdown_connections 2024-03-08T23:09:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.526+0000 7f39e6600640 1 -- 172.21.15.46:0/3533905559 wait complete. 2024-03-08T23:09:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 Processor -- start 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 -- start start 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e0102e80 0x7f39e01006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e010b930 0x7f39e0100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e0111730 con 0x7f39e0102e80 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01115b0 con 0x7f39e0102560 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.527+0000 7f39e6600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01118b0 con 0x7f39e010b930 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39dea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e0102e80 0x7f39e01006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40976/0 (socket says 172.21.15.46:40976) 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 -- 172.21.15.46:0/2536036831 learned_addr learned my addr 172.21.15.46:0/2536036831 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39dfe00640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e010b930 0x7f39e0100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 -- 172.21.15.46:0/2536036831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e010b930 msgr2=0x7f39e0100c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e010b930 0x7f39e0100c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 -- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e0102e80 msgr2=0x7f39e01006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e0102e80 0x7f39e01006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.528+0000 7f39df400640 1 -- 172.21.15.46:0/2536036831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e01a6090 con 0x7f39e0102560 2024-03-08T23:09:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39df400640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39d000ebc0 tx=0x7f39d000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39d7e00640 1 -- 172.21.15.46:0/2536036831 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0010630 con 0x7f39e0102560 2024-03-08T23:09:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e01a6380 con 0x7f39e0102560 2024-03-08T23:09:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39d7e00640 1 -- 172.21.15.46:0/2536036831 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d0010cb0 con 0x7f39e0102560 2024-03-08T23:09:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39d7e00640 1 -- 172.21.15.46:0/2536036831 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d0018bd0 con 0x7f39e0102560 2024-03-08T23:09:14.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.529+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e01a68d0 con 0x7f39e0102560 2024-03-08T23:09:14.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.532+0000 7f39e9375740 1 Processor -- start 2024-03-08T23:09:14.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.532+0000 7f39d7e00640 1 -- 172.21.15.46:0/2536036831 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f39d00107d0 con 0x7f39e0102560 2024-03-08T23:09:14.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.532+0000 7f39e9375740 1 -- start start 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561bab310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561badc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561b06dd0 con 0x557561ba2350 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561aafc70 con 0x557561ba2c70 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39e9375740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561b02e70 con 0x557561bab850 2024-03-08T23:09:14.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39df400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561bab310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42294/0 (socket says 172.21.15.46:42294) 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 -- 172.21.15.46:0/1441386542 learned_addr learned my addr 172.21.15.46:0/1441386542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dea00640 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561badc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 -- 172.21.15.46:0/1441386542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 msgr2=0x557561badc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561badc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 -- 172.21.15.46:0/1441386542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 msgr2=0x557561bab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.533+0000 7f39dfe00640 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561bab310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39dfe00640 1 -- 172.21.15.46:0/1441386542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557561b692b0 con 0x557561ba2350 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39d7e00640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 0x7f39c40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39dfe00640 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f39d802c2d0 tx=0x7f39d8009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc66d3d5143d37c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39d7e00640 1 -- 172.21.15.46:0/2536036831 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f39d00cfad0 con 0x7f39e0102560 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39d4c00640 1 -- 172.21.15.46:0/1441386542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8043070 con 0x557561ba2350 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39df400640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 0x7f39c40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39d4c00640 1 -- 172.21.15.46:0/1441386542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d802c910 con 0x557561ba2350 2024-03-08T23:09:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.534+0000 7f39d4c00640 1 -- 172.21.15.46:0/1441386542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8004ae0 con 0x557561ba2350 2024-03-08T23:09:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 -- 172.21.15.46:0/1441386542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 msgr2=0x557561ba2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39df400640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 0x7f39c40a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ccd40 tx=0x7f39d00d3560 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f39d802c2d0 tx=0x7f39d8009610 comp rx=0 tx=0).stop 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 -- 172.21.15.46:0/1441386542 shutdown_connections 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561badc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561bab310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 --2- 172.21.15.46:0/1441386542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561ba2730 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 -- 172.21.15.46:0/1441386542 >> 172.21.15.46:0/1441386542 conn(0x557561b9ca80 msgr2=0x557561b9ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 -- 172.21.15.46:0/1441386542 shutdown_connections 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.535+0000 7f39e9375740 1 -- 172.21.15.46:0/1441386542 wait complete. 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.536+0000 7f39e9375740 1 -- start start 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.536+0000 7f39e9375740 1 Processor -- start 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.536+0000 7f39e9375740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.536+0000 7f39e9375740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561ba85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.536+0000 7f39e9375740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561ba8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39e9375740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561bae5e0 con 0x557561ba2350 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39e9375740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561ac30c0 con 0x557561ba2c70 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39e9375740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557561bae760 con 0x557561bab850 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39e9375740 10 client.0 _ll_register_callbacks cb 0x557561afdec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561ba85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42310/0 (socket says 172.21.15.46:42310) 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 -- 172.21.15.46:0/4073820046 learned_addr learned my addr 172.21.15.46:0/4073820046 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561ba8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 -- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 msgr2=0x557561ba8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561ba8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 -- 172.21.15.46:0/4073820046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 msgr2=0x557561ba85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561ba85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 -- 172.21.15.46:0/4073820046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55756181b3f0 con 0x557561ba2350 2024-03-08T23:09:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39dfe00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005ab0 tx=0x7f39d8009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a7e2bdf5912b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.537+0000 7f39cd600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8043040 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f39d8005100 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39d802c910 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39d8057050 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39e9375740 10 client.15783 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5575618e57d0 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.538+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d802c910 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f39d8048c10 con 0x557561ba2350 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 10 client.15783 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 10 client.15783 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 20 client.15783 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561bbb730 0x557561bbdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 4 client.15783 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557561adb4f0 con 0x557561bbb730 2024-03-08T23:09:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.539+0000 7f39dea00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561bbb730 0x557561bbdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.540+0000 7f39dea00640 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561bbb730 0x557561bbdb20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8f393346abc84455 server_cookie=96085e59a6b701c in_seq=0 out_seq=0 2024-03-08T23:09:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.540+0000 7f39cd600640 10 client.15783 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39cd600640 1 -- 172.21.15.46:0/4073820046 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557561adb4f0 con 0x557561bbb730 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39cd600640 10 client.15783 handle_command_reply: tid=0 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 client.15783 shutdown 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 msgr2=0x557561bab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f39d8005ab0 tx=0x7f39d8009680 comp rx=0 tx=0).stop 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 shutdown_connections 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561bbb730 0x557561bbdb20 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561bab850 0x557561ba8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2c70 0x557561ba85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 --2- 172.21.15.46:0/4073820046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bab220 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39cd600640 0 client.15783 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 >> 172.21.15.46:0/4073820046 conn(0x557561b9ca80 msgr2=0x557561b9e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.541+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 shutdown_connections 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.542+0000 7f39e9375740 1 -- 172.21.15.46:0/4073820046 wait complete. 2024-03-08T23:09:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.542+0000 7f39e9375740 20 client.15783 trim_cache size 0 max 16384 2024-03-08T23:09:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 Processor -- start 2024-03-08T23:09:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 -- start start 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2350 0x557561ba7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561baace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01b2b00 con 0x557561bab850 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01b06a0 con 0x557561ba2350 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39e9375740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01ae740 con 0x557561ba2c70 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42314/0 (socket says 172.21.15.46:42314) 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39dfe00640 1 -- 172.21.15.46:0/56725406 learned_addr learned my addr 172.21.15.46:0/56725406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39dfe00640 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2350 0x557561ba7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.555+0000 7f39dea00640 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561baace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 -- 172.21.15.46:0/56725406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 msgr2=0x557561baace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561baace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 -- 172.21.15.46:0/56725406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2350 msgr2=0x557561ba7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2350 0x557561ba7590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 -- 172.21.15.46:0/56725406 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557561bc2970 con 0x557561bab850 2024-03-08T23:09:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39dfe00640 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f39d803f680 tx=0x7f39d8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a59565d3504d1f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39cfe00640 1 -- 172.21.15.46:0/56725406 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8043070 con 0x557561bab850 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39cfe00640 1 -- 172.21.15.46:0/56725406 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d802c910 con 0x557561bab850 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.556+0000 7f39cfe00640 1 -- 172.21.15.46:0/56725406 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8005180 con 0x557561bab850 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 -- 172.21.15.46:0/56725406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 msgr2=0x557561bab220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f39d803f680 tx=0x7f39d8005fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 -- 172.21.15.46:0/56725406 shutdown_connections 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561bab850 0x557561bab220 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561baace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 --2- 172.21.15.46:0/56725406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561ba2350 0x557561ba7590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 -- 172.21.15.46:0/56725406 >> 172.21.15.46:0/56725406 conn(0x557561b9ca80 msgr2=0x557561ba6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 -- 172.21.15.46:0/56725406 shutdown_connections 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.557+0000 7f39e9375740 1 -- 172.21.15.46:0/56725406 wait complete. 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 -- start start 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 Processor -- start 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bb2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561bab850 0x557561bb1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01aed40 con 0x557561ba2350 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01b0040 con 0x557561bab850 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e01af380 con 0x557561ba2c70 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.558+0000 7f39e9375740 10 client.0 _ll_register_callbacks cb 0x7f39d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bb2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59914/0 (socket says 172.21.15.46:59914) 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dea00640 1 -- 172.21.15.46:0/3187799347 learned_addr learned my addr 172.21.15.46:0/3187799347 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561bab850 0x557561bb1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 -- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 msgr2=0x557561bb0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 -- 172.21.15.46:0/3187799347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561bab850 msgr2=0x557561bb1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561bab850 0x557561bb1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 -- 172.21.15.46:0/3187799347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557561bb1c40 con 0x557561ba2350 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dea00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39dfe00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bb2c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009900 tx=0x7f39d803f700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60bc3242441ff88c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39cd600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39d8043040 con 0x557561ba2350 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.559+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f39d8004040 con 0x557561ba2350 2024-03-08T23:09:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39d802c910 con 0x557561ba2350 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39d80046c0 con 0x557561ba2350 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 10 client.15789 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x557561c127e0 con 0x557561ba2350 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39d802c910 con 0x557561ba2350 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f39d80035f0 con 0x557561ba2350 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 10 client.15789 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 10 client.15789 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 20 client.15789 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561c12d40 0x557561c15130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 4 client.15789 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.560+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557561b14310 con 0x557561c12d40 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.561+0000 7f39dea00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561c12d40 0x557561c15130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.561+0000 7f39dea00640 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561c12d40 0x557561c15130 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9b3922531b1794c server_cookie=b47e10b0aeffb67 in_seq=0 out_seq=0 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.561+0000 7f39cd600640 10 client.15789 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39cd600640 1 -- 172.21.15.46:0/3187799347 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557561b14310 con 0x557561c12d40 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39cd600640 10 client.15789 handle_command_reply: tid=0 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 client.15789 shutdown 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 msgr2=0x557561bb2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bb2c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f39d8009900 tx=0x7f39d803f700 comp rx=0 tx=0).stop 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 shutdown_connections 2024-03-08T23:09:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x557561c12d40 0x557561c15130 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557561bab850 0x557561bb1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557561ba2c70 0x557561bb0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 --2- 172.21.15.46:0/3187799347 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557561ba2350 0x557561bb2c20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 >> 172.21.15.46:0/3187799347 conn(0x557561b9ca80 msgr2=0x557561b9b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39cd600640 0 client.15789 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 shutdown_connections 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 1 -- 172.21.15.46:0/3187799347 wait complete. 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.562+0000 7f39e9375740 20 client.15789 trim_cache size 0 max 16384 2024-03-08T23:09:14.570 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 msgr2=0x7f39c40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 0x7f39c40a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f39d00ccd40 tx=0x7f39d00d3560 comp rx=0 tx=0).stop 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 msgr2=0x7f39e01001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f39d000ebc0 tx=0x7f39d000c6a0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 shutdown_connections 2024-03-08T23:09:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f39c40a2a90 0x7f39c40a4f50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f39e010b930 0x7f39e0100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f39e0102e80 0x7f39e01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 --2- 172.21.15.46:0/2536036831 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f39e0102560 0x7f39e01001b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 >> 172.21.15.46:0/2536036831 conn(0x7f39e0077350 msgr2=0x7f39e00fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 shutdown_connections 2024-03-08T23:09:14.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.563+0000 7f39e6600640 1 -- 172.21.15.46:0/2536036831 wait complete. 2024-03-08T23:09:14.586 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:14.586 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 -- 172.21.15.46:0/1037084062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac075820 msgr2=0x7f8cac075c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/1037084062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac075820 0x7f8cac075c00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009900 tx=0x7f8ca002bc50 comp rx=0 tx=0).stop 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 -- 172.21.15.46:0/1037084062 shutdown_connections 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/1037084062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac112130 0x7f8cac114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/1037084062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac076140 0x7f8cac111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/1037084062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac075820 0x7f8cac075c00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 -- 172.21.15.46:0/1037084062 >> 172.21.15.46:0/1037084062 conn(0x7f8cac0fef60 msgr2=0x7f8cac101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 -- 172.21.15.46:0/1037084062 shutdown_connections 2024-03-08T23:09:14.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.802+0000 7f8cb3c00640 1 -- 172.21.15.46:0/1037084062 wait complete. 2024-03-08T23:09:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.803+0000 7f8cb3c00640 1 Processor -- start 2024-03-08T23:09:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.803+0000 7f8cb3c00640 1 -- start start 2024-03-08T23:09:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 0x7f8cac1a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac117ce0 con 0x7f8cac076140 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac117b60 con 0x7f8cac112130 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb3c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac117e60 con 0x7f8cac075820 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 0x7f8cac1a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59928/0 (socket says 172.21.15.46:59928) 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1200640 1 -- 172.21.15.46:0/2646492935 learned_addr learned my addr 172.21.15.46:0/2646492935 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42326/0 (socket says 172.21.15.46:42326) 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 -- 172.21.15.46:0/2646492935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 msgr2=0x7f8cac19e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 -- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 msgr2=0x7f8cac19e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1c00640 1 -- 172.21.15.46:0/2646492935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cac1a3190 con 0x7f8cac112130 2024-03-08T23:09:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cb1200640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.804+0000 7f8cabe00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8cb1c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 0x7f8cac1a2a60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4005fd0 tx=0x7f8ca400edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8ca9600640 1 -- 172.21.15.46:0/2646492935 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca4004d20 con 0x7f8cac112130 2024-03-08T23:09:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8ca9600640 1 -- 172.21.15.46:0/2646492935 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca4005020 con 0x7f8cac112130 2024-03-08T23:09:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cac1a3480 con 0x7f8cac112130 2024-03-08T23:09:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8ca9600640 1 -- 172.21.15.46:0/2646492935 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca40116d0 con 0x7f8cac112130 2024-03-08T23:09:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.805+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cac1aad10 con 0x7f8cac112130 2024-03-08T23:09:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 Processor -- start 2024-03-08T23:09:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8ca9600640 1 -- 172.21.15.46:0/2646492935 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f8ca4005540 con 0x7f8cac112130 2024-03-08T23:09:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 -- start start 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fa8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584f0cdd0 con 0x559584fa8c70 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584eb5c70 con 0x559584fb1850 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.809+0000 7f8cb698e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584f08e70 con 0x559584fa8350 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42334/0 (socket says 172.21.15.46:42334) 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 -- 172.21.15.46:0/778497211 learned_addr learned my addr 172.21.15.46:0/778497211 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cabe00640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fa8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cabe00640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 -- 172.21.15.46:0/778497211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 msgr2=0x559584fa8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fa8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 -- 172.21.15.46:0/778497211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 msgr2=0x559584fb3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 -- 172.21.15.46:0/778497211 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559584f6f2b0 con 0x559584fa8c70 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cabe00640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fa8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8ca9600640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 0x7f8c940a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.810+0000 7f8cb1200640 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036040 tx=0x7f8ca0005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2160162e512098d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8ca9600640 1 -- 172.21.15.46:0/2646492935 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8ca40cf7b0 con 0x7f8cac112130 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8c9e000640 1 -- 172.21.15.46:0/778497211 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0004c70 con 0x559584fa8c70 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8c9e000640 1 -- 172.21.15.46:0/778497211 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca002c8d0 con 0x559584fa8c70 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8c9e000640 1 -- 172.21.15.46:0/778497211 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0005550 con 0x559584fa8c70 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cabe00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 0x7f8c940a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 -- 172.21.15.46:0/778497211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 msgr2=0x559584fb1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0036040 tx=0x7f8ca0005e60 comp rx=0 tx=0).stop 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 -- 172.21.15.46:0/778497211 shutdown_connections 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb1310 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 --2- 172.21.15.46:0/778497211 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fa8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cabe00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 0x7f8c940a4f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x559584fa3c10 tx=0x7f8c98005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.811+0000 7f8cb698e740 1 -- 172.21.15.46:0/778497211 >> 172.21.15.46:0/778497211 conn(0x559584fa2a80 msgr2=0x559584fa2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.812+0000 7f8cb698e740 1 -- 172.21.15.46:0/778497211 shutdown_connections 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.812+0000 7f8cb698e740 1 -- 172.21.15.46:0/778497211 wait complete. 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.812+0000 7f8cb698e740 1 -- start start 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.812+0000 7f8cb698e740 1 Processor -- start 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584faea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584fb45e0 con 0x559584fa8350 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584ec90c0 con 0x559584fb1850 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559584fb4760 con 0x559584fa8c70 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb698e740 10 client.0 _ll_register_callbacks cb 0x559584f03ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42346/0 (socket says 172.21.15.46:42346) 2024-03-08T23:09:14.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 -- 172.21.15.46:0/263540442 learned_addr learned my addr 172.21.15.46:0/263540442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cabe00640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1c00640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584faea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 -- 172.21.15.46:0/263540442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 msgr2=0x559584fae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 -- 172.21.15.46:0/263540442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 msgr2=0x559584faea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584faea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.813+0000 7f8cb1200640 1 -- 172.21.15.46:0/263540442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559584c213f0 con 0x559584fa8350 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8cb1200640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009690 tx=0x7f8ca0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65e2918a642d6f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8c92a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0055060 con 0x559584fa8350 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8ca0005290 con 0x559584fa8350 2024-03-08T23:09:14.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ca002c8d0 con 0x559584fa8350 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.814+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ca005c050 con 0x559584fa8350 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 10 client.15801 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559584ceb7d0 con 0x559584fa8350 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca002c8d0 con 0x559584fa8350 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ca0046400 con 0x559584fa8350 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 10 client.15801 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 10 client.15801 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 20 client.15801 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559584fc16b0 0x559584fc3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 4 client.15801 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.815+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559584ee14f0 con 0x559584fc16b0 2024-03-08T23:09:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.816+0000 7f8cabe00640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559584fc16b0 0x559584fc3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.816+0000 7f8cabe00640 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559584fc16b0 0x559584fc3aa0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3d447d3841e0ddfa server_cookie=f2b5eacebfc197ef in_seq=0 out_seq=0 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.816+0000 7f8c92a00640 10 client.15801 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8c92a00640 1 -- 172.21.15.46:0/263540442 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559584ee14f0 con 0x559584fc16b0 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8c92a00640 10 client.15801 handle_command_reply: tid=0 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 client.15801 shutdown 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 msgr2=0x559584fb1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8ca0009690 tx=0x7f8ca0009260 comp rx=0 tx=0).stop 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 shutdown_connections 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584faea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559584fc16b0 0x559584fc3aa0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 --2- 172.21.15.46:0/263540442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8350 0x559584fb1190 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8c92a00640 0 client.15801 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:14.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 >> 172.21.15.46:0/263540442 conn(0x559584fa2a80 msgr2=0x559584fa47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 shutdown_connections 2024-03-08T23:09:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.817+0000 7f8cb698e740 1 -- 172.21.15.46:0/263540442 wait complete. 2024-03-08T23:09:14.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.818+0000 7f8cb698e740 20 client.15801 trim_cache size 0 max 16384 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.830+0000 7f8cb698e740 1 Processor -- start 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 -- start start 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fb1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 0x559584fab7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b6fd0 con 0x559584fb1850 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b4b70 con 0x559584fa8350 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb698e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b2c10 con 0x559584fa8c70 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb1c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 0x559584fab7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41050/0 (socket says 172.21.15.46:41050) 2024-03-08T23:09:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb1c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 0x559584fab7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42358/0 (socket says 172.21.15.46:42358) 2024-03-08T23:09:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cb1200640 1 -- 172.21.15.46:0/2217874395 learned_addr learned my addr 172.21.15.46:0/2217874395 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.831+0000 7f8cabe00640 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fb1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 -- 172.21.15.46:0/2217874395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 msgr2=0x559584fb1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fb1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 -- 172.21.15.46:0/2217874395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 msgr2=0x559584fab7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 0x559584fab7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 -- 172.21.15.46:0/2217874395 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559584faca70 con 0x559584fa8350 2024-03-08T23:09:14.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8cb1200640 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00096c0 tx=0x7f8ca0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88bb0deebe356a3c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8c9d600640 1 -- 172.21.15.46:0/2217874395 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0065070 con 0x559584fa8350 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8c9d600640 1 -- 172.21.15.46:0/2217874395 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca002cc90 con 0x559584fa8350 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.832+0000 7f8c9d600640 1 -- 172.21.15.46:0/2217874395 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0055070 con 0x559584fa8350 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- 172.21.15.46:0/2217874395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 msgr2=0x559584fb0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8ca00096c0 tx=0x7f8ca0005fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- 172.21.15.46:0/2217874395 shutdown_connections 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fb1850 0x559584fab7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8c70 0x559584fb1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 --2- 172.21.15.46:0/2217874395 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fa8350 0x559584fb0c50 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- 172.21.15.46:0/2217874395 >> 172.21.15.46:0/2217874395 conn(0x559584fa2a80 msgr2=0x559584facdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- 172.21.15.46:0/2217874395 shutdown_connections 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- 172.21.15.46:0/2217874395 wait complete. 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.833+0000 7f8cb698e740 1 -- start start 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 Processor -- start 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b3210 con 0x559584fa8c70 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b4510 con 0x559584fb1850 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cac1b3850 con 0x559584fa8350 2024-03-08T23:09:14.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb698e740 10 client.0 _ll_register_callbacks cb 0x7f8ca0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb1200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59960/0 (socket says 172.21.15.46:59960) 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cb1200640 1 -- 172.21.15.46:0/3339924247 learned_addr learned my addr 172.21.15.46:0/3339924247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.834+0000 7f8cabe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42370/0 (socket says 172.21.15.46:42370) 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1c00640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 -- 172.21.15.46:0/3339924247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 msgr2=0x559584fb7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 -- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 msgr2=0x559584fb7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 -- 172.21.15.46:0/3339924247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5595850178f0 con 0x559584fa8350 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cabe00640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb1200640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ca002c260 tx=0x7f8ca0005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f61aecb8eb05368 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:14.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8c92a00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0065030 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ca002cc90 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.835+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8ca004b040 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.836+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ca0055040 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.836+0000 7f8cb698e740 10 client.25942 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.836+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x559585018bf0 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.836+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ca002cc90 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.836+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ca0034440 con 0x559584fa8350 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 10 client.25942 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 10 client.25942 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 20 client.25942 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559585018f80 0x55958501b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 4 client.25942 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559584f1a310 con 0x559585018f80 2024-03-08T23:09:14.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cabe00640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559585018f80 0x55958501b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8cabe00640 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559585018f80 0x55958501b3a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d62831e5f92a8aa5 server_cookie=fb470c0f7db93edd in_seq=0 out_seq=0 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.837+0000 7f8c92a00640 10 client.25942 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8c92a00640 1 -- 172.21.15.46:0/3339924247 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559584f1a310 con 0x559585018f80 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8c92a00640 10 client.25942 handle_command_reply: tid=0 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 client.25942 shutdown 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 msgr2=0x559584fb74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8ca002c260 tx=0x7f8ca0005a20 comp rx=0 tx=0).stop 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 shutdown_connections 2024-03-08T23:09:14.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559584fb1850 0x559584fb7f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559584fa8c70 0x559584fb7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x559585018f80 0x55958501b3a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 --2- 172.21.15.46:0/3339924247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559584fa8350 0x559584fb74f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8c92a00640 0 client.25942 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 >> 172.21.15.46:0/3339924247 conn(0x559584fa2a80 msgr2=0x559584fa9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 shutdown_connections 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 1 -- 172.21.15.46:0/3339924247 wait complete. 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.838+0000 7f8cb698e740 20 client.25942 trim_cache size 0 max 16384 2024-03-08T23:09:14.847 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 msgr2=0x7f8c940a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 0x7f8c940a4f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x559584fa3c10 tx=0x7f8c98005f90 comp rx=0 tx=0).stop 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 msgr2=0x7f8cac1a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 0x7f8cac1a2a60 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8ca4005fd0 tx=0x7f8ca400edd0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 shutdown_connections 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8c940a2a90 0x7f8c940a4f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8cac112130 0x7f8cac1a2a60 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8cac076140 0x7f8cac19e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 --2- 172.21.15.46:0/2646492935 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8cac075820 0x7f8cac19e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 >> 172.21.15.46:0/2646492935 conn(0x7f8cac0fef60 msgr2=0x7f8cac113740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 shutdown_connections 2024-03-08T23:09:14.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:14.839+0000 7f8cb3c00640 1 -- 172.21.15.46:0/2646492935 wait complete. 2024-03-08T23:09:14.859 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/255940706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:14.861 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:14.862 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:09:14.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/255940706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:15.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/255940706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 -- 172.21.15.46:0/1183686874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8112130 msgr2=0x7fa2e8114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 --2- 172.21.15.46:0/1183686874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8112130 0x7fa2e8114520 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009c10 tx=0x7fa2e002bee0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 -- 172.21.15.46:0/1183686874 shutdown_connections 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 --2- 172.21.15.46:0/1183686874 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8112130 0x7fa2e8114520 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 --2- 172.21.15.46:0/1183686874 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 0x7fa2e8111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 --2- 172.21.15.46:0/1183686874 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8075820 0x7fa2e8075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 -- 172.21.15.46:0/1183686874 >> 172.21.15.46:0/1183686874 conn(0x7fa2e80fefa0 msgr2=0x7fa2e81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 -- 172.21.15.46:0/1183686874 shutdown_connections 2024-03-08T23:09:15.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.073+0000 7fa2ef800640 1 -- 172.21.15.46:0/1183686874 wait complete. 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.074+0000 7fa2ef800640 1 Processor -- start 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 -- start start 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 0x7fa2e819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8112130 0x7fa2e81a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8117ce0 con 0x7fa2e8112130 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8117b60 con 0x7fa2e8075820 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ef800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8117e60 con 0x7fa2e8076140 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41088/0 (socket says 172.21.15.46:41088) 2024-03-08T23:09:15.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ece00640 1 -- 172.21.15.46:0/2161128337 learned_addr learned my addr 172.21.15.46:0/2161128337 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2ed800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8112130 0x7fa2e81a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.075+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 0x7fa2e819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 -- 172.21.15.46:0/2161128337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 msgr2=0x7fa2e819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 0x7fa2e819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 -- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8112130 msgr2=0x7fa2e81a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8112130 0x7fa2e81a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 -- 172.21.15.46:0/2161128337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e81a31e0 con 0x7fa2e8075820 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9c0 tx=0x7fa2dc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2e5600640 1 -- 172.21.15.46:0/2161128337 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc00cde0 con 0x7fa2e8075820 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2e5600640 1 -- 172.21.15.46:0/2161128337 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2dc0035f0 con 0x7fa2e8075820 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e81a34d0 con 0x7fa2e8075820 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.076+0000 7fa2e5600640 1 -- 172.21.15.46:0/2161128337 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc010690 con 0x7fa2e8075820 2024-03-08T23:09:15.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.077+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e81aad60 con 0x7fa2e8075820 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.081+0000 7fa2f25c7740 1 Processor -- start 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.081+0000 7fa2f25c7740 1 -- start start 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.081+0000 7fa2e5600640 1 -- 172.21.15.46:0/2161128337 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fa2dc0037a0 con 0x7fa2e8075820 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1ca3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1cac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1c07dd0 con 0x55afd1ca3350 2024-03-08T23:09:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1bb0c70 con 0x55afd1cac850 2024-03-08T23:09:15.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1c03e70 con 0x55afd1ca3c70 2024-03-08T23:09:15.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2ed800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1ca3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1cac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41104/0 (socket says 172.21.15.46:41104) 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 -- 172.21.15.46:0/1241073001 learned_addr learned my addr 172.21.15.46:0/1241073001 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 -- 172.21.15.46:0/1241073001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 msgr2=0x55afd1cac310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1cac310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 -- 172.21.15.46:0/1241073001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 msgr2=0x55afd1ca3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1ca3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.082+0000 7fa2e7e00640 1 -- 172.21.15.46:0/1241073001 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd1c6a2b0 con 0x55afd1cac850 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009c20 tx=0x7fa2d8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69f22f090e14e885 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2d6000640 1 -- 172.21.15.46:0/1241073001 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8004230 con 0x55afd1cac850 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2e5600640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 0x7fa2d00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2d6000640 1 -- 172.21.15.46:0/1241073001 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d80043d0 con 0x55afd1cac850 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2d6000640 1 -- 172.21.15.46:0/1241073001 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8004d00 con 0x55afd1cac850 2024-03-08T23:09:15.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.083+0000 7fa2e5600640 1 -- 172.21.15.46:0/2161128337 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa2dc0cf7c0 con 0x7fa2e8075820 2024-03-08T23:09:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2ed800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 0x7fa2d00a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 -- 172.21.15.46:0/1241073001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 msgr2=0x55afd1caec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8009c20 tx=0x7fa2d8011810 comp rx=0 tx=0).stop 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 -- 172.21.15.46:0/1241073001 shutdown_connections 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1cac850 0x55afd1caec40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1cac310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/1241073001 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1ca3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 -- 172.21.15.46:0/1241073001 >> 172.21.15.46:0/1241073001 conn(0x55afd1c9da80 msgr2=0x55afd1c9de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2ed800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 0x7fa2d00a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2e002c420 tx=0x7fa2e0005fd0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 -- 172.21.15.46:0/1241073001 shutdown_connections 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.084+0000 7fa2f25c7740 1 -- 172.21.15.46:0/1241073001 wait complete. 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.085+0000 7fa2f25c7740 1 -- start start 2024-03-08T23:09:15.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.085+0000 7fa2f25c7740 1 Processor -- start 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1ca94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 0x55afd1ca9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1caf640 con 0x55afd1cac850 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1bc40c0 con 0x55afd1ca3350 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55afd1caf7c0 con 0x55afd1ca3c70 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2f25c7740 10 client.0 _ll_register_callbacks cb 0x55afd1bfeec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41112/0 (socket says 172.21.15.46:41112) 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 -- 172.21.15.46:0/2742231724 learned_addr learned my addr 172.21.15.46:0/2742231724 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ed800640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 0x55afd1ca9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1ca94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 -- 172.21.15.46:0/2742231724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 msgr2=0x55afd1ca94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1ca94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 -- 172.21.15.46:0/2742231724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 msgr2=0x55afd1ca9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 0x55afd1ca9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ece00640 1 -- 172.21.15.46:0/2742231724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afd191c3f0 con 0x55afd1ca3350 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1ca94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.086+0000 7fa2ed800640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 0x55afd1ca9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc09b570 tx=0x7fa2dc09b3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2ae31e7a9e6977a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2caa00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc09b870 con 0x55afd1ca3350 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa2dc0cf400 con 0x55afd1ca3350 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2d80043d0 con 0x55afd1ca3350 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2dc0cf600 con 0x55afd1ca3350 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2f25c7740 10 client.25940 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:15.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.087+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55afd19e67d0 con 0x55afd1ca3350 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d80043d0 con 0x55afd1ca3350 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa2dc09cba0 con 0x55afd1ca3350 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 10 client.25940 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 10 client.25940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 20 client.25940 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1cbc790 0x55afd1cbeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 4 client.25940 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55afd1bdc4f0 con 0x55afd1cbc790 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.088+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1cbc790 0x55afd1cbeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.089+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1cbc790 0x55afd1cbeb80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34426bee186eb5b3 server_cookie=924a5597534ba5e5 in_seq=0 out_seq=0 2024-03-08T23:09:15.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.089+0000 7fa2caa00640 10 client.25940 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:15.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2caa00640 1 -- 172.21.15.46:0/2742231724 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55afd1bdc4f0 con 0x55afd1cbc790 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2caa00640 10 client.25940 handle_command_reply: tid=0 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 client.25940 shutdown 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 msgr2=0x55afd1cac150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc09b570 tx=0x7fa2dc09b3b0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 shutdown_connections 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1cbc790 0x55afd1cbeb80 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1cac850 0x55afd1ca9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1ca3c70 0x55afd1ca94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2742231724 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3350 0x55afd1cac150 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2caa00640 0 client.25940 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 >> 172.21.15.46:0/2742231724 conn(0x55afd1c9da80 msgr2=0x55afd1c9f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.090+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 shutdown_connections 2024-03-08T23:09:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.091+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2742231724 wait complete. 2024-03-08T23:09:15.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.091+0000 7fa2f25c7740 20 client.25940 trim_cache size 0 max 16384 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 Processor -- start 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 -- start start 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cac150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1ca67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b6fd0 con 0x55afd1ca3350 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b4b70 con 0x55afd1ca3c70 2024-03-08T23:09:15.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.110+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b2c10 con 0x55afd1cac850 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ed800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1ca67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cac150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42424/0 (socket says 172.21.15.46:42424) 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ed800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1ca67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:60006/0 (socket says 172.21.15.46:60006) 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ece00640 1 -- 172.21.15.46:0/2317243289 learned_addr learned my addr 172.21.15.46:0/2317243289 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 -- 172.21.15.46:0/2317243289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 msgr2=0x55afd1ca67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1ca67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 -- 172.21.15.46:0/2317243289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 msgr2=0x55afd1cabc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2e7e00640 1 -- 172.21.15.46:0/2317243289 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55afd1cbeef0 con 0x55afd1ca3c70 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.111+0000 7fa2ece00640 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.112+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cac150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8011d10 tx=0x7fa2d8009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ad92cc364af5834 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.112+0000 7fa2d5600640 1 -- 172.21.15.46:0/2317243289 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8004030 con 0x55afd1ca3c70 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.112+0000 7fa2d5600640 1 -- 172.21.15.46:0/2317243289 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d801f040 con 0x55afd1ca3c70 2024-03-08T23:09:15.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.112+0000 7fa2d5600640 1 -- 172.21.15.46:0/2317243289 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8003870 con 0x55afd1ca3c70 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2317243289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 msgr2=0x55afd1cac150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cac150 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8011d10 tx=0x7fa2d8009fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2317243289 shutdown_connections 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1ca67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cac150 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/2317243289 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cabc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2317243289 >> 172.21.15.46:0/2317243289 conn(0x55afd1c9da80 msgr2=0x55afd1ca7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2317243289 shutdown_connections 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.113+0000 7fa2f25c7740 1 -- 172.21.15.46:0/2317243289 wait complete. 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 -- start start 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 Processor -- start 2024-03-08T23:09:15.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cb24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cb29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b3210 con 0x55afd1ca3350 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b4510 con 0x55afd1ca3c70 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81b3850 con 0x55afd1cac850 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.114+0000 7fa2f25c7740 10 client.0 _ll_register_callbacks cb 0x7fa2d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ece00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cb24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:60008/0 (socket says 172.21.15.46:60008) 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 -- 172.21.15.46:0/3077494990 learned_addr learned my addr 172.21.15.46:0/3077494990 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2e7e00640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cb29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 -- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 msgr2=0x55afd1cb29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cb29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 -- 172.21.15.46:0/3077494990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 msgr2=0x55afd1cb24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cb24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ed800640 1 -- 172.21.15.46:0/3077494990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55afd1d12cb0 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.115+0000 7fa2ece00640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cb24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2ed800640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa2e806a780 tx=0x7fa2e0009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30f75e3abcba2395 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2caa00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e0003c70 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa2e0004590 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2d801f040 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2e0061050 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2f25c7740 10 client.25957 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.116+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55afd1d13f90 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2d801f040 con 0x55afd1cac850 2024-03-08T23:09:15.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa2e004f440 con 0x55afd1cac850 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 10 client.25957 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 10 client.25957 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 20 client.25957 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1d146c0 0x55afd1d16ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 4 client.25957 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55afd1c15310 con 0x55afd1d146c0 2024-03-08T23:09:15.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.117+0000 7fa2ece00640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1d146c0 0x55afd1d16ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:15.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.118+0000 7fa2caa00640 10 client.25957 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:15.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.118+0000 7fa2ece00640 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1d146c0 0x55afd1d16ae0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e131deca1d6f7d85 server_cookie=b2dd6064fc5450db in_seq=0 out_seq=0 2024-03-08T23:09:15.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2caa00640 1 -- 172.21.15.46:0/3077494990 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55afd1c15310 con 0x55afd1d146c0 2024-03-08T23:09:15.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2caa00640 10 client.25957 handle_command_reply: tid=0 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 client.25957 shutdown 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 msgr2=0x55afd1cb2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa2e806a780 tx=0x7fa2e0009500 comp rx=0 tx=0).stop 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 shutdown_connections 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55afd1d146c0 0x55afd1d16ae0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55afd1cac850 0x55afd1cb2f20 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55afd1ca3c70 0x55afd1cb29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 --2- 172.21.15.46:0/3077494990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55afd1ca3350 0x55afd1cb24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2caa00640 0 client.25957 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 >> 172.21.15.46:0/3077494990 conn(0x55afd1c9da80 msgr2=0x55afd1ca4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 shutdown_connections 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 1 -- 172.21.15.46:0/3077494990 wait complete. 2024-03-08T23:09:15.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.119+0000 7fa2f25c7740 20 client.25957 trim_cache size 0 max 16384 2024-03-08T23:09:15.127 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:15.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 msgr2=0x7fa2d00a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 0x7fa2d00a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa2e002c420 tx=0x7fa2e0005fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 msgr2=0x7fa2e819e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc00e9c0 tx=0x7fa2dc00ee80 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 shutdown_connections 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2d00a2a90 0x7fa2d00a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2e8112130 0x7fa2e81a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2e8076140 0x7fa2e819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 --2- 172.21.15.46:0/2161128337 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa2e8075820 0x7fa2e819e240 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 >> 172.21.15.46:0/2161128337 conn(0x7fa2e80fefa0 msgr2=0x7fa2e8101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 shutdown_connections 2024-03-08T23:09:15.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:15.120+0000 7fa2ef800640 1 -- 172.21.15.46:0/2161128337 wait complete. 2024-03-08T23:09:15.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:15 smithi062 ceph-mon[35761]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:15.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:15 smithi046 ceph-mon[31859]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:16.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:15 smithi192 ceph-mon[36700]: pgmap v47: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:16.142 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:09:16.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:09:16.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:09:16.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 -- 172.21.15.46:0/2339338077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 msgr2=0x7f8a30114560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 --2- 172.21.15.46:0/2339338077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a30114560 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009ca0 tx=0x7f8a2802c190 comp rx=0 tx=0).stop 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 -- 172.21.15.46:0/2339338077 shutdown_connections 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 --2- 172.21.15.46:0/2339338077 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a30114560 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 --2- 172.21.15.46:0/2339338077 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a30111c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 --2- 172.21.15.46:0/2339338077 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a30073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 -- 172.21.15.46:0/2339338077 >> 172.21.15.46:0/2339338077 conn(0x7f8a300fcc50 msgr2=0x7f8a300ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 -- 172.21.15.46:0/2339338077 shutdown_connections 2024-03-08T23:09:16.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.360+0000 7f8a2f200640 1 -- 172.21.15.46:0/2339338077 wait complete. 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.361+0000 7f8a2f200640 1 Processor -- start 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.361+0000 7f8a2f200640 1 -- start start 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a3019e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a301a2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30117b70 con 0x7f8a30073fb0 2024-03-08T23:09:16.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301179f0 con 0x7f8a30073690 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a30117cf0 con 0x7f8a30112170 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a3019e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a301a2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42452/0 (socket says 172.21.15.46:42452) 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a3019e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41148/0 (socket says 172.21.15.46:41148) 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 -- 172.21.15.46:0/3900302622 learned_addr learned my addr 172.21.15.46:0/3900302622 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 -- 172.21.15.46:0/3900302622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 msgr2=0x7f8a301a2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a301a2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 -- 172.21.15.46:0/3900302622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 msgr2=0x7f8a3019e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a3019e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2d400640 1 -- 172.21.15.46:0/3900302622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a301a3060 con 0x7f8a30073fb0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.362+0000 7f8a2e800640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a301a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a2d400640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00e9f0 tx=0x7f8a1c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a26a00640 1 -- 172.21.15.46:0/3900302622 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c00cd80 con 0x7f8a30073fb0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a26a00640 1 -- 172.21.15.46:0/3900302622 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a1c010910 con 0x7f8a30073fb0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a301a3350 con 0x7f8a30073fb0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a26a00640 1 -- 172.21.15.46:0/3900302622 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c018990 con 0x7f8a30073fb0 2024-03-08T23:09:16.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.363+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a301aabe0 con 0x7f8a30073fb0 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 Processor -- start 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 -- start start 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a26a00640 1 -- 172.21.15.46:0/3900302622 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f8a1c010ac0 con 0x7f8a30073fb0 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb245760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb248090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb1a0a50 con 0x564abb23d0c0 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb149b00 con 0x564abb245ca0 2024-03-08T23:09:16.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb19caf0 con 0x564abb23c7a0 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.367+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:60038/0 (socket says 172.21.15.46:60038) 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2e800640 1 -- 172.21.15.46:0/2868449375 learned_addr learned my addr 172.21.15.46:0/2868449375 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb245760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2d400640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb248090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 -- 172.21.15.46:0/2868449375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 msgr2=0x564abb23cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 -- 172.21.15.46:0/2868449375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 msgr2=0x564abb248090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb248090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 -- 172.21.15.46:0/2868449375 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564abb1671e0 con 0x564abb23d0c0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2e800640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.368+0000 7f8a2de00640 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb245760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009c00 tx=0x7f8a20011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93be753f2b0f844d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a1b400640 1 -- 172.21.15.46:0/2868449375 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004230 con 0x564abb23d0c0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a1b400640 1 -- 172.21.15.46:0/2868449375 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a200043d0 con 0x564abb23d0c0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a1b400640 1 -- 172.21.15.46:0/2868449375 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20004d70 con 0x564abb23d0c0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a26a00640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 0x7f8a140a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a26a00640 1 -- 172.21.15.46:0/3900302622 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8a1c0cf720 con 0x7f8a30073fb0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.369+0000 7f8a2e800640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 0x7f8a140a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- 172.21.15.46:0/2868449375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 msgr2=0x564abb245760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb245760 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f8a20009c00 tx=0x7f8a20011810 comp rx=0 tx=0).stop 2024-03-08T23:09:16.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- 172.21.15.46:0/2868449375 shutdown_connections 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb248090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb245760 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/2868449375 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23c7a0 0x564abb23cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- 172.21.15.46:0/2868449375 >> 172.21.15.46:0/2868449375 conn(0x564abb236ed0 msgr2=0x564abb2372b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- 172.21.15.46:0/2868449375 shutdown_connections 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a2e800640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 0x7f8a140a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009a60 tx=0x7f8a28005d80 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- 172.21.15.46:0/2868449375 wait complete. 2024-03-08T23:09:16.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.370+0000 7f8a37dd7740 1 -- start start 2024-03-08T23:09:16.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 Processor -- start 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb243230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 0x564abb243770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb248a90 con 0x564abb23d0c0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb15cf60 con 0x564abb23c7a0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564abb248c10 con 0x564abb245ca0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a37dd7740 10 client.0 _ll_register_callbacks cb 0x564abb148b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a2d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb243230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41166/0 (socket says 172.21.15.46:41166) 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a2de00640 1 -- 172.21.15.46:0/1901292151 learned_addr learned my addr 172.21.15.46:0/1901292151 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.371+0000 7f8a2e800640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 0x564abb243770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 -- 172.21.15.46:0/1901292151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 msgr2=0x564abb243770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 0x564abb243770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 -- 172.21.15.46:0/1901292151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 msgr2=0x564abb243230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb243230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 -- 172.21.15.46:0/1901292151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564abaf7d7e0 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2e800640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 0x564abb243770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8a20011d10 tx=0x7f8a200199e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8e08383b4360f85 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.372+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20028020 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8a20024620 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a200043d0 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a20033050 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a37dd7740 10 client.25958 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564abafddf70 con 0x564abb23c7a0 2024-03-08T23:09:16.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.373+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a200043d0 con 0x564abb23c7a0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8a20018070 con 0x564abb23c7a0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 10 client.25958 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 10 client.25958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 20 client.25958 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb255fe0 0x564abb2583d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 4 client.25958 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564abb1adf90 con 0x564abb255fe0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.374+0000 7f8a2d400640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb255fe0 0x564abb2583d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.375+0000 7f8a2d400640 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb255fe0 0x564abb2583d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=362a1b3e341d966d server_cookie=2ca2f7584bae3885 in_seq=0 out_seq=0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.375+0000 7f8a0fe00640 10 client.25958 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.375+0000 7f8a0fe00640 1 -- 172.21.15.46:0/1901292151 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564abb1adf90 con 0x564abb255fe0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.375+0000 7f8a0fe00640 10 client.25958 handle_command_reply: tid=0 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.375+0000 7f8a37dd7740 1 client.25958 shutdown 2024-03-08T23:09:16.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 msgr2=0x564abb242cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8a20011d10 tx=0x7f8a200199e0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 shutdown_connections 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb255fe0 0x564abb2583d0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb245ca0 0x564abb243770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23d0c0 0x564abb243230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a0fe00640 0 client.25958 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1901292151 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb242cf0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 >> 172.21.15.46:0/1901292151 conn(0x564abb236ed0 msgr2=0x564abb238c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 shutdown_connections 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1901292151 wait complete. 2024-03-08T23:09:16.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.376+0000 7f8a37dd7740 20 client.25958 trim_cache size 0 max 16384 2024-03-08T23:09:16.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 Processor -- start 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 -- start start 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb2424a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb245ca0 0x564abb23ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b6f40 con 0x564abb245ca0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b4ae0 con 0x564abb23c7a0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b2b80 con 0x564abb23d0c0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb245ca0 0x564abb23ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a2de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41168/0 (socket says 172.21.15.46:41168) 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a2de00640 1 -- 172.21.15.46:0/1642327329 learned_addr learned my addr 172.21.15.46:0/1642327329 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.394+0000 7f8a2d400640 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb2424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 -- 172.21.15.46:0/1642327329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 msgr2=0x564abb2424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb2424a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 -- 172.21.15.46:0/1642327329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb245ca0 msgr2=0x564abb23ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb245ca0 0x564abb23ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 -- 172.21.15.46:0/1642327329 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564abae75170 con 0x564abb23c7a0 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2d400640 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb2424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a2de00640 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8a20011850 tx=0x7f8a20009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4106df26502e3c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a1aa00640 1 -- 172.21.15.46:0/1642327329 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20028070 con 0x564abb23c7a0 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a1aa00640 1 -- 172.21.15.46:0/1642327329 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a2001f040 con 0x564abb23c7a0 2024-03-08T23:09:16.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.395+0000 7f8a1aa00640 1 -- 172.21.15.46:0/1642327329 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a20024420 con 0x564abb23c7a0 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1642327329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 msgr2=0x564abb245420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8a20011850 tx=0x7f8a20009620 comp rx=0 tx=0).stop 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1642327329 shutdown_connections 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb245ca0 0x564abb23ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb2424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/1642327329 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb23c7a0 0x564abb245420 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1642327329 >> 172.21.15.46:0/1642327329 conn(0x564abb236ed0 msgr2=0x564abb238bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1642327329 shutdown_connections 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- 172.21.15.46:0/1642327329 wait complete. 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.396+0000 7f8a37dd7740 1 -- start start 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 Processor -- start 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23c7a0 0x564abb24b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb24c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b3180 con 0x564abb23c7a0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b4480 con 0x564abb245ca0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a301b37c0 con 0x564abb23d0c0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a37dd7740 10 client.0 _ll_register_callbacks cb 0x7f8a20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a2d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb24c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a2d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:60074/0 (socket says 172.21.15.46:60074) 2024-03-08T23:09:16.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb24c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41178/0 (socket says 172.21.15.46:41178) 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.397+0000 7f8a2d400640 1 -- 172.21.15.46:0/621731065 learned_addr learned my addr 172.21.15.46:0/621731065 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2de00640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23c7a0 0x564abb24b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 -- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 msgr2=0x564abb24c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb24c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 -- 172.21.15.46:0/621731065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23c7a0 msgr2=0x564abb24b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23c7a0 0x564abb24b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 -- 172.21.15.46:0/621731065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564abb16c050 con 0x564abb23d0c0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a2d400640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0cd110 tx=0x7f8a1c0d1c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f177c59416d88fb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a0fe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.398+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c09b880 con 0x564abb23d0c0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f8a1c097ef0 con 0x564abb23d0c0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8a2001f040 con 0x564abb23d0c0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a1c09f920 con 0x564abb23d0c0 2024-03-08T23:09:16.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 10 client.25975 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564abb2ad4c0 con 0x564abb23d0c0 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a2001f040 con 0x564abb23d0c0 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8a1c0a3ae0 con 0x564abb23d0c0 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 10 client.25975 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 10 client.25975 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 20 client.25975 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb2adbf0 0x564abb2b0010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 4 client.25975 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.399+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564abb1a82e0 con 0x564abb2adbf0 2024-03-08T23:09:16.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.400+0000 7f8a2de00640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb2adbf0 0x564abb2b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.400+0000 7f8a2de00640 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb2adbf0 0x564abb2b0010 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b37c7fe39771278f server_cookie=26acbd3712c281de in_seq=0 out_seq=0 2024-03-08T23:09:16.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.400+0000 7f8a0fe00640 10 client.25975 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a0fe00640 1 -- 172.21.15.46:0/621731065 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564abb1a82e0 con 0x564abb2adbf0 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a0fe00640 10 client.25975 handle_command_reply: tid=0 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 client.25975 shutdown 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 msgr2=0x564abb24bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c0cd110 tx=0x7f8a1c0d1c30 comp rx=0 tx=0).stop 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 shutdown_connections 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564abb2adbf0 0x564abb2b0010 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564abb245ca0 0x564abb24c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564abb23d0c0 0x564abb24bd30 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 --2- 172.21.15.46:0/621731065 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564abb23c7a0 0x564abb24b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a0fe00640 0 client.25975 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 >> 172.21.15.46:0/621731065 conn(0x564abb236ed0 msgr2=0x564abb2356e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 shutdown_connections 2024-03-08T23:09:16.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.401+0000 7f8a37dd7740 1 -- 172.21.15.46:0/621731065 wait complete. 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a37dd7740 20 client.25975 trim_cache size 0 max 16384 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "6fd69973-a7cc-431e-a4f3-5d141a3f5f31", 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:09:16.409 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 msgr2=0x7f8a140a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 0x7f8a140a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f8a28009a60 tx=0x7f8a28005d80 comp rx=0 tx=0).stop 2024-03-08T23:09:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 msgr2=0x7f8a3019e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00e9f0 tx=0x7f8a1c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 shutdown_connections 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f8a140a2a90 0x7f8a140a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8a30112170 0x7f8a301a2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8a30073fb0 0x7f8a3019e5b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 --2- 172.21.15.46:0/3900302622 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8a30073690 0x7f8a3019e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.402+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 >> 172.21.15.46:0/3900302622 conn(0x7f8a300fcc50 msgr2=0x7f8a300fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.403+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 shutdown_connections 2024-03-08T23:09:16.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.403+0000 7f8a2f200640 1 -- 172.21.15.46:0/3900302622 wait complete. 2024-03-08T23:09:16.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:09:16.423 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:09:16.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 -- 172.21.15.46:0/497851999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 msgr2=0x7f7d3010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 --2- 172.21.15.46:0/497851999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3010e1a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009ca0 tx=0x7f7d2c02c100 comp rx=0 tx=0).stop 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 -- 172.21.15.46:0/497851999 shutdown_connections 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 --2- 172.21.15.46:0/497851999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3010e1a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 --2- 172.21.15.46:0/497851999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d30101eb0 0x7f7d30105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 --2- 172.21.15.46:0/497851999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 0x7f7d301018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 -- 172.21.15.46:0/497851999 >> 172.21.15.46:0/497851999 conn(0x7f7d300fcd20 msgr2=0x7f7d300ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 -- 172.21.15.46:0/497851999 shutdown_connections 2024-03-08T23:09:16.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.622+0000 7f7d38800640 1 -- 172.21.15.46:0/497851999 wait complete. 2024-03-08T23:09:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.623+0000 7f7d38800640 1 Processor -- start 2024-03-08T23:09:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.623+0000 7f7d38800640 1 -- start start 2024-03-08T23:09:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 0x7f7d30199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 0x7f7d3019ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30111670 con 0x7f7d3019e620 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301114f0 con 0x7f7d301065d0 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d38800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301117f0 con 0x7f7d30101500 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d35400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 0x7f7d3019ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d35400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41198/0 (socket says 172.21.15.46:41198) 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d35400640 1 -- 172.21.15.46:0/1189240235 learned_addr learned my addr 172.21.15.46:0/1189240235 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d35e00640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 0x7f7d30199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 -- 172.21.15.46:0/1189240235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 msgr2=0x7f7d30199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 0x7f7d30199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 -- 172.21.15.46:0/1189240235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 msgr2=0x7f7d3019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.624+0000 7f7d36800640 1 -- 172.21.15.46:0/1189240235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3019f110 con 0x7f7d3019e620 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d35400640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d36800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 0x7f7d3019ead0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009bf0 tx=0x7f7d2c035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d2aa00640 1 -- 172.21.15.46:0/1189240235 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c039040 con 0x7f7d3019e620 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d2aa00640 1 -- 172.21.15.46:0/1189240235 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d2c044070 con 0x7f7d3019e620 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3019f3a0 con 0x7f7d3019e620 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d2aa00640 1 -- 172.21.15.46:0/1189240235 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c0042a0 con 0x7f7d3019e620 2024-03-08T23:09:16.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.625+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d301a6b00 con 0x7f7d3019e620 2024-03-08T23:09:16.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 Processor -- start 2024-03-08T23:09:16.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d2aa00640 1 -- 172.21.15.46:0/1189240235 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 26) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f7d2c004440 con 0x7f7d3019e620 2024-03-08T23:09:16.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 -- start start 2024-03-08T23:09:16.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737f7a80 0x5609737f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609737b0410 con 0x5609737ee580 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560973752d50 con 0x5609737f7a80 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.630+0000 7f7d3b587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609736fbb80 con 0x5609737eeea0 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:42506/0 (socket says 172.21.15.46:42506) 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35400640 1 -- 172.21.15.46:0/2484815380 learned_addr learned my addr 172.21.15.46:0/2484815380 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35400640 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737f7a80 0x5609737f9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 -- 172.21.15.46:0/2484815380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737f7a80 msgr2=0x5609737f9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737f7a80 0x5609737f9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 -- 172.21.15.46:0/2484815380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 msgr2=0x5609737ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35e00640 1 -- 172.21.15.46:0/2484815380 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609735317d0 con 0x5609737eeea0 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.631+0000 7f7d35400640 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d2aa00640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 0x7f7d180a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f7540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009c20 tx=0x7f7d24011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80f2b963fb09d112 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d2aa00640 1 -- 172.21.15.46:0/1189240235 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f7d2c0f1670 con 0x7f7d3019e620 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d35400640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 0x7f7d180a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d1f000640 1 -- 172.21.15.46:0/2484815380 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24004050 con 0x5609737eeea0 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d1f000640 1 -- 172.21.15.46:0/2484815380 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d24005500 con 0x5609737eeea0 2024-03-08T23:09:16.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.632+0000 7f7d1f000640 1 -- 172.21.15.46:0/2484815380 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24004db0 con 0x5609737eeea0 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 -- 172.21.15.46:0/2484815380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 msgr2=0x5609737f7540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f7540 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009c20 tx=0x7f7d24011810 comp rx=0 tx=0).stop 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d35400640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 0x7f7d180a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d20005f60 tx=0x7f7d20009590 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 -- 172.21.15.46:0/2484815380 shutdown_connections 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737f7a80 0x5609737f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f7540 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2484815380 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x5609737ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 -- 172.21.15.46:0/2484815380 >> 172.21.15.46:0/2484815380 conn(0x5609737e8cb0 msgr2=0x5609737e9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 -- 172.21.15.46:0/2484815380 shutdown_connections 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.633+0000 7f7d3b587740 1 -- 172.21.15.46:0/2484815380 wait complete. 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 -- start start 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 Processor -- start 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f5020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737f5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56097370eee0 con 0x5609737f7a80 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56097374edf0 con 0x5609737ee580 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5609737fa870 con 0x5609737eeea0 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.634+0000 7f7d3b587740 10 client.0 _ll_register_callbacks cb 0x5609737158d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41222/0 (socket says 172.21.15.46:41222) 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 -- 172.21.15.46:0/2623364770 learned_addr learned my addr 172.21.15.46:0/2623364770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35400640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d36800640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737f5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 -- 172.21.15.46:0/2623364770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 msgr2=0x5609737f5020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f5020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 -- 172.21.15.46:0/2623364770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 msgr2=0x5609737f5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737f5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 -- 172.21.15.46:0/2623364770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5609734673f0 con 0x5609737ee580 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d36800640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400a000 tx=0x7f7d24004500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8288ec95cccdac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.635+0000 7f7d13400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24029070 con 0x5609737ee580 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f7d24004f00 con 0x5609737ee580 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d24005500 con 0x5609737ee580 2024-03-08T23:09:16.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d24035050 con 0x5609737ee580 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d3b587740 10 client.25976 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5609734270f0 con 0x5609737ee580 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.636+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d24005500 con 0x5609737ee580 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7d24003d10 con 0x5609737ee580 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 10 client.25976 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 10 client.25976 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 20 client.25976 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973807ba0 0x560973809f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 4 client.25976 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560973760290 con 0x560973807ba0 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d35400640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973807ba0 0x560973809f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d13400640 10 client.25976 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.637+0000 7f7d35400640 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973807ba0 0x560973809f90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f9cb4231b7a48ad1 server_cookie=f69e7dff644df976 in_seq=0 out_seq=0 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d13400640 1 -- 172.21.15.46:0/2623364770 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560973760290 con 0x560973807ba0 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d13400640 10 client.25976 handle_command_reply: tid=0 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 client.25976 shutdown 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 msgr2=0x5609737f4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7d2400a000 tx=0x7f7d24004500 comp rx=0 tx=0).stop 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 shutdown_connections 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973807ba0 0x560973809f90 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737f5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737f5020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2623364770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f4ae0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 >> 172.21.15.46:0/2623364770 conn(0x5609737e8cb0 msgr2=0x5609737e7f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d13400640 0 client.25976 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 shutdown_connections 2024-03-08T23:09:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 1 -- 172.21.15.46:0/2623364770 wait complete. 2024-03-08T23:09:16.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.639+0000 7f7d3b587740 20 client.25976 trim_cache size 0 max 16384 2024-03-08T23:09:16.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.658+0000 7f7d3b587740 1 Processor -- start 2024-03-08T23:09:16.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.658+0000 7f7d3b587740 1 -- start start 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.658+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f7050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.658+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d3b587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301b2c00 con 0x5609737f7a80 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d3b587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301b07a0 con 0x5609737ee580 2024-03-08T23:09:16.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d3b587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301ae840 con 0x5609737eeea0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41232/0 (socket says 172.21.15.46:41232) 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35e00640 1 -- 172.21.15.46:0/1535227508 learned_addr learned my addr 172.21.15.46:0/1535227508 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d36800640 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 -- 172.21.15.46:0/1535227508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 msgr2=0x5609737f7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f7050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 -- 172.21.15.46:0/1535227508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 msgr2=0x5609737feff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737feff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d35400640 1 -- 172.21.15.46:0/1535227508 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5609737f3050 con 0x5609737eeea0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.659+0000 7f7d36800640 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d35400640 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737feab0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5609737f4340 tx=0x7f7d200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=532dd5517f69b9fa server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d1e600640 1 -- 172.21.15.46:0/1535227508 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000fbe0 con 0x5609737eeea0 2024-03-08T23:09:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d1e600640 1 -- 172.21.15.46:0/1535227508 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d2002b890 con 0x5609737eeea0 2024-03-08T23:09:16.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d1e600640 1 -- 172.21.15.46:0/1535227508 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20020420 con 0x5609737eeea0 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d3b587740 1 -- 172.21.15.46:0/1535227508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 msgr2=0x5609737feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.660+0000 7f7d3b587740 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737feab0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x5609737f4340 tx=0x7f7d200046c0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 -- 172.21.15.46:0/1535227508 shutdown_connections 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737f7a80 0x5609737feff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737eeea0 0x5609737feab0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 --2- 172.21.15.46:0/1535227508 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737ee580 0x5609737f7050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 -- 172.21.15.46:0/1535227508 >> 172.21.15.46:0/1535227508 conn(0x5609737e8cb0 msgr2=0x5609737ef2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 -- 172.21.15.46:0/1535227508 shutdown_connections 2024-03-08T23:09:16.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 -- 172.21.15.46:0/1535227508 wait complete. 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 -- start start 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.661+0000 7f7d3b587740 1 Processor -- start 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x560973860660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737f7a80 0x56097385aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301aee40 con 0x5609737ee580 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301b0140 con 0x5609737eeea0 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d301af480 con 0x5609737f7a80 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d3b587740 10 client.0 _ll_register_callbacks cb 0x7f7d240097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:41236/0 (socket says 172.21.15.46:41236) 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35400640 1 -- 172.21.15.46:0/2773230358 learned_addr learned my addr 172.21.15.46:0/2773230358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d36800640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737f7a80 0x56097385aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x560973860660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35400640 1 -- 172.21.15.46:0/2773230358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737f7a80 msgr2=0x56097385aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.662+0000 7f7d35400640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737f7a80 0x56097385aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d35400640 1 -- 172.21.15.46:0/2773230358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 msgr2=0x560973860660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d35400640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x560973860660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d35400640 1 -- 172.21.15.46:0/2773230358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56097385b160 con 0x5609737eeea0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d35400640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000fed0 tx=0x7f7d20004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7a33f311e8a2750 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d13400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000fbe0 con 0x5609737eeea0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7d2002b890 con 0x5609737eeea0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f7d2002b3b0 con 0x5609737eeea0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.663+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d20041050 con 0x5609737eeea0 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 10 client.25982 fetch_fsmap learned FSMap version 24 2024-03-08T23:09:16.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560973867de0 con 0x5609737eeea0 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d2002b890 con 0x5609737eeea0 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 24) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7d2001d070 con 0x5609737eeea0 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 10 client.25982 fetch_fsmap finished waiting for FSMap version 24 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 10 client.25982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 20 client.25982 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:16.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973868450 0x56097386a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 4 client.25982 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56097375a5e0 con 0x560973868450 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.664+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973868450 0x56097386a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d35e00640 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973868450 0x56097386a870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12ea02b6b708c2f4 server_cookie=b8eff05bbd5d597c in_seq=0 out_seq=0 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d13400640 10 client.25982 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d13400640 1 -- 172.21.15.46:0/2773230358 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56097375a5e0 con 0x560973868450 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d13400640 10 client.25982 handle_command_reply: tid=0 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 client.25982 shutdown 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 msgr2=0x560973860ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000fed0 tx=0x7f7d20004730 comp rx=0 tx=0).stop 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 shutdown_connections 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560973868450 0x56097386a870 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5609737f7a80 0x56097385aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5609737eeea0 0x560973860ba0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 --2- 172.21.15.46:0/2773230358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5609737ee580 0x560973860660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d13400640 0 client.25982 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.665+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 >> 172.21.15.46:0/2773230358 conn(0x5609737e8cb0 msgr2=0x5609737efe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.666+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 shutdown_connections 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.666+0000 7f7d3b587740 1 -- 172.21.15.46:0/2773230358 wait complete. 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.666+0000 7f7d3b587740 20 client.25982 trim_cache size 0 max 16384 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:09:16.673 INFO:teuthology.orchestra.run.smithi046.stdout: "6fd69973-a7cc-431e-a4f3-5d141a3f5f31": { 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "6fd69973-a7cc-431e-a4f3-5d141a3f5f31", 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:09:16.674 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 msgr2=0x7f7d180a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 0x7f7d180a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d20005f60 tx=0x7f7d20009590 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 msgr2=0x7f7d3019ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 0x7f7d3019ead0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009bf0 tx=0x7f7d2c035d10 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 shutdown_connections 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7d180a2a90 0x7f7d180a4f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7d3019e620 0x7f7d3019ead0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7d301065d0 0x7f7d3019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 --2- 172.21.15.46:0/1189240235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d30101500 0x7f7d30199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 >> 172.21.15.46:0/1189240235 conn(0x7f7d300fcd20 msgr2=0x7f7d300feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 shutdown_connections 2024-03-08T23:09:16.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:16.667+0000 7f7d38800640 1 -- 172.21.15.46:0/1189240235 wait complete. 2024-03-08T23:09:16.689 INFO:tasks.cephfs.filesystem:scrub status for tag:6fd69973-a7cc-431e-a4f3-5d141a3f5f31 - {'path': '/', 'tag': '6fd69973-a7cc-431e-a4f3-5d141a3f5f31', 'options': 'recursive,force'} 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:17 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:09:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:09:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:17.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:17.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:17 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: pgmap v48: 129 pgs: 129 active+clean; 1.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:09:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:17 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:09:19.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:19 smithi062 ceph-mon[35761]: pgmap v49: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-03-08T23:09:19.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:19 smithi062 ceph-mon[35761]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-08T23:09:19.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:19 smithi046 ceph-mon[31859]: pgmap v49: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-03-08T23:09:19.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:19 smithi046 ceph-mon[31859]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-08T23:09:20.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:19 smithi192 ceph-mon[36700]: pgmap v49: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2024-03-08T23:09:20.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:19 smithi192 ceph-mon[36700]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-03-08T23:09:21.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:21 smithi062 ceph-mon[35761]: pgmap v50: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-08T23:09:21.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:21 smithi046 ceph-mon[31859]: pgmap v50: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-08T23:09:22.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:21 smithi192 ceph-mon[36700]: pgmap v50: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 1.1 MiB/s wr, 10 op/s 2024-03-08T23:09:23.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:23 smithi062 ceph-mon[35761]: pgmap v51: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-03-08T23:09:23.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:23 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:09:23.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:23 smithi046 ceph-mon[31859]: pgmap v51: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-03-08T23:09:23.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:23 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:09:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:23 smithi192 ceph-mon[36700]: pgmap v51: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 1.2 MiB/s wr, 16 op/s 2024-03-08T23:09:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:23 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:09:25.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:25 smithi062 ceph-mon[35761]: pgmap v52: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:25.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:25 smithi046 ceph-mon[31859]: pgmap v52: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:26.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:25 smithi192 ceph-mon[36700]: pgmap v52: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:27.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:27 smithi062 ceph-mon[35761]: pgmap v53: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:27.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:27 smithi046 ceph-mon[31859]: pgmap v53: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:28.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:27 smithi192 ceph-mon[36700]: pgmap v53: 129 pgs: 129 active+clean; 15 MiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 14 op/s 2024-03-08T23:09:29.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:29 smithi062 ceph-mon[35761]: pgmap v54: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 22 op/s 2024-03-08T23:09:29.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:29 smithi046 ceph-mon[31859]: pgmap v54: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 22 op/s 2024-03-08T23:09:30.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:29 smithi192 ceph-mon[36700]: pgmap v54: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 1.4 MiB/s wr, 22 op/s 2024-03-08T23:09:31.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:31 smithi062 ceph-mon[35761]: pgmap v55: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 250 KiB/s wr, 14 op/s 2024-03-08T23:09:31.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:31 smithi046 ceph-mon[31859]: pgmap v55: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 250 KiB/s wr, 14 op/s 2024-03-08T23:09:32.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:31 smithi192 ceph-mon[36700]: pgmap v55: 129 pgs: 129 active+clean; 17 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 250 KiB/s wr, 14 op/s 2024-03-08T23:09:33.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:33 smithi062 ceph-mon[35761]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 607 KiB/s wr, 35 op/s 2024-03-08T23:09:33.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:33 smithi046 ceph-mon[31859]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 607 KiB/s wr, 35 op/s 2024-03-08T23:09:34.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:33 smithi192 ceph-mon[36700]: pgmap v56: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 607 KiB/s wr, 35 op/s 2024-03-08T23:09:35.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:35 smithi046 ceph-mon[31859]: pgmap v57: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:36.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:35 smithi192 ceph-mon[36700]: pgmap v57: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:36.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:35 smithi062 ceph-mon[35761]: pgmap v57: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:37.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:37 smithi046 ceph-mon[31859]: pgmap v58: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:37 smithi192 ceph-mon[36700]: pgmap v58: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:37 smithi062 ceph-mon[35761]: pgmap v58: 129 pgs: 129 active+clean; 21 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 563 KiB/s wr, 29 op/s 2024-03-08T23:09:39.294 INFO:tasks.workunit.client.0.smithi046.stderr:Updating files: 11% (490/4261) Updating files: 12% (512/4261) Updating files: 13% (554/4261) Updating files: 14% (597/4261) Updating files: 15% (640/4261) Updating files: 16% (682/4261) Updating files: 17% (725/4261) Updating files: 18% (767/4261) Updating files: 19% (810/4261) Updating files: 19% (820/4261) Updating files: 20% (853/4261) Updating files: 21% (895/4261) Updating files: 22% (938/4261) Updating files: 23% (981/4261) Updating files: 24% (1023/4261) Updating files: 25% (1066/4261) Updating files: 26% (1108/4261) Updating files: 27% (1151/4261) Updating files: 28% (1194/4261) Updating files: 29% (1236/4261) Updating files: 29% (1267/4261) Updating files: 30% (1279/4261) Updating files: 31% (1321/4261) Updating files: 32% (1364/4261) Updating files: 33% (1407/4261) Updating files: 34% (1449/4261) Updating files: 35% (1492/4261) Updating files: 36% (1534/4261) Updating files: 37% (1577/4261) Updating files: 37% (1597/4261) Updating files: 38% (1620/4261) Updating files: 39% (1662/4261) Updating files: 40% (1705/4261) Updating files: 41% (1748/4261) Updating files: 42% (1790/4261) Updating files: 43% (1833/4261) Updating files: 44% (1875/4261) Updating files: 45% (1918/4261) Updating files: 46% (1961/4261) Updating files: 47% (2003/4261) Updating files: 48% (2046/4261) Updating files: 49% (2088/4261) Updating files: 50% (2131/4261) Updating files: 51% (2174/4261) Updating files: 52% (2216/4261) Updating files: 52% (2224/4261) Updating files: 53% (2259/4261) Updating files: 54% (2301/4261) Updating files: 55% (2344/4261) Updating files: 56% (2387/4261) Updating files: 57% (2429/4261) Updating files: 58% (2472/4261) Updating files: 59% (2514/4261) Updating files: 60% (2557/4261) Updating files: 61% (2600/4261) Updating files: 62% (2642/4261) Updating files: 63% (2685/4261) Updating files: 64% (2728/4261) Updating files: 64% (2736/4261) Updating files: 64% (2757/4261) Updating files: 65% (2770/4261) Updating files: 66% (2813/4261) Updating files: 67% (2855/4261) Updating files: 68% (2898/4261) Updating files: 69% (2941/4261) Updating files: 70% (2983/4261) Updating files: 70% (2995/4261) Updating files: 71% (3026/4261) Updating files: 72% (3068/4261) Updating files: 73% (3111/4261) Updating files: 74% (3154/4261) Updating files: 75% (3196/4261) Updating files: 76% (3239/4261) Updating files: 76% (3273/4261) Updating files: 77% (3281/4261) Updating files: 78% (3324/4261) Updating files: 79% (3367/4261) Updating files: 80% (3409/4261) Updating files: 81% (3452/4261) Updating files: 82% (3495/4261) Updating files: 83% (3537/4261) Updating files: 84% (3580/4261) Updating files: 85% (3622/4261) Updating files: 86% (3665/4261) Updating files: 87% (3708/4261) Updating files: 88% (3750/4261) Updating files: 89% (3793/4261) Updating files: 90% (3835/4261) Updating files: 91% (3878/4261) Updating files: 91% (3916/4261) Updating files: 92% (3921/4261) Updating files: 93% (3963/4261) Updating files: 94% (4006/4261) Updating files: 95% (4048/4261) Updating files: 96% (4091/4261) Updating files: 97% (4134/4261) Updating files: 98% (4176/4261) Updating files: 99% (4219/4261) Updating files: 100% (4261/4261) Updating files: 100% (4261/4261), done. 2024-03-08T23:09:39.325 INFO:tasks.workunit.client.0.smithi046.stderr:+ cd xfstests-dev 2024-03-08T23:09:39.326 INFO:tasks.workunit.client.0.smithi046.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-08T23:09:39.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:39 smithi046 ceph-mon[31859]: pgmap v59: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 764 KiB/s wr, 42 op/s 2024-03-08T23:09:40.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:39 smithi192 ceph-mon[36700]: pgmap v59: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 764 KiB/s wr, 42 op/s 2024-03-08T23:09:40.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:39 smithi062 ceph-mon[35761]: pgmap v59: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 764 KiB/s wr, 42 op/s 2024-03-08T23:09:41.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:41 smithi046 ceph-mon[31859]: pgmap v60: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 558 KiB/s wr, 33 op/s 2024-03-08T23:09:42.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:41 smithi192 ceph-mon[36700]: pgmap v60: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 558 KiB/s wr, 33 op/s 2024-03-08T23:09:42.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:41 smithi062 ceph-mon[35761]: pgmap v60: 129 pgs: 129 active+clean; 23 MiB data, 397 MiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 558 KiB/s wr, 33 op/s 2024-03-08T23:09:43.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:43 smithi046 ceph-mon[31859]: pgmap v61: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 375 op/s 2024-03-08T23:09:44.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:43 smithi192 ceph-mon[36700]: pgmap v61: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 375 op/s 2024-03-08T23:09:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:43 smithi062 ceph-mon[35761]: pgmap v61: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.2 MiB/s wr, 375 op/s 2024-03-08T23:09:45.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:45 smithi046 ceph-mon[31859]: pgmap v62: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:46.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:45 smithi192 ceph-mon[36700]: pgmap v62: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:46.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:45 smithi062 ceph-mon[35761]: pgmap v62: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:46.690 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:09:46.690 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:09:46.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.940+0000 7f1979c00640 1 -- 172.21.15.46:0/1725765496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19741086b0 msgr2=0x7f1974110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.940+0000 7f1979c00640 1 --2- 172.21.15.46:0/1725765496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19741086b0 0x7f1974110280 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f196c009be0 tx=0x7f196c02bef0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 -- 172.21.15.46:0/1725765496 shutdown_connections 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 --2- 172.21.15.46:0/1725765496 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f19741086b0 0x7f1974110280 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 --2- 172.21.15.46:0/1725765496 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1974103f90 0x7f1974107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 --2- 172.21.15.46:0/1725765496 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741035e0 0x7f19741039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 -- 172.21.15.46:0/1725765496 >> 172.21.15.46:0/1725765496 conn(0x7f19740fee00 msgr2=0x7f1974101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 -- 172.21.15.46:0/1725765496 shutdown_connections 2024-03-08T23:09:47.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.941+0000 7f1979c00640 1 -- 172.21.15.46:0/1725765496 wait complete. 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 Processor -- start 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 -- start start 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741086b0 0x7f197419fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1974199a80 0x7f1974199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741137a0 con 0x7f19741086b0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974113620 con 0x7f19741035e0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.942+0000 7f1979c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974113920 con 0x7f1974199a80 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43216/0 (socket says 172.21.15.46:43216) 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 -- 172.21.15.46:0/2420086687 learned_addr learned my addr 172.21.15.46:0/2420086687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973400640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741086b0 0x7f197419fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1979200640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1974199a80 0x7f1974199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 -- 172.21.15.46:0/2420086687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1974199a80 msgr2=0x7f1974199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1974199a80 0x7f1974199f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 -- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741086b0 msgr2=0x7f197419fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741086b0 0x7f197419fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 -- 172.21.15.46:0/2420086687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197419a5a0 con 0x7f19741035e0 2024-03-08T23:09:47.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.943+0000 7f1973e00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f196400e9f0 tx=0x7f196400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.944+0000 7f1970c00640 1 -- 172.21.15.46:0/2420086687 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196400cd80 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.944+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f197419a890 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.944+0000 7f1970c00640 1 -- 172.21.15.46:0/2420086687 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1964010910 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.944+0000 7f1970c00640 1 -- 172.21.15.46:0/2420086687 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19640188f0 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.944+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19741a6ae0 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.947+0000 7f197de65740 1 Processor -- start 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.947+0000 7f197de65740 1 -- start start 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.947+0000 7f1970c00640 1 -- 172.21.15.46:0/2420086687 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1964010ac0 con 0x7f19741035e0 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.947+0000 7f197de65740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.947+0000 7f197de65740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f197de65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e7575f8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f197de65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7575ae410 con 0x55e7575f5c90 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f197de65740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e757550d50 con 0x55e7575ec790 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f197de65740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7574f9b80 con 0x55e7575ed0b0 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43224/0 (socket says 172.21.15.46:43224) 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 -- 172.21.15.46:0/92941591 learned_addr learned my addr 172.21.15.46:0/92941591 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1973e00640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1973400640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e7575f8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 -- 172.21.15.46:0/92941591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 msgr2=0x55e7575f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 -- 172.21.15.46:0/92941591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 msgr2=0x55e7575f8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e7575f8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1979200640 1 -- 172.21.15.46:0/92941591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e75732f7d0 con 0x55e7575ec790 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1973400640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e7575f8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.948+0000 7f1973e00640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1979200640 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f196c009be0 tx=0x7f196c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eece164cfcffa059 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1969600640 1 -- 172.21.15.46:0/92941591 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c043030 con 0x55e7575ec790 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1969600640 1 -- 172.21.15.46:0/92941591 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f196c033c00 con 0x55e7575ec790 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1969600640 1 -- 172.21.15.46:0/92941591 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196c02c890 con 0x55e7575ec790 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1970c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 0x7f19580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.949+0000 7f1970c00640 1 -- 172.21.15.46:0/2420086687 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f19640cf5e0 con 0x7f19741035e0 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f1973e00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 0x7f19580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 -- 172.21.15.46:0/92941591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 msgr2=0x55e7575ecb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f196c009be0 tx=0x7f196c009840 comp rx=0 tx=0).stop 2024-03-08T23:09:47.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 -- 172.21.15.46:0/92941591 shutdown_connections 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e7575f8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 --2- 172.21.15.46:0/92941591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ec790 0x55e7575ecb70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 -- 172.21.15.46:0/92941591 >> 172.21.15.46:0/92941591 conn(0x55e7575e6ec0 msgr2=0x55e7575e72a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f1973e00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 0x7f19580a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e7575ec260 tx=0x7f19640d5110 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 -- 172.21.15.46:0/92941591 shutdown_connections 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.950+0000 7f197de65740 1 -- 172.21.15.46:0/92941591 wait complete. 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 -- start start 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 Processor -- start 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575f3790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e75750cee0 con 0x55e7575ec790 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e75754cdf0 con 0x55e7575f5c90 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.951+0000 7f197de65740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7575f8a20 con 0x55e7575ed0b0 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f197de65740 10 client.0 _ll_register_callbacks cb 0x55e7575138d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55426/0 (socket says 172.21.15.46:55426) 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 -- 172.21.15.46:0/2699002076 learned_addr learned my addr 172.21.15.46:0/2699002076 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1979200640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1979200640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 -- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 msgr2=0x55e7575f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 -- 172.21.15.46:0/2699002076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 msgr2=0x55e7575f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 -- 172.21.15.46:0/2699002076 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7572653f0 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1979200640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f1973400640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f19600116c0 tx=0x7f1960011b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e51d8cf459c523d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f195e000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.952+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196000fbb0 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f196c033c00 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f196000fd50 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1960027050 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 10 client.26005 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e7572250f0 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f196c033c00 con 0x55e7575ed0b0 2024-03-08T23:09:47.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f1960022030 con 0x55e7575ed0b0 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 10 client.26005 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 20 client.26005 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e757605f20 0x55e757608310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 4 client.26005 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.953+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e75755e290 con 0x55e757605f20 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.954+0000 7f1979200640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e757605f20 0x55e757608310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.954+0000 7f1979200640 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e757605f20 0x55e757608310 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=103c902b2f47ffbc server_cookie=9393fd6be377e824 in_seq=0 out_seq=0 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.954+0000 7f195e000640 10 client.26005 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f195e000640 1 -- 172.21.15.46:0/2699002076 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e75755e290 con 0x55e757605f20 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f195e000640 10 client.26005 handle_command_reply: tid=0 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 client.26005 shutdown 2024-03-08T23:09:47.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 msgr2=0x55e7575f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f19600116c0 tx=0x7f1960011b80 comp rx=0 tx=0).stop 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 shutdown_connections 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e757605f20 0x55e757608310 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575f3250 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 --2- 172.21.15.46:0/2699002076 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 >> 172.21.15.46:0/2699002076 conn(0x55e7575e6ec0 msgr2=0x55e7575e8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.955+0000 7f195e000640 0 client.26005 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.956+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 shutdown_connections 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.956+0000 7f197de65740 1 -- 172.21.15.46:0/2699002076 wait complete. 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.956+0000 7f197de65740 20 client.26005 trim_cache size 0 max 16384 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 Processor -- start 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 -- start start 2024-03-08T23:09:47.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575fcc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575fd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741b2a10 con 0x55e7575ec790 2024-03-08T23:09:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741b05b0 con 0x55e7575f5c90 2024-03-08T23:09:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.976+0000 7f197de65740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741ae650 con 0x55e7575ed0b0 2024-03-08T23:09:47.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1979200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1979200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33068/0 (socket says 172.21.15.46:33068) 2024-03-08T23:09:47.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575fcc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1979200640 1 -- 172.21.15.46:0/4191472127 learned_addr learned my addr 172.21.15.46:0/4191472127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1979200640 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575fd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 -- 172.21.15.46:0/4191472127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 msgr2=0x55e7575fd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575fd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 -- 172.21.15.46:0/4191472127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 msgr2=0x55e7575f5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 -- 172.21.15.46:0/4191472127 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7575e2570 con 0x55e7575ed0b0 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1979200640 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.977+0000 7f1973400640 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575fcc60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1960006010 tx=0x7f1960009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9a49a7864a4228a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.978+0000 7f1968c00640 1 -- 172.21.15.46:0/4191472127 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960009d50 con 0x55e7575ed0b0 2024-03-08T23:09:47.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.978+0000 7f1968c00640 1 -- 172.21.15.46:0/4191472127 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19600198d0 con 0x55e7575ed0b0 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.978+0000 7f1968c00640 1 -- 172.21.15.46:0/4191472127 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960005040 con 0x55e7575ed0b0 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 -- 172.21.15.46:0/4191472127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 msgr2=0x55e7575fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575fcc60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1960006010 tx=0x7f1960009c70 comp rx=0 tx=0).stop 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 -- 172.21.15.46:0/4191472127 shutdown_connections 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575f5c90 0x55e7575fd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ed0b0 0x55e7575fcc60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 --2- 172.21.15.46:0/4191472127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575ec790 0x55e7575f5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 -- 172.21.15.46:0/4191472127 >> 172.21.15.46:0/4191472127 conn(0x55e7575e6ec0 msgr2=0x55e7575f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 -- 172.21.15.46:0/4191472127 shutdown_connections 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.979+0000 7f197de65740 1 -- 172.21.15.46:0/4191472127 wait complete. 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.980+0000 7f197de65740 1 -- start start 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.980+0000 7f197de65740 1 Processor -- start 2024-03-08T23:09:47.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.980+0000 7f197de65740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ec790 0x55e757655cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.980+0000 7f197de65740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f197de65740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e757650380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f197de65740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741aec50 con 0x55e7575f5c90 2024-03-08T23:09:47.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f197de65740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741aff50 con 0x55e7575ed0b0 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f197de65740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19741af290 con 0x55e7575ec790 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f197de65740 10 client.0 _ll_register_callbacks cb 0x7f1960005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1979200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e757650380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43258/0 (socket says 172.21.15.46:43258) 2024-03-08T23:09:47.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 -- 172.21.15.46:0/1988725258 learned_addr learned my addr 172.21.15.46:0/1988725258 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1979200640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ec790 0x55e757655cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 -- 172.21.15.46:0/1988725258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ec790 msgr2=0x55e757655cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ec790 0x55e757655cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 -- 172.21.15.46:0/1988725258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 msgr2=0x55e757650380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e757650380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1973400640 1 -- 172.21.15.46:0/1988725258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e757650ac0 con 0x55e7575ed0b0 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.981+0000 7f1979200640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e757650380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f1973400640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19600093f0 tx=0x7f1960019ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcfb5acb8021d47c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f195e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1960009d50 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f1960019390 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19600198d0 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1960040050 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f197de65740 10 client.25997 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e75765d4e0 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.982+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19600198d0 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f196003b030 con 0x55e7575ed0b0 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 10 client.25997 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:47.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 20 client.25997 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e75765dbc0 0x55e75765ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 4 client.25997 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e7575585e0 con 0x55e75765dbc0 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.983+0000 7f1979200640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e75765dbc0 0x55e75765ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f1979200640 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e75765dbc0 0x55e75765ffe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b4900423e66037 server_cookie=b0f0006ff59c330f in_seq=0 out_seq=0 2024-03-08T23:09:47.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f195e000640 10 client.25997 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f195e000640 1 -- 172.21.15.46:0/1988725258 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e7575585e0 con 0x55e75765dbc0 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f195e000640 10 client.25997 handle_command_reply: tid=0 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 client.25997 shutdown 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 msgr2=0x55e7576561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19600093f0 tx=0x7f1960019ac0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 shutdown_connections 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e75765dbc0 0x55e75765ffe0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e7575f5c90 0x55e757650380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e7575ed0b0 0x55e7576561f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f195e000640 0 client.25997 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 --2- 172.21.15.46:0/1988725258 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e7575ec790 0x55e757655cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 >> 172.21.15.46:0/1988725258 conn(0x55e7575e6ec0 msgr2=0x55e7575e8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 shutdown_connections 2024-03-08T23:09:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 1 -- 172.21.15.46:0/1988725258 wait complete. 2024-03-08T23:09:47.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.984+0000 7f197de65740 20 client.25997 trim_cache size 0 max 16384 2024-03-08T23:09:47.076 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:47.076 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:09:47.076 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:09:47.076 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:47.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 msgr2=0x7f19580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 0x7f19580a4e30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x55e7575ec260 tx=0x7f19640d5110 comp rx=0 tx=0).stop 2024-03-08T23:09:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 msgr2=0x7f197419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f196400e9f0 tx=0x7f196400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 shutdown_connections 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f19580a2970 0x7f19580a4e30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1974199a80 0x7f1974199f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f19741086b0 0x7f197419fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 --2- 172.21.15.46:0/2420086687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f19741035e0 0x7f197419f8d0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.985+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 >> 172.21.15.46:0/2420086687 conn(0x7f19740fee00 msgr2=0x7f19740ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.986+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 shutdown_connections 2024-03-08T23:09:47.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:46.986+0000 7f1979c00640 1 -- 172.21.15.46:0/2420086687 wait complete. 2024-03-08T23:09:47.082 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:09:47.083 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:09:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 -- 172.21.15.46:0/3131984084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008109590 msgr2=0x7fd008109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 --2- 172.21.15.46:0/3131984084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008109590 0x7fd008109970 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fcff8009930 tx=0x7fcff802bc30 comp rx=0 tx=0).stop 2024-03-08T23:09:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 -- 172.21.15.46:0/3131984084 shutdown_connections 2024-03-08T23:09:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 --2- 172.21.15.46:0/3131984084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008103e90 0x7fd008110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 --2- 172.21.15.46:0/3131984084 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 --2- 172.21.15.46:0/3131984084 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008109590 0x7fd008109970 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 -- 172.21.15.46:0/3131984084 >> 172.21.15.46:0/3131984084 conn(0x7fd0080fefa0 msgr2=0x7fd0081013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 -- 172.21.15.46:0/3131984084 shutdown_connections 2024-03-08T23:09:47.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.292+0000 7fd00ec00640 1 -- 172.21.15.46:0/3131984084 wait complete. 2024-03-08T23:09:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.293+0000 7fd00ec00640 1 Processor -- start 2024-03-08T23:09:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.293+0000 7fd00ec00640 1 -- start start 2024-03-08T23:09:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 0x7fd00819a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008109590 0x7fd00819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0081138a0 con 0x7fd0081034d0 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008113720 con 0x7fd008103e90 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00ec00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd008113a20 con 0x7fd008109590 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd007e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fcfffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 0x7fd00819a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd007e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008199d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33086/0 (socket says 172.21.15.46:33086) 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd007e00640 1 -- 172.21.15.46:0/1216455386 learned_addr learned my addr 172.21.15.46:0/1216455386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fd00cc00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008109590 0x7fd00819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fcfffe00640 1 -- 172.21.15.46:0/1216455386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008109590 msgr2=0x7fd00819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fcfffe00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008109590 0x7fd00819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.294+0000 7fcfffe00640 1 -- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 msgr2=0x7fd008199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fcfffe00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fcfffe00640 1 -- 172.21.15.46:0/1216455386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00819ed20 con 0x7fd008103e90 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fcfffe00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 0x7fd00819a2c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e9f0 tx=0x7fcff000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff000cd80 con 0x7fd008103e90 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcff00035f0 con 0x7fd008103e90 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00819f010 con 0x7fd008103e90 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff0010670 con 0x7fd008103e90 2024-03-08T23:09:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.295+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0081a68a0 con 0x7fd008103e90 2024-03-08T23:09:47.301 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.296+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0080680e0 con 0x7fd008103e90 2024-03-08T23:09:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.299+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcff00037a0 con 0x7fd008103e90 2024-03-08T23:09:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.301+0000 7fd005600640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 0x7fcfec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.302+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcff00cf400 con 0x7fd008103e90 2024-03-08T23:09:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.302+0000 7fd007e00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 0x7fcfec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.302+0000 7fd007e00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 0x7fcfec0a4eb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcff80096c0 tx=0x7fcff802df90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.305+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcff009be00 con 0x7fd008103e90 2024-03-08T23:09:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.568+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd00802ce70 con 0x7fd008103e90 2024-03-08T23:09:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.569+0000 7fd005600640 1 -- 172.21.15.46:0/1216455386 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11337 (secure 0 0 0) 0x7fcff009b700 con 0x7fd008103e90 2024-03-08T23:09:47.572 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:09:47.572 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:09:21.814153+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:09:47.572 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 25 2024-03-08T23:09:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 msgr2=0x7fcfec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 0x7fcfec0a4eb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcff80096c0 tx=0x7fcff802df90 comp rx=0 tx=0).stop 2024-03-08T23:09:47.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 msgr2=0x7fd00819a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 0x7fd00819a2c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fcff000e9f0 tx=0x7fcff000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 shutdown_connections 2024-03-08T23:09:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcfec0a29f0 0x7fcfec0a4eb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd008109590 0x7fd00819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd008103e90 0x7fd00819a2c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 --2- 172.21.15.46:0/1216455386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd0081034d0 0x7fd008199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 >> 172.21.15.46:0/1216455386 conn(0x7fd0080fefa0 msgr2=0x7fd008100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 shutdown_connections 2024-03-08T23:09:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.572+0000 7fd00ec00640 1 -- 172.21.15.46:0/1216455386 wait complete. 2024-03-08T23:09:47.593 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:47.593 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:09:47.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 -- 172.21.15.46:0/4183049854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 msgr2=0x7f4eb410b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 --2- 172.21.15.46:0/4183049854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb410b120 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0009900 tx=0x7f4ea002bcc0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 -- 172.21.15.46:0/4183049854 shutdown_connections 2024-03-08T23:09:47.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 --2- 172.21.15.46:0/4183049854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 0x7f4eb410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 --2- 172.21.15.46:0/4183049854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb410b120 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 --2- 172.21.15.46:0/4183049854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb4102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 -- 172.21.15.46:0/4183049854 >> 172.21.15.46:0/4183049854 conn(0x7f4eb40fbb40 msgr2=0x7f4eb40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 -- 172.21.15.46:0/4183049854 shutdown_connections 2024-03-08T23:09:47.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.792+0000 7f4eb9000640 1 -- 172.21.15.46:0/4183049854 wait complete. 2024-03-08T23:09:47.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.793+0000 7f4eb9000640 1 Processor -- start 2024-03-08T23:09:47.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.793+0000 7f4eb9000640 1 -- start start 2024-03-08T23:09:47.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 0x7f4eb4199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb4111570 con 0x7f4eb4102c40 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41113f0 con 0x7f4eb4102320 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb9000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41116f0 con 0x7f4eb410b740 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33094/0 (socket says 172.21.15.46:33094) 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb2a00640 1 -- 172.21.15.46:0/2959013768 learned_addr learned my addr 172.21.15.46:0/2959013768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb3400640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.794+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 0x7f4eb4199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 -- 172.21.15.46:0/2959013768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 msgr2=0x7f4eb4199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 0x7f4eb4199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 -- 172.21.15.46:0/2959013768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 msgr2=0x7f4eb419f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb419f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 -- 172.21.15.46:0/2959013768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb419a200 con 0x7f4eb4102c40 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb3400640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0000c00 tx=0x7f4ea00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eabe00640 1 -- 172.21.15.46:0/2959013768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0042020 con 0x7f4eb4102c40 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eabe00640 1 -- 172.21.15.46:0/2959013768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea0005000 con 0x7f4eb4102c40 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eabe00640 1 -- 172.21.15.46:0/2959013768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00039f0 con 0x7f4eb4102c40 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.795+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb419a490 con 0x7f4eb4102c40 2024-03-08T23:09:47.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.796+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb41a6790 con 0x7f4eb4102c40 2024-03-08T23:09:47.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 Processor -- start 2024-03-08T23:09:47.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 -- start start 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c74192a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192ce20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c741885dd0 con 0x55c741921530 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74182ec70 con 0x55c741921e50 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4ebd228740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c741881e70 con 0x55c74192aa30 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.800+0000 7f4eabe00640 1 -- 172.21.15.46:0/2959013768 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4ea0045020 con 0x7f4eb4102c40 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33096/0 (socket says 172.21.15.46:33096) 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 -- 172.21.15.46:0/1151963023 learned_addr learned my addr 172.21.15.46:0/1151963023 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c74192a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 -- 172.21.15.46:0/1151963023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 msgr2=0x55c74192ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 -- 172.21.15.46:0/1151963023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 msgr2=0x55c74192a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c74192a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb3e00640 1 -- 172.21.15.46:0/1151963023 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c7418e82b0 con 0x55c741921530 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.801+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:47.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009f50 tx=0x7f4eac0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306cabae1b445f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4ea8c00640 1 -- 172.21.15.46:0/1151963023 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac004d20 con 0x55c741921530 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4ea8c00640 1 -- 172.21.15.46:0/1151963023 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4eac004050 con 0x55c741921530 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4ea8c00640 1 -- 172.21.15.46:0/1151963023 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac005280 con 0x55c741921530 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4eabe00640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 0x7f4e980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.802+0000 7f4eabe00640 1 -- 172.21.15.46:0/2959013768 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4ea00f0bf0 con 0x7f4eb4102c40 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4eb3400640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 0x7f4e980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 -- 172.21.15.46:0/1151963023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 msgr2=0x55c741921910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009f50 tx=0x7f4eac0118d0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 -- 172.21.15.46:0/1151963023 shutdown_connections 2024-03-08T23:09:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c74192a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1151963023 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741921910 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 -- 172.21.15.46:0/1151963023 >> 172.21.15.46:0/1151963023 conn(0x55c74191bc60 msgr2=0x55c74191c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 -- 172.21.15.46:0/1151963023 shutdown_connections 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4ebd228740 1 -- 172.21.15.46:0/1151963023 wait complete. 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.803+0000 7f4eb3400640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 0x7f4e980a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c741921000 tx=0x7f4ea4005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.804+0000 7f4ebd228740 1 -- start start 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.804+0000 7f4ebd228740 1 Processor -- start 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741927710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c74192aa30 0x55c741927c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74192d820 con 0x55c74192aa30 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c7418420c0 con 0x55c741921e50 2024-03-08T23:09:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c74192d9a0 con 0x55c741921530 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4ebd228740 10 client.0 _ll_register_callbacks cb 0x55c74187cec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55490/0 (socket says 172.21.15.46:55490) 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 -- 172.21.15.46:0/1849933083 learned_addr learned my addr 172.21.15.46:0/1849933083 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c74192aa30 0x55c741927c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741927710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 -- 172.21.15.46:0/1849933083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 msgr2=0x55c741927710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741927710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 -- 172.21.15.46:0/1849933083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c74192aa30 msgr2=0x55c741927c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c74192aa30 0x55c741927c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.805+0000 7f4eb3400640 1 -- 172.21.15.46:0/1849933083 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c74159a3f0 con 0x55c741921530 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4003640 tx=0x7f4ea4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46644275a6fea544 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4e9d600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4005020 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4eac004050 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4ea40051c0 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.806+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ea40053c0 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 10 client.26029 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c7416647d0 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4eac004050 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4ea4027070 con 0x55c741921530 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 10 client.26029 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 10 client.26029 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 20 client.26029 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c74193aa80 0x55c74193ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 4 client.26029 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:47.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c74185a4f0 con 0x55c74193aa80 2024-03-08T23:09:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.807+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c74193aa80 0x55c74193ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.808+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c74193aa80 0x55c74193ce70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5ad416dec9041d3a server_cookie=91e0c3bfaf25f508 in_seq=0 out_seq=0 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.808+0000 7f4e9d600640 10 client.26029 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.808+0000 7f4e9d600640 1 -- 172.21.15.46:0/1849933083 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c74185a4f0 con 0x55c74193aa80 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.808+0000 7f4e9d600640 10 client.26029 handle_command_reply: tid=0 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 client.26029 shutdown 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 msgr2=0x55c74192a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4003640 tx=0x7f4ea4004730 comp rx=0 tx=0).stop 2024-03-08T23:09:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 shutdown_connections 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c74193aa80 0x55c74193ce70 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c74192aa30 0x55c741927c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741927710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4e9d600640 0 client.26029 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1849933083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c741921530 0x55c74192a380 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 >> 172.21.15.46:0/1849933083 conn(0x55c74191bc60 msgr2=0x55c74191d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 shutdown_connections 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 1 -- 172.21.15.46:0/1849933083 wait complete. 2024-03-08T23:09:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.809+0000 7f4ebd228740 20 client.26029 trim_cache size 0 max 16384 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 Processor -- start 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 -- start start 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921e50 0x55c741929e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41b2a60 con 0x55c741921e50 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41b0600 con 0x55c741921530 2024-03-08T23:09:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.828+0000 7f4ebd228740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41ae6a0 con 0x55c74192aa30 2024-03-08T23:09:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921e50 0x55c741929e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43320/0 (socket says 172.21.15.46:43320) 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 -- 172.21.15.46:0/3572152067 learned_addr learned my addr 172.21.15.46:0/3572152067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 -- 172.21.15.46:0/3572152067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 msgr2=0x55c74192a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 -- 172.21.15.46:0/3572152067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921e50 msgr2=0x55c741929e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921e50 0x55c741929e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 -- 172.21.15.46:0/3572152067 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c741927fe0 con 0x55c741921530 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.829+0000 7f4eb3400640 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000c00 tx=0x7f4ea4012450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2463e97fa5af2e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.830+0000 7f4e9fe00640 1 -- 172.21.15.46:0/3572152067 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4005020 con 0x55c741921530 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.830+0000 7f4e9fe00640 1 -- 172.21.15.46:0/3572152067 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea40051c0 con 0x55c741921530 2024-03-08T23:09:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.830+0000 7f4e9fe00640 1 -- 172.21.15.46:0/3572152067 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea401f690 con 0x55c741921530 2024-03-08T23:09:47.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.830+0000 7f4ebd228740 1 -- 172.21.15.46:0/3572152067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 msgr2=0x55c7419417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.830+0000 7f4ebd228740 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4000c00 tx=0x7f4ea4012450 comp rx=0 tx=0).stop 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 -- 172.21.15.46:0/3572152067 shutdown_connections 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c74192a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921e50 0x55c741929e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 --2- 172.21.15.46:0/3572152067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921530 0x55c7419417d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 -- 172.21.15.46:0/3572152067 >> 172.21.15.46:0/3572152067 conn(0x55c74191bc60 msgr2=0x55c74191c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 -- 172.21.15.46:0/3572152067 shutdown_connections 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 -- 172.21.15.46:0/3572152067 wait complete. 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.831+0000 7f4ebd228740 1 -- start start 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 Processor -- start 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741930370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c7419308b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41aeca0 con 0x55c741921530 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41affa0 con 0x55c741921e50 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb41af2e0 con 0x55c74192aa30 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.832+0000 7f4ebd228740 10 client.0 _ll_register_callbacks cb 0x7f4eac009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33130/0 (socket says 172.21.15.46:33130) 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 -- 172.21.15.46:0/1777008824 learned_addr learned my addr 172.21.15.46:0/1777008824 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3e00640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c7419308b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741930370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 -- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 msgr2=0x55c7419308b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c7419308b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 -- 172.21.15.46:0/1777008824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 msgr2=0x55c741930370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741930370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.833+0000 7f4eb3400640 1 -- 172.21.15.46:0/1777008824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c7419310e0 con 0x55c741921530 2024-03-08T23:09:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4eb3400640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4004cc0 tx=0x7f4ea400fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45018413b908bdd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4e9d600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4023040 con 0x55c741921530 2024-03-08T23:09:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4ea401f630 con 0x55c741921530 2024-03-08T23:09:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ea40051c0 con 0x55c741921530 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ea4005020 con 0x55c741921530 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4ebd228740 10 client.15900 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.834+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c741991fc0 con 0x55c741921530 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ea40051c0 con 0x55c741921530 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4ea4027070 con 0x55c741921530 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 10 client.15900 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 10 client.15900 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 20 client.15900 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c7419924d0 0x55c7419948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 4 client.15900 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c741893310 con 0x55c7419924d0 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.835+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c7419924d0 0x55c7419948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4eb2a00640 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c7419924d0 0x55c7419948c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=80fb5b6b68727dc6 server_cookie=e9abdb6bfcf9730b in_seq=0 out_seq=0 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4e9d600640 10 client.15900 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4e9d600640 1 -- 172.21.15.46:0/1777008824 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c741893310 con 0x55c7419924d0 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4e9d600640 10 client.15900 handle_command_reply: tid=0 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 client.15900 shutdown 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 msgr2=0x55c741931fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4004cc0 tx=0x7f4ea400fe40 comp rx=0 tx=0).stop 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 shutdown_connections 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c7419924d0 0x55c7419948c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c74192aa30 0x55c7419308b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4e9d600640 0 client.15900 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c741921e50 0x55c741930370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 --2- 172.21.15.46:0/1777008824 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c741921530 0x55c741931fd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.836+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 >> 172.21.15.46:0/1777008824 conn(0x55c74191bc60 msgr2=0x55c7419417d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 shutdown_connections 2024-03-08T23:09:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4ebd228740 1 -- 172.21.15.46:0/1777008824 wait complete. 2024-03-08T23:09:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4ebd228740 20 client.15900 trim_cache size 0 max 16384 2024-03-08T23:09:47.843 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 msgr2=0x7f4e980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 0x7f4e980a4e30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55c741921000 tx=0x7f4ea4005f90 comp rx=0 tx=0).stop 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 msgr2=0x7f4eb419fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.837+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4ea0000c00 tx=0x7f4ea00042f0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 shutdown_connections 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4e980a2970 0x7f4e980a4e30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4eb410b740 0x7f4eb4199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4eb4102c40 0x7f4eb419fe10 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 --2- 172.21.15.46:0/2959013768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4eb4102320 0x7f4eb419f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:47.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 >> 172.21.15.46:0/2959013768 conn(0x7f4eb40fbb40 msgr2=0x7f4eb40fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:47.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 shutdown_connections 2024-03-08T23:09:47.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:47.838+0000 7f4eb9000640 1 -- 172.21.15.46:0/2959013768 wait complete. 2024-03-08T23:09:47.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:47 smithi046 ceph-mon[31859]: pgmap v63: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:47.854 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:47 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1216455386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:47.857 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:47.857 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:09:48.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 -- 172.21.15.46:0/3487007932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 msgr2=0x7fc9fc103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 --2- 172.21.15.46:0/3487007932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc103930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8009900 tx=0x7fc9e802bc30 comp rx=0 tx=0).stop 2024-03-08T23:09:48.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 -- 172.21.15.46:0/3487007932 shutdown_connections 2024-03-08T23:09:48.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 --2- 172.21.15.46:0/3487007932 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 0x7fc9fc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 --2- 172.21.15.46:0/3487007932 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc103930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 --2- 172.21.15.46:0/3487007932 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 0x7fc9fc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 -- 172.21.15.46:0/3487007932 >> 172.21.15.46:0/3487007932 conn(0x7fc9fc0fef60 msgr2=0x7fc9fc101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 -- 172.21.15.46:0/3487007932 shutdown_connections 2024-03-08T23:09:48.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.059+0000 7fca00e00640 1 -- 172.21.15.46:0/3487007932 wait complete. 2024-03-08T23:09:48.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:47 smithi192 ceph-mon[36700]: pgmap v63: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:48.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:47 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1216455386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.060+0000 7fca00e00640 1 Processor -- start 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 -- start start 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 0x7fc9fc19fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 0x7fc9fc199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1138a0 con 0x7fc9fc1034b0 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc113720 con 0x7fc9fc109570 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fca00e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc113a20 con 0x7fc9fc103e70 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.061+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33152/0 (socket says 172.21.15.46:33152) 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 -- 172.21.15.46:0/905947292 learned_addr learned my addr 172.21.15.46:0/905947292 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9400640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 0x7fc9fc19fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9fa800640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 0x7fc9fc199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 -- 172.21.15.46:0/905947292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 msgr2=0x7fc9fc19fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 0x7fc9fc19fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 -- 172.21.15.46:0/905947292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 msgr2=0x7fc9fc199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 0x7fc9fc199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 -- 172.21.15.46:0/905947292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc19a260 con 0x7fc9fc1034b0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.062+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00e9f0 tx=0x7fc9ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.063+0000 7fc9f2a00640 1 -- 172.21.15.46:0/905947292 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec00cd80 con 0x7fc9fc1034b0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.063+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc19a550 con 0x7fc9fc1034b0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.063+0000 7fc9f2a00640 1 -- 172.21.15.46:0/905947292 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9ec010910 con 0x7fc9fc1034b0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.063+0000 7fc9f2a00640 1 -- 172.21.15.46:0/905947292 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec0188f0 con 0x7fc9fc1034b0 2024-03-08T23:09:48.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.063+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc1a67f0 con 0x7fc9fc1034b0 2024-03-08T23:09:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.066+0000 7fca03bc9740 1 Processor -- start 2024-03-08T23:09:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.066+0000 7fca03bc9740 1 -- start start 2024-03-08T23:09:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081efb310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081efdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081e56dd0 con 0x557081ef2350 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081dffc70 con 0x557081efb850 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fca03bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081e52e70 con 0x557081ef2c70 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fc9fa800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fc9fa800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33154/0 (socket says 172.21.15.46:33154) 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fc9fa800640 1 -- 172.21.15.46:0/1186000687 learned_addr learned my addr 172.21.15.46:0/1186000687 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.067+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081efdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 -- 172.21.15.46:0/1186000687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 msgr2=0x557081efb310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081efb310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 -- 172.21.15.46:0/1186000687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 msgr2=0x557081ef2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 -- 172.21.15.46:0/1186000687 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557081eb92b0 con 0x557081efb850 2024-03-08T23:09:48.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9fa800640 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081efdc40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc9e802c170 tx=0x7fc9e8005df0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a749093abad85331 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9f2a00640 1 -- 172.21.15.46:0/905947292 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc9ec010ac0 con 0x7fc9fc1034b0 2024-03-08T23:09:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9e7400640 1 -- 172.21.15.46:0/1186000687 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8042070 con 0x557081efb850 2024-03-08T23:09:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9e7400640 1 -- 172.21.15.46:0/1186000687 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e802c690 con 0x557081efb850 2024-03-08T23:09:48.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.068+0000 7fc9e7400640 1 -- 172.21.15.46:0/1186000687 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8004c00 con 0x557081efb850 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.069+0000 7fca03bc9740 1 -- 172.21.15.46:0/1186000687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 msgr2=0x557081efdc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.069+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081efdc40 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc9e802c170 tx=0x7fc9e8005df0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fc9f2a00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 0x7fc9e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 -- 172.21.15.46:0/1186000687 shutdown_connections 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081efdc40 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081efb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1186000687 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081ef2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 -- 172.21.15.46:0/1186000687 >> 172.21.15.46:0/1186000687 conn(0x557081eeca80 msgr2=0x557081eece60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fc9f2a00640 1 -- 172.21.15.46:0/905947292 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc9ec0cf610 con 0x7fc9fc1034b0 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fc9fa800640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 0x7fc9e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 -- 172.21.15.46:0/1186000687 shutdown_connections 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.070+0000 7fca03bc9740 1 -- 172.21.15.46:0/1186000687 wait complete. 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.071+0000 7fca03bc9740 1 -- start start 2024-03-08T23:09:48.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.071+0000 7fc9fa800640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 0x7fc9e00a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005fd0 tx=0x7fc9f4009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.071+0000 7fca03bc9740 1 Processor -- start 2024-03-08T23:09:48.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.071+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.071+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 0x557081ef85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081ef8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fca03bc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081efe5e0 con 0x557081ef2c70 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fca03bc9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081e130c0 con 0x557081efb850 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fca03bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557081efe760 con 0x557081ef2350 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fca03bc9740 10 client.0 _ll_register_callbacks cb 0x557081e4dec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 0x557081ef85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55530/0 (socket says 172.21.15.46:55530) 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 -- 172.21.15.46:0/1482910105 learned_addr learned my addr 172.21.15.46:0/1482910105 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 0x557081ef85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33158/0 (socket says 172.21.15.46:33158) 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9fa800640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081ef8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 -- 172.21.15.46:0/1482910105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 msgr2=0x557081ef8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081ef8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 -- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 msgr2=0x557081ef85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 0x557081ef85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.072+0000 7fc9f9e00640 1 -- 172.21.15.46:0/1482910105 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557081b6b3f0 con 0x557081ef2350 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0ccab0 tx=0x7fc9ec09c150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f2e7eadce451087 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9ec0cf290 con 0x557081ef2350 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc9ec0860e0 con 0x557081ef2350 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9e802c690 con 0x557081ef2350 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9ec0862e0 con 0x557081ef2350 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fca03bc9740 10 client.26044 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:48.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557081c357d0 con 0x557081ef2350 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.073+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e802c690 con 0x557081ef2350 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc9ec097660 con 0x557081ef2350 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 10 client.26044 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 10 client.26044 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 20 client.26044 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f0b7f0 0x557081f0dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 4 client.26044 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557081e2b4f0 con 0x557081f0b7f0 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.074+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f0b7f0 0x557081f0dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.075+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f0b7f0 0x557081f0dbe0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9506612f5795caa3 server_cookie=979f00abb1509cca in_seq=0 out_seq=0 2024-03-08T23:09:48.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.075+0000 7fc9dbe00640 10 client.26044 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:48.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1482910105 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557081e2b4f0 con 0x557081f0b7f0 2024-03-08T23:09:48.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fc9dbe00640 10 client.26044 handle_command_reply: tid=0 2024-03-08T23:09:48.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 client.26044 shutdown 2024-03-08T23:09:48.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 msgr2=0x557081efb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec0ccab0 tx=0x7fc9ec09c150 comp rx=0 tx=0).stop 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 shutdown_connections 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081efb850 0x557081ef8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2c70 0x557081ef85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f0b7f0 0x557081f0dbe0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1482910105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2350 0x557081efb220 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 >> 172.21.15.46:0/1482910105 conn(0x557081eeca80 msgr2=0x557081eee7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fc9dbe00640 0 client.26044 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 shutdown_connections 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 1 -- 172.21.15.46:0/1482910105 wait complete. 2024-03-08T23:09:48.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.077+0000 7fca03bc9740 20 client.26044 trim_cache size 0 max 16384 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.093+0000 7fca03bc9740 1 Processor -- start 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.093+0000 7fca03bc9740 1 -- start start 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2350 0x557081ef8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081efb850 0x557081efb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1b2b30 con 0x557081efb850 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1b06d0 con 0x557081ef2350 2024-03-08T23:09:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fca03bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1ae770 con 0x557081ef2c70 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55542/0 (socket says 172.21.15.46:55542) 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2350 0x557081ef8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 -- 172.21.15.46:0/206847870 learned_addr learned my addr 172.21.15.46:0/206847870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9fa800640 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081efb850 0x557081efb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 -- 172.21.15.46:0/206847870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2350 msgr2=0x557081ef8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2350 0x557081ef8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 -- 172.21.15.46:0/206847870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081efb850 msgr2=0x557081efb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081efb850 0x557081efb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.094+0000 7fc9f9400640 1 -- 172.21.15.46:0/206847870 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557081ef6b80 con 0x557081ef2c70 2024-03-08T23:09:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.095+0000 7fc9f9400640 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x557081eedc10 tx=0x7fc9e802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d794be18ee492bd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.095+0000 7fc9e6a00640 1 -- 172.21.15.46:0/206847870 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8055070 con 0x557081ef2c70 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.095+0000 7fc9e6a00640 1 -- 172.21.15.46:0/206847870 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e8009520 con 0x557081ef2c70 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.095+0000 7fc9e6a00640 1 -- 172.21.15.46:0/206847870 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8042040 con 0x557081ef2c70 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 -- 172.21.15.46:0/206847870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 msgr2=0x557081eface0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x557081eedc10 tx=0x7fc9e802e040 comp rx=0 tx=0).stop 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 -- 172.21.15.46:0/206847870 shutdown_connections 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081efb850 0x557081efb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081ef2c70 0x557081eface0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 --2- 172.21.15.46:0/206847870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2350 0x557081ef8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 -- 172.21.15.46:0/206847870 >> 172.21.15.46:0/206847870 conn(0x557081eeca80 msgr2=0x557081f12480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 -- 172.21.15.46:0/206847870 shutdown_connections 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.096+0000 7fca03bc9740 1 -- 172.21.15.46:0/206847870 wait complete. 2024-03-08T23:09:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.097+0000 7fca03bc9740 1 -- start start 2024-03-08T23:09:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.097+0000 7fca03bc9740 1 Processor -- start 2024-03-08T23:09:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081f01620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 0x557081f01b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081efb850 0x557081f020a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1aed70 con 0x557081ef2350 2024-03-08T23:09:48.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1b0070 con 0x557081ef2c70 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc1af3b0 con 0x557081efb850 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fca03bc9740 10 client.0 _ll_register_callbacks cb 0x7fc9f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081f01620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fc9f9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081f01620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33184/0 (socket says 172.21.15.46:33184) 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fc9f9e00640 1 -- 172.21.15.46:0/1982287224 learned_addr learned my addr 172.21.15.46:0/1982287224 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 0x557081f01b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.098+0000 7fc9fa800640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081efb850 0x557081f020a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 -- 172.21.15.46:0/1982287224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081efb850 msgr2=0x557081f020a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081efb850 0x557081f020a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 -- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 msgr2=0x557081f01620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081f01620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 -- 172.21.15.46:0/1982287224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557081f61e30 con 0x557081ef2c70 2024-03-08T23:09:48.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9f9400640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 0x557081f01b60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x557081efe150 tx=0x7fc9e8005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82022875af56235d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e8055070 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9e8009520 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc9e802cab0 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9e805d050 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fca03bc9740 10 client.26039 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.099+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557081f63110 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e8009520 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fc9e8058030 con 0x557081ef2c70 2024-03-08T23:09:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 10 client.26039 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 10 client.26039 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 20 client.26039 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f632e0 0x557081f656d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 4 client.26039 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557081e64310 con 0x557081f632e0 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.100+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f632e0 0x557081f656d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.101+0000 7fc9f9e00640 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f632e0 0x557081f656d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=14258a9788c5ec9c server_cookie=9a42d154c8716b59 in_seq=0 out_seq=0 2024-03-08T23:09:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.101+0000 7fc9dbe00640 10 client.26039 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fc9dbe00640 1 -- 172.21.15.46:0/1982287224 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557081e64310 con 0x557081f632e0 2024-03-08T23:09:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fc9dbe00640 10 client.26039 handle_command_reply: tid=0 2024-03-08T23:09:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 client.26039 shutdown 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 msgr2=0x557081f01b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 0x557081f01b60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x557081efe150 tx=0x7fc9e8005a20 comp rx=0 tx=0).stop 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 shutdown_connections 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557081f632e0 0x557081f656d0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557081efb850 0x557081f020a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557081ef2c70 0x557081f01b60 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 --2- 172.21.15.46:0/1982287224 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557081ef2350 0x557081f01620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 >> 172.21.15.46:0/1982287224 conn(0x557081eeca80 msgr2=0x557081ef3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fc9dbe00640 0 client.26039 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 shutdown_connections 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 1 -- 172.21.15.46:0/1982287224 wait complete. 2024-03-08T23:09:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.103+0000 7fca03bc9740 20 client.26039 trim_cache size 0 max 16384 2024-03-08T23:09:48.109 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 msgr2=0x7fc9e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 0x7fc9e00a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4005fd0 tx=0x7fc9f4009500 comp rx=0 tx=0).stop 2024-03-08T23:09:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 msgr2=0x7fc9fc19f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc9ec00e9f0 tx=0x7fc9ec00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 shutdown_connections 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9e00a2970 0x7fc9e00a4e30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc9fc109570 0x7fc9fc199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9fc103e70 0x7fc9fc19fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 --2- 172.21.15.46:0/905947292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9fc1034b0 0x7fc9fc19f930 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 >> 172.21.15.46:0/905947292 conn(0x7fc9fc0fef60 msgr2=0x7fc9fc1006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 shutdown_connections 2024-03-08T23:09:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.104+0000 7fca00e00640 1 -- 172.21.15.46:0/905947292 wait complete. 2024-03-08T23:09:48.124 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:09:48.124 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:09:48.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:47 smithi062 ceph-mon[35761]: pgmap v63: 129 pgs: 129 active+clean; 40 MiB data, 498 MiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 1.9 MiB/s wr, 354 op/s 2024-03-08T23:09:48.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:47 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1216455386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:09:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 -- 172.21.15.46:0/370278710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 msgr2=0x7f59380ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 --2- 172.21.15.46:0/370278710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f59380ffac0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5928009900 tx=0x7f592802bc30 comp rx=0 tx=0).stop 2024-03-08T23:09:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 -- 172.21.15.46:0/370278710 shutdown_connections 2024-03-08T23:09:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 --2- 172.21.15.46:0/370278710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 0x7f5938110010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 --2- 172.21.15.46:0/370278710 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 0x7f593810d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 --2- 172.21.15.46:0/370278710 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f59380ffac0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 -- 172.21.15.46:0/370278710 >> 172.21.15.46:0/370278710 conn(0x7f5938074e90 msgr2=0x7f59380752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 -- 172.21.15.46:0/370278710 shutdown_connections 2024-03-08T23:09:48.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.337+0000 7f5937200640 1 -- 172.21.15.46:0/370278710 wait complete. 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.338+0000 7f5937200640 1 Processor -- start 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.338+0000 7f5937200640 1 -- start start 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 0x7f593819fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 0x7f5938199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381137d0 con 0x7f5938100000 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938113650 con 0x7f59380ff6e0 2024-03-08T23:09:48.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5937200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5938113950 con 0x7f593810dc20 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43376/0 (socket says 172.21.15.46:43376) 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 -- 172.21.15.46:0/2562988803 learned_addr learned my addr 172.21.15.46:0/2562988803 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 0x7f593819fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5936800640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 0x7f5938199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 -- 172.21.15.46:0/2562988803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 msgr2=0x7f5938199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 0x7f5938199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 -- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 msgr2=0x7f593819fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 0x7f593819fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.339+0000 7f5935e00640 1 -- 172.21.15.46:0/2562988803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593819a210 con 0x7f59380ff6e0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f5935e00640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5928005940 tx=0x7f592802cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f592ea00640 1 -- 172.21.15.46:0/2562988803 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928041070 con 0x7f59380ff6e0 2024-03-08T23:09:48.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f592ea00640 1 -- 172.21.15.46:0/2562988803 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5928004450 con 0x7f59380ff6e0 2024-03-08T23:09:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f593819a4a0 con 0x7f59380ff6e0 2024-03-08T23:09:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f592ea00640 1 -- 172.21.15.46:0/2562988803 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928003df0 con 0x7f59380ff6e0 2024-03-08T23:09:48.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.340+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59381a67a0 con 0x7f59380ff6e0 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.343+0000 7f592ea00640 1 -- 172.21.15.46:0/2562988803 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5928044070 con 0x7f59380ff6e0 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 Processor -- start 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 -- start start 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d60259f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025aae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d602503dd0 con 0x55d60259f530 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6024acc70 con 0x55d60259fe50 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f593fa26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6024ffe70 con 0x55d6025a8a30 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f5935e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.344+0000 7f5935400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43392/0 (socket says 172.21.15.46:43392) 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935e00640 1 -- 172.21.15.46:0/746188990 learned_addr learned my addr 172.21.15.46:0/746188990 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5936800640 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d60259f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 -- 172.21.15.46:0/746188990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 msgr2=0x55d6025a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 -- 172.21.15.46:0/746188990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 msgr2=0x55d60259f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d60259f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 -- 172.21.15.46:0/746188990 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6025662b0 con 0x55d6025a8a30 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935e00640 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f592ea00640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 0x7f591c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.345+0000 7f5935400640 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025aae20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f593819b280 tx=0x7f5924011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9519536b0f3c6b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f5923400640 1 -- 172.21.15.46:0/746188990 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924004050 con 0x55d6025a8a30 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f592ea00640 1 -- 172.21.15.46:0/2562988803 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f59280f0bc0 con 0x7f59380ff6e0 2024-03-08T23:09:48.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f5936800640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 0x7f591c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f5923400640 1 -- 172.21.15.46:0/746188990 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5924005500 con 0x55d6025a8a30 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f5923400640 1 -- 172.21.15.46:0/746188990 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5924004db0 con 0x55d6025a8a30 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 -- 172.21.15.46:0/746188990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 msgr2=0x55d6025aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025aae20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f593819b280 tx=0x7f5924011810 comp rx=0 tx=0).stop 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f5936800640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 0x7f591c0a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f593806ab80 tx=0x7f5930009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 -- 172.21.15.46:0/746188990 shutdown_connections 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025aae20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 --2- 172.21.15.46:0/746188990 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d60259f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.346+0000 7f593fa26740 1 -- 172.21.15.46:0/746188990 >> 172.21.15.46:0/746188990 conn(0x55d602599c60 msgr2=0x55d60259a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.347+0000 7f593fa26740 1 -- 172.21.15.46:0/746188990 shutdown_connections 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.347+0000 7f593fa26740 1 -- 172.21.15.46:0/746188990 wait complete. 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.347+0000 7f593fa26740 1 -- start start 2024-03-08T23:09:48.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.347+0000 7f593fa26740 1 Processor -- start 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6025ab820 con 0x55d60259f530 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6024c00c0 con 0x55d60259fe50 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6025ab9a0 con 0x55d6025a8a30 2024-03-08T23:09:48.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f593fa26740 10 client.0 _ll_register_callbacks cb 0x55d6024faec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33222/0 (socket says 172.21.15.46:33222) 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 -- 172.21.15.46:0/3702567384 learned_addr learned my addr 172.21.15.46:0/3702567384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5936800640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 -- 172.21.15.46:0/3702567384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 msgr2=0x55d6025a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 -- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 msgr2=0x55d6025a5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5935e00640 1 -- 172.21.15.46:0/3702567384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6022183f0 con 0x55d60259f530 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.348+0000 7f5936800640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f5935e00640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f59280c51e0 tx=0x7f59280ee080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435a67a3bb2beb72 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f5917e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592804b040 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f59280c5280 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5924005500 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f59280bcd40 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f593fa26740 10 client.15930 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.349+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d6022e27d0 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5924005500 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f59280bc3c0 con 0x55d60259f530 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 10 client.15930 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 20 client.15930 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6025b88b0 0x55d6025bad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 4 client.15930 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:48.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6024d84f0 con 0x55d6025b88b0 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.350+0000 7f5935400640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6025b88b0 0x55d6025bad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f5935400640 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6025b88b0 0x55d6025bad00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3500623e84635cf server_cookie=d45c38b493b02bfe in_seq=0 out_seq=0 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f5917e00640 10 client.15930 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f5917e00640 1 -- 172.21.15.46:0/3702567384 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d6024d84f0 con 0x55d6025b88b0 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f5917e00640 10 client.15930 handle_command_reply: tid=0 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 client.15930 shutdown 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 msgr2=0x55d6025a8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f59280c51e0 tx=0x7f59280ee080 comp rx=0 tx=0).stop 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 shutdown_connections 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6025b88b0 0x55d6025bad00 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6025a8a30 0x55d6025a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d60259fe50 0x55d6025a5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 --2- 172.21.15.46:0/3702567384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a8300 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f5917e00640 0 client.15930 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.351+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 >> 172.21.15.46:0/3702567384 conn(0x55d602599c60 msgr2=0x55d60259b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.352+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 shutdown_connections 2024-03-08T23:09:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.352+0000 7f593fa26740 1 -- 172.21.15.46:0/3702567384 wait complete. 2024-03-08T23:09:48.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.352+0000 7f593fa26740 20 client.15930 trim_cache size 0 max 16384 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.365+0000 7f593fa26740 1 Processor -- start 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 -- start start 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025a8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381b2a50 con 0x55d60259f530 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381b05f0 con 0x55d6025a8a30 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f593fa26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381ae690 con 0x55d60259fe50 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5936800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5935e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5936800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43410/0 (socket says 172.21.15.46:43410) 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5936800640 1 -- 172.21.15.46:0/2295396850 learned_addr learned my addr 172.21.15.46:0/2295396850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5935e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33236/0 (socket says 172.21.15.46:33236) 2024-03-08T23:09:48.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.366+0000 7f5935400640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 -- 172.21.15.46:0/2295396850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 msgr2=0x55d6025a8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025a8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 -- 172.21.15.46:0/2295396850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 msgr2=0x55d6025a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 -- 172.21.15.46:0/2295396850 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d602599a90 con 0x55d6025a8a30 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5935400640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5935e00640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5936800640 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f593001f480 tx=0x7f5930016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02ce1256432073e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5922a00640 1 -- 172.21.15.46:0/2295396850 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593002d070 con 0x55d6025a8a30 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5922a00640 1 -- 172.21.15.46:0/2295396850 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59300288f0 con 0x55d6025a8a30 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.367+0000 7f5922a00640 1 -- 172.21.15.46:0/2295396850 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593001c760 con 0x55d6025a8a30 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 -- 172.21.15.46:0/2295396850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 msgr2=0x55d6025a28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f593001f480 tx=0x7f5930016000 comp rx=0 tx=0).stop 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 -- 172.21.15.46:0/2295396850 shutdown_connections 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025a28f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025a8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 --2- 172.21.15.46:0/2295396850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 -- 172.21.15.46:0/2295396850 >> 172.21.15.46:0/2295396850 conn(0x55d602599c60 msgr2=0x55d6025a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 -- 172.21.15.46:0/2295396850 shutdown_connections 2024-03-08T23:09:48.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.368+0000 7f593fa26740 1 -- 172.21.15.46:0/2295396850 wait complete. 2024-03-08T23:09:48.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 -- start start 2024-03-08T23:09:48.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 Processor -- start 2024-03-08T23:09:48.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025ae830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025aed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381aec90 con 0x55d60259f530 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381aff90 con 0x55d6025a8a30 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59381af2d0 con 0x55d60259fe50 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.369+0000 7f593fa26740 10 client.0 _ll_register_callbacks cb 0x7f59240097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5935e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43426/0 (socket says 172.21.15.46:43426) 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 -- 172.21.15.46:0/210573261 learned_addr learned my addr 172.21.15.46:0/210573261 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5935400640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 -- 172.21.15.46:0/210573261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 msgr2=0x55d6025aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 -- 172.21.15.46:0/210573261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 msgr2=0x55d6025ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 -- 172.21.15.46:0/210573261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d60260f0d0 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5935e00640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5936800640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5930009b50 tx=0x7f5930028880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12597fe71a52823f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.370+0000 7f5917e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593002d070 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f5930028cb0 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59300288f0 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5930004a40 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 10 client.26057 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55d6026103b0 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59300288f0 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5930005040 con 0x55d6025a8a30 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 10 client.26057 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:48.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 10 client.26057 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 20 client.26057 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d602610a60 0x55d602612e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 4 client.26057 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.371+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d602511310 con 0x55d602610a60 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.372+0000 7f5935e00640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d602610a60 0x55d602612e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.372+0000 7f5935e00640 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d602610a60 0x55d602612e80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=56d1a8c661f9bc88 server_cookie=f5591c5f854adfe8 in_seq=0 out_seq=0 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.372+0000 7f5917e00640 10 client.26057 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f5917e00640 1 -- 172.21.15.46:0/210573261 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d602511310 con 0x55d602610a60 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f5917e00640 10 client.26057 handle_command_reply: tid=0 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 client.26057 shutdown 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 msgr2=0x55d6025af2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5930009b50 tx=0x7f5930028880 comp rx=0 tx=0).stop 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 shutdown_connections 2024-03-08T23:09:48.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d602610a60 0x55d602612e80 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6025a8a30 0x55d6025af2b0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d60259fe50 0x55d6025aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 --2- 172.21.15.46:0/210573261 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d60259f530 0x55d6025ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f5917e00640 0 client.26057 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 >> 172.21.15.46:0/210573261 conn(0x55d602599c60 msgr2=0x55d6025a0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 shutdown_connections 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.373+0000 7f593fa26740 1 -- 172.21.15.46:0/210573261 wait complete. 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f593fa26740 20 client.26057 trim_cache size 0 max 16384 2024-03-08T23:09:48.380 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 msgr2=0x7f591c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 0x7f591c0a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f593806ab80 tx=0x7f5930009500 comp rx=0 tx=0).stop 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 msgr2=0x7f593819f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5928005940 tx=0x7f592802cb30 comp rx=0 tx=0).stop 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 shutdown_connections 2024-03-08T23:09:48.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f591c0a29f0 0x7f591c0a4eb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f593810dc20 0x7f5938199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5938100000 0x7f593819fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 --2- 172.21.15.46:0/2562988803 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f59380ff6e0 0x7f593819f8e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 >> 172.21.15.46:0/2562988803 conn(0x7f5938074e90 msgr2=0x7f593810e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.374+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 shutdown_connections 2024-03-08T23:09:48.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:48.375+0000 7f5937200640 1 -- 172.21.15.46:0/2562988803 wait complete. 2024-03-08T23:09:49.394 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:09:49.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:09:49.395 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 -- 172.21.15.46:0/2553328233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8710101d70 msgr2=0x7f8710105db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 --2- 172.21.15.46:0/2553328233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8710101d70 0x7f8710105db0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f86fc009900 tx=0x7f86fc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 -- 172.21.15.46:0/2553328233 shutdown_connections 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 --2- 172.21.15.46:0/2553328233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f871010e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 --2- 172.21.15.46:0/2553328233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f8710101d70 0x7f8710105db0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 --2- 172.21.15.46:0/2553328233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87101013c0 0x7f87101017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 -- 172.21.15.46:0/2553328233 >> 172.21.15.46:0/2553328233 conn(0x7f87100fcc60 msgr2=0x7f87100ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 -- 172.21.15.46:0/2553328233 shutdown_connections 2024-03-08T23:09:49.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.640+0000 7f8715600640 1 -- 172.21.15.46:0/2553328233 wait complete. 2024-03-08T23:09:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.641+0000 7f8715600640 1 Processor -- start 2024-03-08T23:09:49.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 -- start start 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 0x7f87101aa270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8710111520 con 0x7f87101013c0 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101113a0 con 0x7f8710106490 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8715600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101116a0 con 0x7f8710108b00 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f870f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f870fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8714c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 0x7f87101aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f870fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33276/0 (socket says 172.21.15.46:33276) 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f8714c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 0x7f87101aa270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55632/0 (socket says 172.21.15.46:55632) 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f870fe00640 1 -- 172.21.15.46:0/1309077374 learned_addr learned my addr 172.21.15.46:0/1309077374 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.642+0000 7f870f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43454/0 (socket says 172.21.15.46:43454) 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 -- 172.21.15.46:0/1309077374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 msgr2=0x7f87101aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 0x7f87101aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 -- 172.21.15.46:0/1309077374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 msgr2=0x7f87101085c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 -- 172.21.15.46:0/1309077374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87101aa7b0 con 0x7f87101013c0 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870f400640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870fe00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9f0 tx=0x7f870000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870cc00640 1 -- 172.21.15.46:0/1309077374 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870000ccc0 con 0x7f87101013c0 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f870cc00640 1 -- 172.21.15.46:0/1309077374 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f870000ce60 con 0x7f87101013c0 2024-03-08T23:09:49.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.643+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87101aaaa0 con 0x7f87101013c0 2024-03-08T23:09:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.644+0000 7f870cc00640 1 -- 172.21.15.46:0/1309077374 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8700010690 con 0x7f87101013c0 2024-03-08T23:09:49.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.644+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87101aaf70 con 0x7f87101013c0 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.647+0000 7f870cc00640 1 -- 172.21.15.46:0/1309077374 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f87000046c0 con 0x7f87101013c0 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.647+0000 7f8719879740 1 Processor -- start 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.647+0000 7f8719879740 1 -- start start 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68633760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68636090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6858ea50 con 0x556b6862b0c0 2024-03-08T23:09:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b68537b00 con 0x556b68633ca0 2024-03-08T23:09:49.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8719879740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6858aaf0 con 0x556b6862a7a0 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8714c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68636090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68633760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8714c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55644/0 (socket says 172.21.15.46:55644) 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8714c00640 1 -- 172.21.15.46:0/385321038 learned_addr learned my addr 172.21.15.46:0/385321038 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 -- 172.21.15.46:0/385321038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 msgr2=0x556b6862ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 -- 172.21.15.46:0/385321038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 msgr2=0x556b68633760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68633760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f870f400640 1 -- 172.21.15.46:0/385321038 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b685551e0 con 0x556b68633ca0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.648+0000 7f8714c00640 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f870cc00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 0x7f86f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f870f400640 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68636090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036040 tx=0x7f86fc005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0df41625f011097 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f870cc00640 1 -- 172.21.15.46:0/1309077374 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f87000cf5e0 con 0x7f87101013c0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f8714c00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 0x7f86f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f8705200640 1 -- 172.21.15.46:0/385321038 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc042070 con 0x556b68633ca0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f8705200640 1 -- 172.21.15.46:0/385321038 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86fc004a70 con 0x556b68633ca0 2024-03-08T23:09:49.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.649+0000 7f8705200640 1 -- 172.21.15.46:0/385321038 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc005210 con 0x556b68633ca0 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 -- 172.21.15.46:0/385321038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 msgr2=0x556b68636090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8714c00640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 0x7f86f40a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87080127c0 tx=0x7f8708009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68636090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f86fc036040 tx=0x7f86fc005e60 comp rx=0 tx=0).stop 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 -- 172.21.15.46:0/385321038 shutdown_connections 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68636090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68633760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 --2- 172.21.15.46:0/385321038 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b6862ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 -- 172.21.15.46:0/385321038 >> 172.21.15.46:0/385321038 conn(0x556b68624ed0 msgr2=0x556b686252b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 -- 172.21.15.46:0/385321038 shutdown_connections 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.650+0000 7f8719879740 1 -- 172.21.15.46:0/385321038 wait complete. 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.651+0000 7f8719879740 1 -- start start 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.651+0000 7f8719879740 1 Processor -- start 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b68630ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68631720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b68636a30 con 0x556b6862a7a0 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b6854af60 con 0x556b68633ca0 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x556b68636bb0 con 0x556b6862b0c0 2024-03-08T23:09:49.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8719879740 10 client.0 _ll_register_callbacks cb 0x556b68536b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8714c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68631720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55650/0 (socket says 172.21.15.46:55650) 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 -- 172.21.15.46:0/2718857628 learned_addr learned my addr 172.21.15.46:0/2718857628 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870fe00640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b68630ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 -- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 msgr2=0x556b68631720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68631720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 -- 172.21.15.46:0/2718857628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 msgr2=0x556b68630ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b68630ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f870f400640 1 -- 172.21.15.46:0/2718857628 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6836b7e0 con 0x556b6862b0c0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.652+0000 7f8714c00640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68631720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f870f400640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86fc02c170 tx=0x7f86fc02cc70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=313db028ccfa9ca9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f86f9600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:49.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc04e060 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f86fc004a70 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f86fc0053a0 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f86fc055050 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f8719879740 10 client.26080 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.653+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556b683cbf70 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86fc004a70 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f86fc03e400 con 0x556b6862b0c0 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 10 client.26080 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 10 client.26080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 20 client.26080 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b68646200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 4 client.26080 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:49.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556b6859bf90 con 0x556b68643e10 2024-03-08T23:09:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.654+0000 7f870fe00640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b68646200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.655+0000 7f870fe00640 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b68646200 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2709ce424d62106b server_cookie=7d32d39cb2fc4b0d in_seq=0 out_seq=0 2024-03-08T23:09:49.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.655+0000 7f86f9600640 10 client.26080 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.655+0000 7f86f9600640 1 -- 172.21.15.46:0/2718857628 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556b6859bf90 con 0x556b68643e10 2024-03-08T23:09:49.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.655+0000 7f86f9600640 10 client.26080 handle_command_reply: tid=0 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.655+0000 7f8719879740 1 client.26080 shutdown 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 msgr2=0x556b686311e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86fc02c170 tx=0x7f86fc02cc70 comp rx=0 tx=0).stop 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 shutdown_connections 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b68646200 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b68631720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b686311e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f86f9600640 0 client.26080 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 --2- 172.21.15.46:0/2718857628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b68630ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 >> 172.21.15.46:0/2718857628 conn(0x556b68624ed0 msgr2=0x556b68626c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 shutdown_connections 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 1 -- 172.21.15.46:0/2718857628 wait complete. 2024-03-08T23:09:49.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.656+0000 7f8719879740 20 client.26080 trim_cache size 0 max 16384 2024-03-08T23:09:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.677+0000 7f8719879740 1 Processor -- start 2024-03-08T23:09:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 -- start start 2024-03-08T23:09:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b686333d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6862e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b6cb0 con 0x556b6862a7a0 2024-03-08T23:09:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b4850 con 0x556b68633ca0 2024-03-08T23:09:49.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8719879740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b28f0 con 0x556b6862b0c0 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f870f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f870fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b686333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f870f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55654/0 (socket says 172.21.15.46:55654) 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f870f400640 1 -- 172.21.15.46:0/2115147924 learned_addr learned my addr 172.21.15.46:0/2115147924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.678+0000 7f8714c00640 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6862e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 -- 172.21.15.46:0/2115147924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 msgr2=0x556b6862e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6862e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 -- 172.21.15.46:0/2115147924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 msgr2=0x556b686333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b686333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 -- 172.21.15.46:0/2115147924 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556b68263170 con 0x556b6862b0c0 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f870f400640 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005ec0 tx=0x7f86fc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d965d236f2045862 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f86fbe00640 1 -- 172.21.15.46:0/2115147924 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc04e070 con 0x556b6862b0c0 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f86fbe00640 1 -- 172.21.15.46:0/2115147924 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86fc02cce0 con 0x556b6862b0c0 2024-03-08T23:09:49.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.679+0000 7f86fbe00640 1 -- 172.21.15.46:0/2115147924 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86fc042070 con 0x556b6862b0c0 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 -- 172.21.15.46:0/2115147924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 msgr2=0x556b6862df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86fc005ec0 tx=0x7f86fc005fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 -- 172.21.15.46:0/2115147924 shutdown_connections 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6862e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862b0c0 0x556b6862df00 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 --2- 172.21.15.46:0/2115147924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862a7a0 0x556b686333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 -- 172.21.15.46:0/2115147924 >> 172.21.15.46:0/2115147924 conn(0x556b68624ed0 msgr2=0x556b6862bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 -- 172.21.15.46:0/2115147924 shutdown_connections 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.680+0000 7f8719879740 1 -- 172.21.15.46:0/2115147924 wait complete. 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.681+0000 7f8719879740 1 -- start start 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.681+0000 7f8719879740 1 Processor -- start 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.681+0000 7f8719879740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68639fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6869a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b2ef0 con 0x556b6862b0c0 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b41f0 con 0x556b68633ca0 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87101b3530 con 0x556b6862a7a0 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8719879740 10 client.0 _ll_register_callbacks cb 0x7f8700005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:49.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f8714c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6869a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55670/0 (socket says 172.21.15.46:55670) 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68639fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 -- 172.21.15.46:0/2347124494 learned_addr learned my addr 172.21.15.46:0/2347124494 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 -- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 msgr2=0x556b6869a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6869a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 -- 172.21.15.46:0/2347124494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 msgr2=0x556b68639fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68639fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 -- 172.21.15.46:0/2347124494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556b6855a050 con 0x556b6862a7a0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.682+0000 7f870fe00640 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f870000c6a0 tx=0x7f87000d5110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea8d0a1e78f64b0b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87000d2e60 con 0x556b6862a7a0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f87000860b0 con 0x556b6862a7a0 2024-03-08T23:09:49.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556b685962e0 con 0x556b6862a7a0 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f870009bf80 con 0x556b6862a7a0 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 10 client.26086 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556b6869b500 con 0x556b6862a7a0 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556b685962e0 con 0x556b6862a7a0 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f870009bae0 con 0x556b6862a7a0 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 10 client.26086 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 10 client.26086 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 20 client.26086 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b6869dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 4 client.26086 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.683+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556b68514a40 con 0x556b68643e10 2024-03-08T23:09:49.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.684+0000 7f870f400640 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b6869dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.684+0000 7f870f400640 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b6869dc20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=799541350584bd47 server_cookie=7be2d2a3615c5515 in_seq=0 out_seq=0 2024-03-08T23:09:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.684+0000 7f86f9600640 10 client.26086 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f86f9600640 1 -- 172.21.15.46:0/2347124494 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556b68514a40 con 0x556b68643e10 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f86f9600640 10 client.26086 handle_command_reply: tid=0 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 client.26086 shutdown 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 msgr2=0x556b68639aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f870000c6a0 tx=0x7f87000d5110 comp rx=0 tx=0).stop 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 shutdown_connections 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556b68643e10 0x556b6869dc20 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556b68633ca0 0x556b6869a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556b6862b0c0 0x556b68639fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 --2- 172.21.15.46:0/2347124494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556b6862a7a0 0x556b68639aa0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f86f9600640 0 client.26086 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.686+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 >> 172.21.15.46:0/2347124494 conn(0x556b68624ed0 msgr2=0x556b68630c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 shutdown_connections 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8719879740 1 -- 172.21.15.46:0/2347124494 wait complete. 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8719879740 20 client.26086 trim_cache size 0 max 16384 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "ad6d6b29-b54a-44d7-b8fb-a1e534086cd7", 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:09:49.692 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 msgr2=0x7f86f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 0x7f86f40a4e30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87080127c0 tx=0x7f8708009500 comp rx=0 tx=0).stop 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 msgr2=0x7f8710108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.687+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f870000e9f0 tx=0x7f870000edf0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 shutdown_connections 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f86f40a2970 0x7f86f40a4e30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8710108b00 0x7f87101aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f8710106490 0x7f87101085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 --2- 172.21.15.46:0/1309077374 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f87101013c0 0x7f8710108080 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 >> 172.21.15.46:0/1309077374 conn(0x7f87100fcc60 msgr2=0x7f87100fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 shutdown_connections 2024-03-08T23:09:49.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.688+0000 7f8715600640 1 -- 172.21.15.46:0/1309077374 wait complete. 2024-03-08T23:09:49.714 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:09:49.714 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:09:49.798 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:49 smithi046 ceph-mon[31859]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 495 op/s 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 -- 172.21.15.46:0/3906170546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc60103e70 msgr2=0x7fcc60110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3906170546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc60103e70 0x7fcc60110330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009c10 tx=0x7fcc5802bee0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 -- 172.21.15.46:0/3906170546 shutdown_connections 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3906170546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc60103e70 0x7fcc60110330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3906170546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc601034b0 0x7fcc60103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3906170546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 0x7fcc60109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 -- 172.21.15.46:0/3906170546 >> 172.21.15.46:0/3906170546 conn(0x7fcc600fef40 msgr2=0x7fcc60101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 -- 172.21.15.46:0/3906170546 shutdown_connections 2024-03-08T23:09:49.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.924+0000 7fcc64c00640 1 -- 172.21.15.46:0/3906170546 wait complete. 2024-03-08T23:09:49.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.925+0000 7fcc64c00640 1 Processor -- start 2024-03-08T23:09:49.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.925+0000 7fcc64c00640 1 -- start start 2024-03-08T23:09:49.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 0x7fcc60199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 0x7fcc6019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60113880 con 0x7fcc60109570 2024-03-08T23:09:49.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60113700 con 0x7fcc60103e70 2024-03-08T23:09:49.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc64c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60113a00 con 0x7fcc601034b0 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43522/0 (socket says 172.21.15.46:43522) 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc5d400640 1 -- 172.21.15.46:0/3522655574 learned_addr learned my addr 172.21.15.46:0/3522655574 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 0x7fcc60199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.926+0000 7fcc5e800640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 0x7fcc6019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 -- 172.21.15.46:0/3522655574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 msgr2=0x7fcc6019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 -- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 msgr2=0x7fcc6019e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 0x7fcc6019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 -- 172.21.15.46:0/3522655574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6019ed50 con 0x7fcc601034b0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 0x7fcc60199d60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800e9f0 tx=0x7fcc4800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc56a00640 1 -- 172.21.15.46:0/3522655574 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4800cd80 con 0x7fcc601034b0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc56a00640 1 -- 172.21.15.46:0/3522655574 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc480035f0 con 0x7fcc601034b0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6019f040 con 0x7fcc601034b0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.927+0000 7fcc56a00640 1 -- 172.21.15.46:0/3522655574 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48010690 con 0x7fcc601034b0 2024-03-08T23:09:49.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.928+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc601a68d0 con 0x7fcc601034b0 2024-03-08T23:09:49.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.932+0000 7fcc67991740 1 Processor -- start 2024-03-08T23:09:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.932+0000 7fcc67991740 1 -- start start 2024-03-08T23:09:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc56a00640 1 -- 172.21.15.46:0/3522655574 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcc48010830 con 0x7fcc601034b0 2024-03-08T23:09:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a420693960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a420655410 con 0x55a420693ea0 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4205f7d50 con 0x55a42069ca80 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc67991740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4205a0b80 con 0x55a420693580 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43526/0 (socket says 172.21.15.46:43526) 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc5d400640 1 -- 172.21.15.46:0/3247020186 learned_addr learned my addr 172.21.15.46:0/3247020186 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.933+0000 7fcc5e800640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a420693960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 -- 172.21.15.46:0/3247020186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 msgr2=0x55a420693960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a420693960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 -- 172.21.15.46:0/3247020186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 msgr2=0x55a42069c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 -- 172.21.15.46:0/3247020186 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4203d67d0 con 0x55a42069ca80 2024-03-08T23:09:49.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:09:49.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc500115f0 tx=0x7fcc50009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32f71f46307093cf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.934+0000 7fcc56a00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 0x7fcc440a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.935+0000 7fcc4f000640 1 -- 172.21.15.46:0/3247020186 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50005020 con 0x55a42069ca80 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.935+0000 7fcc56a00640 1 -- 172.21.15.46:0/3522655574 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcc480cf4b0 con 0x7fcc601034b0 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.935+0000 7fcc4f000640 1 -- 172.21.15.46:0/3247020186 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc500047a0 con 0x55a42069ca80 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.935+0000 7fcc4f000640 1 -- 172.21.15.46:0/3247020186 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5000faa0 con 0x55a42069ca80 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.935+0000 7fcc5e800640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 0x7fcc440a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 -- 172.21.15.46:0/3247020186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 msgr2=0x55a42069ee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc500115f0 tx=0x7fcc50009f90 comp rx=0 tx=0).stop 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc5e800640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 0x7fcc440a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009be0 tx=0x7fcc58005fd0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 -- 172.21.15.46:0/3247020186 shutdown_connections 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a42069ee70 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 --2- 172.21.15.46:0/3247020186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a420693960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 -- 172.21.15.46:0/3247020186 >> 172.21.15.46:0/3247020186 conn(0x55a42068dcb0 msgr2=0x55a42068e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 -- 172.21.15.46:0/3247020186 shutdown_connections 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.936+0000 7fcc67991740 1 -- 172.21.15.46:0/3247020186 wait complete. 2024-03-08T23:09:49.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.937+0000 7fcc67991740 1 -- start start 2024-03-08T23:09:49.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.937+0000 7fcc67991740 1 Processor -- start 2024-03-08T23:09:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a42069a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4205b3ee0 con 0x55a420693ea0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4205f3df0 con 0x55a420693580 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42069f870 con 0x55a42069ca80 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc67991740 10 client.0 _ll_register_callbacks cb 0x55a4205ba8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43528/0 (socket says 172.21.15.46:43528) 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 -- 172.21.15.46:0/4271550039 learned_addr learned my addr 172.21.15.46:0/4271550039 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5e800640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a42069a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 -- 172.21.15.46:0/4271550039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 msgr2=0x55a42069a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a42069a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 -- 172.21.15.46:0/4271550039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 msgr2=0x55a42069a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.938+0000 7fcc5de00640 1 -- 172.21.15.46:0/4271550039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a42030c3f0 con 0x55a420693580 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc5de00640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c6a0 tx=0x7fcc4809b200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20cd46ff06aba7a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc43400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc4809f900 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcc480ccb20 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc500047a0 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc480a3f50 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc67991740 10 client.26087 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.939+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a4202cc0f0 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc500047a0 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcc480cf0f0 con 0x55a420693580 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 10 client.26087 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 10 client.26087 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 20 client.26087 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a4206acc10 0x55a4206af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 4 client.26087 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:09:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a420605290 con 0x55a4206acc10 2024-03-08T23:09:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.940+0000 7fcc5d400640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a4206acc10 0x55a4206af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.941+0000 7fcc5d400640 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a4206acc10 0x55a4206af040 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b866c288e34320f1 server_cookie=1b11fa468644c94c in_seq=0 out_seq=0 2024-03-08T23:09:49.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.941+0000 7fcc43400640 10 client.26087 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.942+0000 7fcc43400640 1 -- 172.21.15.46:0/4271550039 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a420605290 con 0x55a4206acc10 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc43400640 10 client.26087 handle_command_reply: tid=0 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 client.26087 shutdown 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 msgr2=0x55a420699ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800c6a0 tx=0x7fcc4809b200 comp rx=0 tx=0).stop 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 shutdown_connections 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a4206acc10 0x55a4206af040 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a42069a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693ea0 0x55a42069a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 --2- 172.21.15.46:0/4271550039 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693580 0x55a420699ad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 >> 172.21.15.46:0/4271550039 conn(0x55a42068dcb0 msgr2=0x55a42068fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc43400640 0 client.26087 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 shutdown_connections 2024-03-08T23:09:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 1 -- 172.21.15.46:0/4271550039 wait complete. 2024-03-08T23:09:49.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.943+0000 7fcc67991740 20 client.26087 trim_cache size 0 max 16384 2024-03-08T23:09:49.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.967+0000 7fcc67991740 1 Processor -- start 2024-03-08T23:09:49.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.967+0000 7fcc67991740 1 -- start start 2024-03-08T23:09:49.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a4206a3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601b2d70 con 0x55a420693580 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601b0910 con 0x55a420693ea0 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc67991740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601ae9b0 con 0x55a42069ca80 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a4206a3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33322/0 (socket says 172.21.15.46:33322) 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43530/0 (socket says 172.21.15.46:43530) 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5d400640 1 -- 172.21.15.46:0/3105173686 learned_addr learned my addr 172.21.15.46:0/3105173686 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.968+0000 7fcc5e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a4206a3ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55718/0 (socket says 172.21.15.46:55718) 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 -- 172.21.15.46:0/3105173686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 msgr2=0x55a4206a3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a4206a3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 -- 172.21.15.46:0/3105173686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 msgr2=0x55a42069c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 -- 172.21.15.46:0/3105173686 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a420698040 con 0x55a420693ea0 2024-03-08T23:09:49.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5de00640 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:09:49.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.969+0000 7fcc5d400640 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcc50000c00 tx=0x7fcc50009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a26ab01d8eb4f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.970+0000 7fcc4e600640 1 -- 172.21.15.46:0/3105173686 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50005360 con 0x55a420693ea0 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.970+0000 7fcc4e600640 1 -- 172.21.15.46:0/3105173686 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5002d020 con 0x55a420693ea0 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.970+0000 7fcc4e600640 1 -- 172.21.15.46:0/3105173686 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50029890 con 0x55a420693ea0 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 -- 172.21.15.46:0/3105173686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 msgr2=0x55a4206a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcc50000c00 tx=0x7fcc50009f90 comp rx=0 tx=0).stop 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 -- 172.21.15.46:0/3105173686 shutdown_connections 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a42069ca80 0x55a4206a3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a420693ea0 0x55a4206a3ab0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 --2- 172.21.15.46:0/3105173686 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a420693580 0x55a42069c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 -- 172.21.15.46:0/3105173686 >> 172.21.15.46:0/3105173686 conn(0x55a42068dcb0 msgr2=0x55a420694280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:49.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 -- 172.21.15.46:0/3105173686 shutdown_connections 2024-03-08T23:09:49.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.971+0000 7fcc67991740 1 -- 172.21.15.46:0/3105173686 wait complete. 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.972+0000 7fcc67991740 1 -- start start 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.972+0000 7fcc67991740 1 Processor -- start 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a4206fcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a4206fd0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 0x55a4206f7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601aefb0 con 0x55a4206f70b0 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601b02b0 con 0x55a42069ca80 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601af5f0 con 0x55a420693580 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc67991740 10 client.0 _ll_register_callbacks cb 0x7fcc50009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a4206fcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a4206fd0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a4206fcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:55732/0 (socket says 172.21.15.46:55732) 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5de00640 1 -- 172.21.15.46:0/2464214089 learned_addr learned my addr 172.21.15.46:0/2464214089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:09:49.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5e800640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 0x55a4206f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5d400640 1 -- 172.21.15.46:0/2464214089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 msgr2=0x55a4206fcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.973+0000 7fcc5d400640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a4206fcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc5d400640 1 -- 172.21.15.46:0/2464214089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 msgr2=0x55a4206f7530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc5d400640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 0x55a4206f7530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc5d400640 1 -- 172.21.15.46:0/2464214089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4206f7b70 con 0x55a42069ca80 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc5e800640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 0x55a4206f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc5d400640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a4206fd0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55a42069f790 tx=0x7fcc50005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2ae26eea49a1e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc43400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:09:49.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500293e0 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc5002d020 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.974+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcc50016040 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc50005570 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 10 client.26093 fetch_fsmap learned FSMap version 25 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a420704740 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc5002d020 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 25) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcc5000f9f0 con 0x55a42069ca80 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 10 client.26093 fetch_fsmap finished waiting for FSMap version 25 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 10 client.26093 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 20 client.26093 populate_metadata read hostname 'smithi046' 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a420704c20 0x55a420707040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 4 client.26093 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.975+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a4205ff5e0 con 0x55a420704c20 2024-03-08T23:09:49.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.976+0000 7fcc5de00640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a420704c20 0x55a420707040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:09:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.976+0000 7fcc43400640 10 client.26093 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:49.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:49.976+0000 7fcc5de00640 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a420704c20 0x55a420707040 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=810ad28156db2c32 server_cookie=ab0ce9af8d5317e0 in_seq=0 out_seq=0 2024-03-08T23:09:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.000+0000 7fcc43400640 1 -- 172.21.15.46:0/2464214089 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55a4205ff5e0 con 0x55a420704c20 2024-03-08T23:09:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.000+0000 7fcc43400640 10 client.26093 handle_command_reply: tid=0 2024-03-08T23:09:50.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.000+0000 7fcc67991740 1 client.26093 shutdown 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 msgr2=0x55a4206fd0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a4206fd0a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55a42069f790 tx=0x7fcc50005de0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 shutdown_connections 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a420704c20 0x55a420707040 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a4206f70b0 0x55a4206f7530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a42069ca80 0x55a4206fd0a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 --2- 172.21.15.46:0/2464214089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a420693580 0x55a4206fcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 >> 172.21.15.46:0/2464214089 conn(0x55a42068dcb0 msgr2=0x55a4206983c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:50.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc43400640 0 client.26093 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:09:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.001+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 shutdown_connections 2024-03-08T23:09:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.002+0000 7fcc67991740 1 -- 172.21.15.46:0/2464214089 wait complete. 2024-03-08T23:09:50.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.002+0000 7fcc67991740 20 client.26093 trim_cache size 0 max 16384 2024-03-08T23:09:50.005 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "scrub active (1300 inodes in the stack)", 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "ad6d6b29-b54a-44d7-b8fb-a1e534086cd7": { 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "ad6d6b29-b54a-44d7-b8fb-a1e534086cd7", 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:09:50.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.003+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 msgr2=0x7fcc440a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.003+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 0x7fcc440a4e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fcc58009be0 tx=0x7fcc58005fd0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.003+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 msgr2=0x7fcc60199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.003+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 0x7fcc60199d60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcc4800e9f0 tx=0x7fcc4800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.003+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 shutdown_connections 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.004+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcc440a2970 0x7fcc440a4e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.004+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcc60109570 0x7fcc6019e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.004+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcc60103e70 0x7fcc6019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.004+0000 7fcc64c00640 1 --2- 172.21.15.46:0/3522655574 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc601034b0 0x7fcc60199d60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.004+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 >> 172.21.15.46:0/3522655574 conn(0x7fcc600fef40 msgr2=0x7fcc60100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.005+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 shutdown_connections 2024-03-08T23:09:50.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:09:50.005+0000 7fcc64c00640 1 -- 172.21.15.46:0/3522655574 wait complete. 2024-03-08T23:09:50.030 INFO:tasks.cephfs.filesystem:scrub status for tag:ad6d6b29-b54a-44d7-b8fb-a1e534086cd7 - {'path': '/', 'tag': 'ad6d6b29-b54a-44d7-b8fb-a1e534086cd7', 'options': 'recursive,force'} 2024-03-08T23:09:50.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:49 smithi192 ceph-mon[36700]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 495 op/s 2024-03-08T23:09:50.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:49 smithi062 ceph-mon[35761]: pgmap v64: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 495 op/s 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:50.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:09:51.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:51.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:51.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:51 smithi046 ceph-mon[31859]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 482 op/s 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:51 smithi192 ceph-mon[36700]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 482 op/s 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:09:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:51 smithi062 ceph-mon[35761]: pgmap v65: 129 pgs: 129 active+clean; 47 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 482 op/s 2024-03-08T23:09:53.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:53 smithi046 ceph-mon[31859]: pgmap v66: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 554 op/s 2024-03-08T23:09:54.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:53 smithi192 ceph-mon[36700]: pgmap v66: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 554 op/s 2024-03-08T23:09:54.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:53 smithi062 ceph-mon[35761]: pgmap v66: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 3.0 MiB/s wr, 554 op/s 2024-03-08T23:09:55.181 INFO:tasks.workunit.client.0.smithi046.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:state without impacting any branches by switching back to a branch. 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr: git switch -c 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.182 INFO:tasks.workunit.client.0.smithi046.stderr:Or undo this operation with: 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr: git switch - 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr: 2024-03-08T23:09:55.183 INFO:tasks.workunit.client.0.smithi046.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-08T23:09:55.214 INFO:tasks.workunit.client.0.smithi046.stderr:+ make -j4 2024-03-08T23:09:55.243 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize -cfi 2024-03-08T23:09:55.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:55 smithi046 ceph-mon[31859]: pgmap v67: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:56.062 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-08T23:09:56.062 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file './config.guess' 2024-03-08T23:09:56.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:55 smithi192 ceph-mon[36700]: pgmap v67: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:56.075 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file './config.sub' 2024-03-08T23:09:56.088 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file './install-sh' 2024-03-08T23:09:56.104 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file './ltmain.sh' 2024-03-08T23:09:56.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:55 smithi062 ceph-mon[35761]: pgmap v67: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:56.199 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-08T23:09:56.199 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-08T23:09:56.232 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-08T23:09:56.272 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-08T23:09:56.315 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-08T23:09:56.352 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-08T23:09:56.382 INFO:tasks.workunit.client.0.smithi046.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-08T23:09:56.386 INFO:tasks.workunit.client.0.smithi046.stdout:cp include/install-sh . 2024-03-08T23:09:56.394 INFO:tasks.workunit.client.0.smithi046.stdout:aclocal -I m4 2024-03-08T23:09:57.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:57 smithi046 ceph-mon[31859]: pgmap v68: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:57.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:57 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:09:57.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:57 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:09:57.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:57 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:09:58.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:57 smithi192 ceph-mon[36700]: pgmap v68: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:58.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:57 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:09:58.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:57 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:09:58.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:57 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:09:58.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:57 smithi062 ceph-mon[35761]: pgmap v68: 129 pgs: 129 active+clean; 50 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 213 op/s 2024-03-08T23:09:58.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:57 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:09:58.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:57 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:09:58.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:57 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:09:58.319 INFO:tasks.workunit.client.0.smithi046.stdout:autoheader 2024-03-08T23:09:58.792 INFO:tasks.workunit.client.0.smithi046.stdout:autoconf 2024-03-08T23:09:58.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:58 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:09:58.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:09:58.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:09:58.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:58 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:09:59.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:58 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:09:59.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:09:59.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:09:59.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:58 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:09:59.072 INFO:tasks.workunit.client.0.smithi046.stdout:./configure \ 2024-03-08T23:09:59.072 INFO:tasks.workunit.client.0.smithi046.stdout: --libexecdir=/usr/lib \ 2024-03-08T23:09:59.072 INFO:tasks.workunit.client.0.smithi046.stdout: --exec_prefix=/var/lib 2024-03-08T23:09:59.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:58 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:09:59.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:09:59.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:09:59.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:58 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:09:59.469 INFO:tasks.workunit.client.0.smithi046.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-08T23:09:59.469 INFO:tasks.workunit.client.0.smithi046.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-08T23:09:59.473 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to print strings... printf 2024-03-08T23:09:59.473 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gcc... gcc 2024-03-08T23:09:59.642 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether the C compiler works... yes 2024-03-08T23:09:59.642 INFO:tasks.workunit.client.0.smithi046.stdout:checking for C compiler default output file name... a.out 2024-03-08T23:09:59.730 INFO:tasks.workunit.client.0.smithi046.stdout:checking for suffix of executables... 2024-03-08T23:09:59.822 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether we are cross compiling... no 2024-03-08T23:09:59.901 INFO:tasks.workunit.client.0.smithi046.stdout:checking for suffix of object files... o 2024-03-08T23:09:59.961 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether we are using the GNU C compiler... yes 2024-03-08T23:09:59.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:09:59 smithi046 ceph-mon[31859]: pgmap v69: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 410 op/s 2024-03-08T23:10:00.023 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether gcc accepts -g... yes 2024-03-08T23:10:00.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:09:59 smithi192 ceph-mon[36700]: pgmap v69: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 410 op/s 2024-03-08T23:10:00.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:09:59 smithi062 ceph-mon[35761]: pgmap v69: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 410 op/s 2024-03-08T23:10:00.161 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-08T23:10:00.174 INFO:tasks.workunit.client.0.smithi046.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-08T23:10:00.184 INFO:tasks.workunit.client.0.smithi046.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-08T23:10:00.189 INFO:tasks.workunit.client.0.smithi046.stdout:checking for egrep... /usr/bin/grep -E 2024-03-08T23:10:00.194 INFO:tasks.workunit.client.0.smithi046.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-08T23:10:00.207 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-08T23:10:00.213 INFO:tasks.workunit.client.0.smithi046.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-08T23:10:00.228 INFO:tasks.workunit.client.0.smithi046.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-08T23:10:00.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:00 smithi046 ceph-mon[31859]: overall HEALTH_OK 2024-03-08T23:10:01.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:00 smithi192 ceph-mon[36700]: overall HEALTH_OK 2024-03-08T23:10:01.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:00 smithi062 ceph-mon[35761]: overall HEALTH_OK 2024-03-08T23:10:01.217 INFO:tasks.workunit.client.0.smithi046.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-08T23:10:01.217 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether ln -s works... yes 2024-03-08T23:10:01.244 INFO:tasks.workunit.client.0.smithi046.stdout:checking the maximum length of command line arguments... 1572864 2024-03-08T23:10:01.256 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-08T23:10:01.256 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-08T23:10:01.256 INFO:tasks.workunit.client.0.smithi046.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-08T23:10:01.257 INFO:tasks.workunit.client.0.smithi046.stdout:checking for objdump... objdump 2024-03-08T23:10:01.257 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to recognize dependent libraries... pass_all 2024-03-08T23:10:01.258 INFO:tasks.workunit.client.0.smithi046.stdout:checking for dlltool... no 2024-03-08T23:10:01.258 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-08T23:10:01.259 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ar... ar 2024-03-08T23:10:01.408 INFO:tasks.workunit.client.0.smithi046.stdout:checking for archiver @FILE support... @ 2024-03-08T23:10:01.409 INFO:tasks.workunit.client.0.smithi046.stdout:checking for strip... strip 2024-03-08T23:10:01.409 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ranlib... ranlib 2024-03-08T23:10:01.410 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gawk... gawk 2024-03-08T23:10:01.656 INFO:tasks.workunit.client.0.smithi046.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-08T23:10:01.662 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sysroot... no 2024-03-08T23:10:01.686 INFO:tasks.workunit.client.0.smithi046.stdout:checking for a working dd... /usr/bin/dd 2024-03-08T23:10:01.709 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-08T23:10:01.810 INFO:tasks.workunit.client.0.smithi046.stdout:checking for mt... no 2024-03-08T23:10:01.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:01 smithi046 ceph-mon[31859]: pgmap v70: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 268 op/s 2024-03-08T23:10:02.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:01 smithi192 ceph-mon[36700]: pgmap v70: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 268 op/s 2024-03-08T23:10:02.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:01 smithi062 ceph-mon[35761]: pgmap v70: 129 pgs: 129 active+clean; 59 MiB data, 536 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 268 op/s 2024-03-08T23:10:02.215 INFO:tasks.workunit.client.0.smithi046.stdout:checking if : is a manifest tool... no 2024-03-08T23:10:02.368 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to run the C preprocessor... gcc -E 2024-03-08T23:10:02.772 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ANSI C header files... yes 2024-03-08T23:10:02.860 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/types.h... yes 2024-03-08T23:10:02.953 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/stat.h... yes 2024-03-08T23:10:03.049 INFO:tasks.workunit.client.0.smithi046.stdout:checking for stdlib.h... yes 2024-03-08T23:10:03.147 INFO:tasks.workunit.client.0.smithi046.stdout:checking for string.h... yes 2024-03-08T23:10:03.256 INFO:tasks.workunit.client.0.smithi046.stdout:checking for memory.h... yes 2024-03-08T23:10:03.354 INFO:tasks.workunit.client.0.smithi046.stdout:checking for strings.h... yes 2024-03-08T23:10:03.450 INFO:tasks.workunit.client.0.smithi046.stdout:checking for inttypes.h... yes 2024-03-08T23:10:03.548 INFO:tasks.workunit.client.0.smithi046.stdout:checking for stdint.h... yes 2024-03-08T23:10:03.645 INFO:tasks.workunit.client.0.smithi046.stdout:checking for unistd.h... yes 2024-03-08T23:10:03.748 INFO:tasks.workunit.client.0.smithi046.stdout:checking for dlfcn.h... yes 2024-03-08T23:10:03.767 INFO:tasks.workunit.client.0.smithi046.stdout:checking for objdir... .libs 2024-03-08T23:10:04.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:03 smithi192 ceph-mon[36700]: pgmap v71: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 354 op/s 2024-03-08T23:10:04.065 INFO:tasks.workunit.client.0.smithi046.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-08T23:10:04.065 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-08T23:10:04.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:03 smithi062 ceph-mon[35761]: pgmap v71: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 354 op/s 2024-03-08T23:10:04.158 INFO:tasks.workunit.client.0.smithi046.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-08T23:10:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:03 smithi046 ceph-mon[31859]: pgmap v71: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 354 op/s 2024-03-08T23:10:04.241 INFO:tasks.workunit.client.0.smithi046.stdout:checking if gcc static flag -static works... no 2024-03-08T23:10:04.387 INFO:tasks.workunit.client.0.smithi046.stdout:checking if gcc supports -c -o file.o... yes 2024-03-08T23:10:04.387 INFO:tasks.workunit.client.0.smithi046.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-08T23:10:04.414 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-08T23:10:04.541 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether -lc should be explicitly linked in... no 2024-03-08T23:10:04.694 INFO:tasks.workunit.client.0.smithi046.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-08T23:10:04.694 INFO:tasks.workunit.client.0.smithi046.stdout:checking how to hardcode library paths into programs... immediate 2024-03-08T23:10:04.698 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether stripping libraries is possible... yes 2024-03-08T23:10:04.698 INFO:tasks.workunit.client.0.smithi046.stdout:checking if libtool supports shared libraries... yes 2024-03-08T23:10:04.698 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether to build shared libraries... yes 2024-03-08T23:10:04.698 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether to build static libraries... yes 2024-03-08T23:10:04.716 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gcc... (cached) gcc 2024-03-08T23:10:04.785 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-08T23:10:04.786 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-08T23:10:04.786 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-08T23:10:04.786 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gmake... /usr/bin/gmake 2024-03-08T23:10:04.787 INFO:tasks.workunit.client.0.smithi046.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-08T23:10:04.787 INFO:tasks.workunit.client.0.smithi046.stdout:checking for tar... /bin/tar 2024-03-08T23:10:04.787 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gzip... /bin/gzip 2024-03-08T23:10:04.833 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether gcc -MM is supported... yes 2024-03-08T23:10:04.834 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sort... /bin/sort 2024-03-08T23:10:04.835 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether ln -s works... yes 2024-03-08T23:10:04.836 INFO:tasks.workunit.client.0.smithi046.stdout:checking for rpm... /bin/rpm 2024-03-08T23:10:04.850 INFO:tasks.workunit.client.0.smithi046.stdout:checking for rpmbuild... no 2024-03-08T23:10:04.850 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ANSI C header files... (cached) yes 2024-03-08T23:10:04.966 INFO:tasks.workunit.client.0.smithi046.stdout:checking assert.h usability... yes 2024-03-08T23:10:05.004 INFO:tasks.workunit.client.0.smithi046.stdout:checking assert.h presence... yes 2024-03-08T23:10:05.004 INFO:tasks.workunit.client.0.smithi046.stdout:checking for assert.h... yes 2024-03-08T23:10:05.118 INFO:tasks.workunit.client.0.smithi046.stdout:checking bstring.h usability... no 2024-03-08T23:10:05.183 INFO:tasks.workunit.client.0.smithi046.stdout:checking bstring.h presence... no 2024-03-08T23:10:05.183 INFO:tasks.workunit.client.0.smithi046.stdout:checking for bstring.h... no 2024-03-08T23:10:05.278 INFO:tasks.workunit.client.0.smithi046.stdout:checking libgen.h usability... yes 2024-03-08T23:10:05.315 INFO:tasks.workunit.client.0.smithi046.stdout:checking libgen.h presence... yes 2024-03-08T23:10:05.315 INFO:tasks.workunit.client.0.smithi046.stdout:checking for libgen.h... yes 2024-03-08T23:10:05.433 INFO:tasks.workunit.client.0.smithi046.stdout:checking dirent.h usability... yes 2024-03-08T23:10:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:checking dirent.h presence... yes 2024-03-08T23:10:05.473 INFO:tasks.workunit.client.0.smithi046.stdout:checking for dirent.h... yes 2024-03-08T23:10:05.582 INFO:tasks.workunit.client.0.smithi046.stdout:checking errno.h usability... yes 2024-03-08T23:10:05.619 INFO:tasks.workunit.client.0.smithi046.stdout:checking errno.h presence... yes 2024-03-08T23:10:05.619 INFO:tasks.workunit.client.0.smithi046.stdout:checking for errno.h... yes 2024-03-08T23:10:05.735 INFO:tasks.workunit.client.0.smithi046.stdout:checking malloc.h usability... yes 2024-03-08T23:10:05.776 INFO:tasks.workunit.client.0.smithi046.stdout:checking malloc.h presence... yes 2024-03-08T23:10:05.776 INFO:tasks.workunit.client.0.smithi046.stdout:checking for malloc.h... yes 2024-03-08T23:10:05.890 INFO:tasks.workunit.client.0.smithi046.stdout:checking uuid.h usability... no 2024-03-08T23:10:05.953 INFO:tasks.workunit.client.0.smithi046.stdout:checking uuid.h presence... no 2024-03-08T23:10:05.953 INFO:tasks.workunit.client.0.smithi046.stdout:checking for uuid.h... no 2024-03-08T23:10:06.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:05 smithi192 ceph-mon[36700]: pgmap v72: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:06.065 INFO:tasks.workunit.client.0.smithi046.stdout:checking uuid/uuid.h usability... yes 2024-03-08T23:10:06.110 INFO:tasks.workunit.client.0.smithi046.stdout:checking uuid/uuid.h presence... yes 2024-03-08T23:10:06.110 INFO:tasks.workunit.client.0.smithi046.stdout:checking for uuid/uuid.h... yes 2024-03-08T23:10:06.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:05 smithi062 ceph-mon[35761]: pgmap v72: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:06.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:05 smithi046 ceph-mon[31859]: pgmap v72: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:06.226 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/uuid.h usability... no 2024-03-08T23:10:06.289 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/uuid.h presence... no 2024-03-08T23:10:06.290 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/uuid.h... no 2024-03-08T23:10:06.384 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/file.h usability... yes 2024-03-08T23:10:06.424 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/file.h presence... yes 2024-03-08T23:10:06.424 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/file.h... yes 2024-03-08T23:10:06.541 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fcntl.h usability... yes 2024-03-08T23:10:06.582 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fcntl.h presence... yes 2024-03-08T23:10:06.582 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/fcntl.h... yes 2024-03-08T23:10:06.698 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/syssgi.h usability... no 2024-03-08T23:10:06.763 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/syssgi.h presence... no 2024-03-08T23:10:06.763 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/syssgi.h... no 2024-03-08T23:10:07.371 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/param.h usability... yes 2024-03-08T23:10:07.419 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/param.h presence... yes 2024-03-08T23:10:07.419 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/param.h... yes 2024-03-08T23:10:07.453 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/stat.h... (cached) yes 2024-03-08T23:10:07.565 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/statvfs.h usability... yes 2024-03-08T23:10:07.606 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/statvfs.h presence... yes 2024-03-08T23:10:07.606 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/statvfs.h... yes 2024-03-08T23:10:07.716 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/time.h usability... yes 2024-03-08T23:10:07.757 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/time.h presence... yes 2024-03-08T23:10:07.757 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/time.h... yes 2024-03-08T23:10:07.869 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/ioctl.h usability... yes 2024-03-08T23:10:07.911 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/ioctl.h presence... yes 2024-03-08T23:10:07.911 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/ioctl.h... yes 2024-03-08T23:10:08.033 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/wait.h usability... yes 2024-03-08T23:10:08.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:07 smithi192 ceph-mon[36700]: pgmap v73: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:08.076 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/wait.h presence... yes 2024-03-08T23:10:08.077 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/wait.h... yes 2024-03-08T23:10:08.112 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/types.h... (cached) yes 2024-03-08T23:10:08.132 INFO:tasks.workunit.client.0.smithi046.stdout:checking for strings.h... (cached) yes 2024-03-08T23:10:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:07 smithi062 ceph-mon[35761]: pgmap v73: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:08.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:07 smithi046 ceph-mon[31859]: pgmap v73: 129 pgs: 129 active+clean; 66 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 1.6 MiB/s wr, 282 op/s 2024-03-08T23:10:08.226 INFO:tasks.workunit.client.0.smithi046.stdout:checking err.h usability... yes 2024-03-08T23:10:08.266 INFO:tasks.workunit.client.0.smithi046.stdout:checking err.h presence... yes 2024-03-08T23:10:08.267 INFO:tasks.workunit.client.0.smithi046.stdout:checking for err.h... yes 2024-03-08T23:10:08.375 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/falloc.h usability... yes 2024-03-08T23:10:08.414 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/falloc.h presence... yes 2024-03-08T23:10:08.414 INFO:tasks.workunit.client.0.smithi046.stdout:checking for linux/falloc.h... yes 2024-03-08T23:10:08.616 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-08T23:10:08.677 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-08T23:10:08.677 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-08T23:10:08.777 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-08T23:10:08.849 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-08T23:10:08.849 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-08T23:10:08.946 INFO:tasks.workunit.client.0.smithi046.stdout:checking xfs/platform_defs.h usability... no 2024-03-08T23:10:09.013 INFO:tasks.workunit.client.0.smithi046.stdout:checking xfs/platform_defs.h presence... no 2024-03-08T23:10:09.014 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/platform_defs.h... no 2024-03-08T23:10:09.115 INFO:tasks.workunit.client.0.smithi046.stdout:checking btrfs/ioctl.h usability... no 2024-03-08T23:10:09.178 INFO:tasks.workunit.client.0.smithi046.stdout:checking btrfs/ioctl.h presence... no 2024-03-08T23:10:09.178 INFO:tasks.workunit.client.0.smithi046.stdout:checking for btrfs/ioctl.h... no 2024-03-08T23:10:09.286 INFO:tasks.workunit.client.0.smithi046.stdout:checking cifs/ioctl.h usability... no 2024-03-08T23:10:09.356 INFO:tasks.workunit.client.0.smithi046.stdout:checking cifs/ioctl.h presence... no 2024-03-08T23:10:09.356 INFO:tasks.workunit.client.0.smithi046.stdout:checking for cifs/ioctl.h... no 2024-03-08T23:10:09.451 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/mman.h usability... yes 2024-03-08T23:10:09.492 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/mman.h presence... yes 2024-03-08T23:10:09.492 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/mman.h... yes 2024-03-08T23:10:09.577 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/xfs_log_format.h... no 2024-03-08T23:10:09.707 INFO:tasks.workunit.client.0.smithi046.stdout:checking for uuid_compare... no 2024-03-08T23:10:09.806 INFO:tasks.workunit.client.0.smithi046.stdout:checking for uuid_compare in -luuid... yes 2024-03-08T23:10:09.933 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/xfs.h... yes 2024-03-08T23:10:10.006 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/libxfs.h... no 2024-03-08T23:10:10.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:09 smithi192 ceph-mon[36700]: pgmap v74: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-08T23:10:10.082 INFO:tasks.workunit.client.0.smithi046.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-08T23:10:10.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:09 smithi062 ceph-mon[35761]: pgmap v74: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-08T23:10:10.173 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/xqm.h... yes 2024-03-08T23:10:10.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:09 smithi046 ceph-mon[31859]: pgmap v74: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-03-08T23:10:10.299 INFO:tasks.workunit.client.0.smithi046.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-08T23:10:10.387 INFO:tasks.workunit.client.0.smithi046.stdout:checking xfs/handle.h usability... yes 2024-03-08T23:10:10.427 INFO:tasks.workunit.client.0.smithi046.stdout:checking xfs/handle.h presence... yes 2024-03-08T23:10:10.427 INFO:tasks.workunit.client.0.smithi046.stdout:checking for xfs/handle.h... yes 2024-03-08T23:10:10.544 INFO:tasks.workunit.client.0.smithi046.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-08T23:10:10.679 INFO:tasks.workunit.client.0.smithi046.stdout:checking attr/attributes.h usability... yes 2024-03-08T23:10:10.721 INFO:tasks.workunit.client.0.smithi046.stdout:checking attr/attributes.h presence... yes 2024-03-08T23:10:10.721 INFO:tasks.workunit.client.0.smithi046.stdout:checking for attr/attributes.h... yes 2024-03-08T23:10:10.839 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/attributes.h usability... no 2024-03-08T23:10:10.903 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/attributes.h presence... no 2024-03-08T23:10:10.903 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/attributes.h... no 2024-03-08T23:10:11.005 INFO:tasks.workunit.client.0.smithi046.stdout:checking for attr_list in -lattr... yes 2024-03-08T23:10:11.100 INFO:tasks.workunit.client.0.smithi046.stdout:checking for attr_set in -lattr... yes 2024-03-08T23:10:11.176 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/acl.h usability... yes 2024-03-08T23:10:11.217 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/acl.h presence... yes 2024-03-08T23:10:11.217 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/acl.h... yes 2024-03-08T23:10:11.295 INFO:tasks.workunit.client.0.smithi046.stdout:checking acl/libacl.h usability... yes 2024-03-08T23:10:11.337 INFO:tasks.workunit.client.0.smithi046.stdout:checking acl/libacl.h presence... yes 2024-03-08T23:10:11.338 INFO:tasks.workunit.client.0.smithi046.stdout:checking for acl/libacl.h... yes 2024-03-08T23:10:11.443 INFO:tasks.workunit.client.0.smithi046.stdout:checking for acl_init in -lacl... yes 2024-03-08T23:10:11.655 INFO:tasks.workunit.client.0.smithi046.stdout:checking gdbm-ndbm.h usability... no 2024-03-08T23:10:11.726 INFO:tasks.workunit.client.0.smithi046.stdout:checking gdbm-ndbm.h presence... no 2024-03-08T23:10:11.726 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gdbm-ndbm.h... no 2024-03-08T23:10:11.813 INFO:tasks.workunit.client.0.smithi046.stdout:checking gdbm/ndbm.h usability... no 2024-03-08T23:10:11.880 INFO:tasks.workunit.client.0.smithi046.stdout:checking gdbm/ndbm.h presence... no 2024-03-08T23:10:11.880 INFO:tasks.workunit.client.0.smithi046.stdout:checking for gdbm/ndbm.h... no 2024-03-08T23:10:11.963 INFO:tasks.workunit.client.0.smithi046.stdout:checking ndbm.h usability... no 2024-03-08T23:10:12.025 INFO:tasks.workunit.client.0.smithi046.stdout:checking ndbm.h presence... no 2024-03-08T23:10:12.026 INFO:tasks.workunit.client.0.smithi046.stdout:checking for ndbm.h... no 2024-03-08T23:10:12.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:11 smithi192 ceph-mon[36700]: pgmap v75: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 859 KiB/s wr, 96 op/s 2024-03-08T23:10:12.110 INFO:tasks.workunit.client.0.smithi046.stdout:checking libaio.h usability... yes 2024-03-08T23:10:12.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:11 smithi062 ceph-mon[35761]: pgmap v75: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 859 KiB/s wr, 96 op/s 2024-03-08T23:10:12.158 INFO:tasks.workunit.client.0.smithi046.stdout:checking libaio.h presence... yes 2024-03-08T23:10:12.158 INFO:tasks.workunit.client.0.smithi046.stdout:checking for libaio.h... yes 2024-03-08T23:10:12.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:11 smithi046 ceph-mon[31859]: pgmap v75: 129 pgs: 129 active+clean; 67 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 859 KiB/s wr, 96 op/s 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout:checking for dm_make_handle in -ldm... no 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout:If you want DMAPI support please install the data migration 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout:from the dmapi source. 2024-03-08T23:10:12.279 INFO:tasks.workunit.client.0.smithi046.stdout: 2024-03-08T23:10:12.356 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/fiemap.h usability... yes 2024-03-08T23:10:12.397 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/fiemap.h presence... yes 2024-03-08T23:10:12.397 INFO:tasks.workunit.client.0.smithi046.stdout:checking for linux/fiemap.h... yes 2024-03-08T23:10:12.501 INFO:tasks.workunit.client.0.smithi046.stdout:checking for fallocate... yes 2024-03-08T23:10:12.610 INFO:tasks.workunit.client.0.smithi046.stdout:checking for open_by_handle_at... yes 2024-03-08T23:10:12.692 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/prctl.h usability... yes 2024-03-08T23:10:12.730 INFO:tasks.workunit.client.0.smithi046.stdout:checking sys/prctl.h presence... yes 2024-03-08T23:10:12.730 INFO:tasks.workunit.client.0.smithi046.stdout:checking for sys/prctl.h... yes 2024-03-08T23:10:12.809 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/fs.h usability... yes 2024-03-08T23:10:12.848 INFO:tasks.workunit.client.0.smithi046.stdout:checking linux/fs.h presence... yes 2024-03-08T23:10:12.848 INFO:tasks.workunit.client.0.smithi046.stdout:checking for linux/fs.h... yes 2024-03-08T23:10:12.954 INFO:tasks.workunit.client.0.smithi046.stdout:checking btrfsutil.h usability... no 2024-03-08T23:10:13.014 INFO:tasks.workunit.client.0.smithi046.stdout:checking btrfsutil.h presence... no 2024-03-08T23:10:13.014 INFO:tasks.workunit.client.0.smithi046.stdout:checking for btrfsutil.h... no 2024-03-08T23:10:13.115 INFO:tasks.workunit.client.0.smithi046.stdout:checking for copy_file_range... yes 2024-03-08T23:10:13.220 INFO:tasks.workunit.client.0.smithi046.stdout:checking for renameat2... yes 2024-03-08T23:10:13.266 INFO:tasks.workunit.client.0.smithi046.stdout:configure: creating ./config.status 2024-03-08T23:10:14.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:13 smithi192 ceph-mon[36700]: pgmap v76: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1015 KiB/s wr, 106 op/s 2024-03-08T23:10:14.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:13 smithi062 ceph-mon[35761]: pgmap v76: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1015 KiB/s wr, 106 op/s 2024-03-08T23:10:14.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:13 smithi046 ceph-mon[31859]: pgmap v76: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 1015 KiB/s wr, 106 op/s 2024-03-08T23:10:14.639 INFO:tasks.workunit.client.0.smithi046.stdout:config.status: creating include/builddefs 2024-03-08T23:10:16.028 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:15 smithi046 ceph-mon[31859]: pgmap v77: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:16.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:15 smithi192 ceph-mon[36700]: pgmap v77: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:16.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:15 smithi062 ceph-mon[35761]: pgmap v77: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:16.849 INFO:tasks.workunit.client.0.smithi046.stdout:config.status: creating include/config.h 2024-03-08T23:10:17.379 INFO:tasks.workunit.client.0.smithi046.stdout:config.status: executing libtool commands 2024-03-08T23:10:17.601 INFO:tasks.workunit.client.0.smithi046.stdout:Building include 2024-03-08T23:10:17.601 INFO:tasks.workunit.client.0.smithi046.stdout:Building lib 2024-03-08T23:10:17.603 INFO:tasks.workunit.client.0.smithi046.stdout:Building tests 2024-03-08T23:10:17.634 INFO:tasks.workunit.client.0.smithi046.stdout: [LTDEP] 2024-03-08T23:10:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:17 smithi192 ceph-mon[36700]: pgmap v78: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:18.119 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] dataascii.lo 2024-03-08T23:10:18.119 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] databin.lo 2024-03-08T23:10:18.120 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] datapid.lo 2024-03-08T23:10:18.121 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] file_lock.lo 2024-03-08T23:10:18.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:17 smithi062 ceph-mon[35761]: pgmap v78: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:18.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:17 smithi046 ceph-mon[31859]: pgmap v78: 129 pgs: 129 active+clean; 69 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 295 KiB/s wr, 20 op/s 2024-03-08T23:10:18.414 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] pattern.lo 2024-03-08T23:10:18.414 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] forker.lo 2024-03-08T23:10:18.447 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] open_flags.lo 2024-03-08T23:10:18.513 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] random_range.lo 2024-03-08T23:10:18.713 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] string_to_tokens.lo 2024-03-08T23:10:18.737 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] str_to_bytes.lo 2024-03-08T23:10:18.758 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] tlibio.lo 2024-03-08T23:10:18.949 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] write_log.lo 2024-03-08T23:10:18.978 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] random.lo 2024-03-08T23:10:19.393 INFO:tasks.workunit.client.0.smithi046.stdout: [LD] 2024-03-08T23:10:19.666 INFO:tasks.workunit.client.0.smithi046.stdout:Building ltp 2024-03-08T23:10:19.666 INFO:tasks.workunit.client.0.smithi046.stdout:Building src 2024-03-08T23:10:19.667 INFO:tasks.workunit.client.0.smithi046.stdout:Building m4 2024-03-08T23:10:19.669 INFO:tasks.workunit.client.0.smithi046.stdout:Building common 2024-03-08T23:10:19.681 INFO:tasks.workunit.client.0.smithi046.stdout: [DEP] 2024-03-08T23:10:19.740 INFO:tasks.workunit.client.0.smithi046.stdout: [DEP] 2024-03-08T23:10:20.031 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:10:20.032 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:10:20.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:19 smithi192 ceph-mon[36700]: pgmap v79: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 770 KiB/s wr, 58 op/s 2024-03-08T23:10:20.110 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:19 smithi046 ceph-mon[31859]: pgmap v79: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 770 KiB/s wr, 58 op/s 2024-03-08T23:10:20.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:19 smithi062 ceph-mon[35761]: pgmap v79: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 770 KiB/s wr, 58 op/s 2024-03-08T23:10:20.223 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] doio 2024-03-08T23:10:20.223 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fsstress 2024-03-08T23:10:20.223 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fsx 2024-03-08T23:10:20.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.257+0000 7f3a66a00640 1 -- 172.21.15.46:0/823006494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 msgr2=0x7f3a6010ff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.257+0000 7f3a66a00640 1 --2- 172.21.15.46:0/823006494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a6010ff30 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009ca0 tx=0x7f3a5802c190 comp rx=0 tx=0).stop 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 -- 172.21.15.46:0/823006494 shutdown_connections 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 --2- 172.21.15.46:0/823006494 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a6010ff30 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 --2- 172.21.15.46:0/823006494 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 0x7f3a6010d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 --2- 172.21.15.46:0/823006494 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 -- 172.21.15.46:0/823006494 >> 172.21.15.46:0/823006494 conn(0x7f3a600786f0 msgr2=0x7f3a60078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.258+0000 7f3a66a00640 1 -- 172.21.15.46:0/823006494 shutdown_connections 2024-03-08T23:10:20.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.259+0000 7f3a66a00640 1 -- 172.21.15.46:0/823006494 wait complete. 2024-03-08T23:10:20.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.259+0000 7f3a66a00640 1 Processor -- start 2024-03-08T23:10:20.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 -- start start 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 0x7f3a60101780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a60101cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a601138c0 con 0x7f3a6010db20 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60113740 con 0x7f3a600736c0 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.260+0000 7f3a66a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60113a40 con 0x7f3a60073fe0 2024-03-08T23:10:20.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53680/0 (socket says 172.21.15.46:53680) 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 -- 172.21.15.46:0/1194567924 learned_addr learned my addr 172.21.15.46:0/1194567924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a64a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a60101cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5f400640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 0x7f3a60101780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 -- 172.21.15.46:0/1194567924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 msgr2=0x7f3a60101780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 0x7f3a60101780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 -- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 msgr2=0x7f3a60101cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a60101cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.261+0000 7f3a5fe00640 1 -- 172.21.15.46:0/1194567924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a601aa490 con 0x7f3a600736c0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a5fe00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000d980 tx=0x7f3a5000de40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a5cc00640 1 -- 172.21.15.46:0/1194567924 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50017070 con 0x7f3a600736c0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a601aa780 con 0x7f3a600736c0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a601aad20 con 0x7f3a600736c0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a5cc00640 1 -- 172.21.15.46:0/1194567924 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a50009c30 con 0x7f3a600736c0 2024-03-08T23:10:20.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.262+0000 7f3a5cc00640 1 -- 172.21.15.46:0/1194567924 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50020060 con 0x7f3a600736c0 2024-03-08T23:10:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.267+0000 7f3a69767740 1 Processor -- start 2024-03-08T23:10:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.267+0000 7f3a69767740 1 -- start start 2024-03-08T23:10:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.267+0000 7f3a5cc00640 1 -- 172.21.15.46:0/1194567924 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3a5000f620 con 0x7f3a600736c0 2024-03-08T23:10:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.267+0000 7f3a69767740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a69767740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a69767740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e5890f8e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a69767740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5890af410 con 0x55e5890edea0 2024-03-08T23:10:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a69767740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e589051d50 con 0x55e5890f6a80 2024-03-08T23:10:20.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a69767740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e588ffab80 con 0x55e5890ed580 2024-03-08T23:10:20.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a5fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53014/0 (socket says 172.21.15.46:53014) 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.268+0000 7f3a64a00640 1 -- 172.21.15.46:0/4095546998 learned_addr learned my addr 172.21.15.46:0/4095546998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 -- 172.21.15.46:0/4095546998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 msgr2=0x55e5890f8e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e5890f8e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 -- 172.21.15.46:0/4095546998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 msgr2=0x55e5890f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 -- 172.21.15.46:0/4095546998 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e588e307d0 con 0x55e5890ed580 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a64a00640 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3a58005e30 tx=0x7f3a58005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2daee8a5b5cea40 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a55200640 1 -- 172.21.15.46:0/4095546998 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58043070 con 0x55e5890ed580 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a55200640 1 -- 172.21.15.46:0/4095546998 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a58003e50 con 0x55e5890ed580 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.269+0000 7f3a55200640 1 -- 172.21.15.46:0/4095546998 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58039040 con 0x55e5890ed580 2024-03-08T23:10:20.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.270+0000 7f3a5cc00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 0x7f3a440a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.270+0000 7f3a5cc00640 1 -- 172.21.15.46:0/1194567924 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3a500cea00 con 0x7f3a600736c0 2024-03-08T23:10:20.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.270+0000 7f3a5fe00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 0x7f3a440a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 -- 172.21.15.46:0/4095546998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 msgr2=0x55e5890ed960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3a58005e30 tx=0x7f3a58005fd0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a5fe00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 0x7f3a440a4f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55e5890ed050 tx=0x7f3a500d4530 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 -- 172.21.15.46:0/4095546998 shutdown_connections 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e5890f8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 --2- 172.21.15.46:0/4095546998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890ed580 0x55e5890ed960 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 -- 172.21.15.46:0/4095546998 >> 172.21.15.46:0/4095546998 conn(0x55e5890e7cb0 msgr2=0x55e5890e8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 -- 172.21.15.46:0/4095546998 shutdown_connections 2024-03-08T23:10:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.272+0000 7f3a69767740 1 -- 172.21.15.46:0/4095546998 wait complete. 2024-03-08T23:10:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.273+0000 7f3a69767740 1 -- start start 2024-03-08T23:10:20.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.273+0000 7f3a69767740 1 Processor -- start 2024-03-08T23:10:20.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890f44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e58900dee0 con 0x55e5890edea0 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e58904ddf0 con 0x55e5890ed580 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5890f9810 con 0x55e5890f6a80 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a69767740 10 client.0 _ll_register_callbacks cb 0x55e5890148d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53694/0 (socket says 172.21.15.46:53694) 2024-03-08T23:10:20.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a64a00640 1 -- 172.21.15.46:0/3649549365 learned_addr learned my addr 172.21.15.46:0/3649549365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a64a00640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890f44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.274+0000 7f3a5f400640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 -- 172.21.15.46:0/3649549365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 msgr2=0x55e5890f44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890f44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 -- 172.21.15.46:0/3649549365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 msgr2=0x55e5890f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 -- 172.21.15.46:0/3649549365 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e588d663f0 con 0x55e5890ed580 2024-03-08T23:10:20.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a64a00640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009bf0 tx=0x7f3a5802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19a6c226bf8e703d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.275+0000 7f3a49600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58039070 con 0x55e5890ed580 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3a58004040 con 0x55e5890ed580 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a58003e50 con 0x55e5890ed580 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a58062040 con 0x55e5890ed580 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a69767740 10 client.26102 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:20.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.276+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e588d260f0 con 0x55e5890ed580 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a58003e50 con 0x55e5890ed580 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3a58048a60 con 0x55e5890ed580 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 10 client.26102 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 10 client.26102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 20 client.26102 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e589106af0 0x55e589108ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 4 client.26102 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.277+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e58905f290 con 0x55e589106af0 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.278+0000 7f3a5f400640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e589106af0 0x55e589108ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.278+0000 7f3a5f400640 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e589106af0 0x55e589108ee0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cea5a0447cfb8b5 server_cookie=8b8655dc12c6c041 in_seq=0 out_seq=0 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.278+0000 7f3a49600640 10 client.26102 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:20.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.279+0000 7f3a49600640 1 -- 172.21.15.46:0/3649549365 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e58905f290 con 0x55e589106af0 2024-03-08T23:10:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.279+0000 7f3a49600640 10 client.26102 handle_command_reply: tid=0 2024-03-08T23:10:20.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.279+0000 7f3a69767740 1 client.26102 shutdown 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 msgr2=0x55e5890f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009bf0 tx=0x7f3a5802e040 comp rx=0 tx=0).stop 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 shutdown_connections 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e589106af0 0x55e589108ee0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890f44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e5890f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 --2- 172.21.15.46:0/3649549365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f3a60 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 >> 172.21.15.46:0/3649549365 conn(0x55e5890e7cb0 msgr2=0x55e5890e9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a49600640 0 client.26102 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 shutdown_connections 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 1 -- 172.21.15.46:0/3649549365 wait complete. 2024-03-08T23:10:20.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.280+0000 7f3a69767740 20 client.26102 trim_cache size 0 max 16384 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 Processor -- start 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 -- start start 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e58910d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890fdae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40012170 con 0x55e5890edea0 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000fce0 con 0x55e5890ed580 2024-03-08T23:10:20.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.304+0000 7f3a69767740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000dd80 con 0x55e5890f6a80 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53706/0 (socket says 172.21.15.46:53706) 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 -- 172.21.15.46:0/1541504075 learned_addr learned my addr 172.21.15.46:0/1541504075 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890fdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a5f400640 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e58910d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 -- 172.21.15.46:0/1541504075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 msgr2=0x55e5890fdae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.309 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890fdae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 -- 172.21.15.46:0/1541504075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 msgr2=0x55e58910d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e58910d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.305+0000 7f3a64a00640 1 -- 172.21.15.46:0/1541504075 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5890edcd0 con 0x55e5890ed580 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.306+0000 7f3a64a00640 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009a60 tx=0x7f3a58005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c70618fea690d1b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.307+0000 7f3a4be00640 1 -- 172.21.15.46:0/1541504075 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58039070 con 0x55e5890ed580 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.307+0000 7f3a4be00640 1 -- 172.21.15.46:0/1541504075 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a580345e0 con 0x55e5890ed580 2024-03-08T23:10:20.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.307+0000 7f3a4be00640 1 -- 172.21.15.46:0/1541504075 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a58043040 con 0x55e5890ed580 2024-03-08T23:10:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 -- 172.21.15.46:0/1541504075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 msgr2=0x55e5890f6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a58009a60 tx=0x7f3a58005d80 comp rx=0 tx=0).stop 2024-03-08T23:10:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 -- 172.21.15.46:0/1541504075 shutdown_connections 2024-03-08T23:10:20.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890f6a80 0x55e5890fdae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890edea0 0x55e58910d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 --2- 172.21.15.46:0/1541504075 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890ed580 0x55e5890f6190 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 -- 172.21.15.46:0/1541504075 >> 172.21.15.46:0/1541504075 conn(0x55e5890e7cb0 msgr2=0x55e5890e8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 -- 172.21.15.46:0/1541504075 shutdown_connections 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.308+0000 7f3a69767740 1 -- 172.21.15.46:0/1541504075 wait complete. 2024-03-08T23:10:20.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.309+0000 7f3a69767740 1 -- start start 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.309+0000 7f3a69767740 1 Processor -- start 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890ed580 0x55e5891550a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e589150c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000e380 con 0x55e5890ed580 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000f680 con 0x55e5890f6a80 2024-03-08T23:10:20.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4000e9c0 con 0x55e5890edea0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a69767740 10 client.0 _ll_register_callbacks cb 0x7f3a58009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a5f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a5f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53030/0 (socket says 172.21.15.46:53030) 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.310+0000 7f3a5f400640 1 -- 172.21.15.46:0/4125856901 learned_addr learned my addr 172.21.15.46:0/4125856901 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 -- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 msgr2=0x55e589150c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e589150c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 -- 172.21.15.46:0/4125856901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890ed580 msgr2=0x55e5891550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890ed580 0x55e5891550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 -- 172.21.15.46:0/4125856901 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e589151490 con 0x55e5890edea0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a5f400640 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c000c00 tx=0x7f3a4c00bb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba263ce060374687 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.311+0000 7f3a49600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.312+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c026060 con 0x55e5890edea0 2024-03-08T23:10:20.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.312+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3a4c00f040 con 0x55e5890edea0 2024-03-08T23:10:20.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.312+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a580345e0 con 0x55e5890edea0 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.313+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a4c018e60 con 0x55e5890edea0 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.313+0000 7f3a69767740 10 client.26119 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.313+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55e58915d8d0 con 0x55e5890edea0 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.313+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a580345e0 con 0x55e5890edea0 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f3a4c021410 con 0x55e5890edea0 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 10 client.26119 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 10 client.26119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 20 client.26119 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:20.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e58915dff0 0x55e589160410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 4 client.26119 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:10:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.314+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e5890595e0 con 0x55e58915dff0 2024-03-08T23:10:20.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.315+0000 7f3a64a00640 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e58915dff0 0x55e589160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.316+0000 7f3a64a00640 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e58915dff0 0x55e589160410 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eae3f3b68ba0a69 server_cookie=a2c26b4e891ee455 in_seq=0 out_seq=0 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.316+0000 7f3a49600640 10 client.26119 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.316+0000 7f3a49600640 1 -- 172.21.15.46:0/4125856901 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e5890595e0 con 0x55e58915dff0 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.316+0000 7f3a49600640 10 client.26119 handle_command_reply: tid=0 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.316+0000 7f3a69767740 1 client.26119 shutdown 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 msgr2=0x55e5891555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c000c00 tx=0x7f3a4c00bb90 comp rx=0 tx=0).stop 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 shutdown_connections 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e58915dff0 0x55e589160410 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e5890f6a80 0x55e589150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e5890edea0 0x55e5891555e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 --2- 172.21.15.46:0/4125856901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e5890ed580 0x55e5891550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 >> 172.21.15.46:0/4125856901 conn(0x55e5890e7cb0 msgr2=0x55e58910d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a49600640 0 client.26119 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 shutdown_connections 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 1 -- 172.21.15.46:0/4125856901 wait complete. 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.317+0000 7f3a69767740 20 client.26119 trim_cache size 0 max 16384 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:10:20.321 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 msgr2=0x7f3a440a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 0x7f3a440a4f30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x55e5890ed050 tx=0x7f3a500d4530 comp rx=0 tx=0).stop 2024-03-08T23:10:20.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 msgr2=0x7f3a60101240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3a5000d980 tx=0x7f3a5000de40 comp rx=0 tx=0).stop 2024-03-08T23:10:20.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 shutdown_connections 2024-03-08T23:10:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3a440a2a70 0x7f3a440a4f30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3a6010db20 0x7f3a60101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3a60073fe0 0x7f3a60101780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 --2- 172.21.15.46:0/1194567924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3a600736c0 0x7f3a60101240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 >> 172.21.15.46:0/1194567924 conn(0x7f3a600786f0 msgr2=0x7f3a600fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 shutdown_connections 2024-03-08T23:10:20.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.318+0000 7f3a66a00640 1 -- 172.21.15.46:0/1194567924 wait complete. 2024-03-08T23:10:20.343 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:10:20.344 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 -- 172.21.15.46:0/4301399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8103550 msgr2=0x7fcab8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 --2- 172.21.15.46:0/4301399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8103550 0x7fcab8103930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fcaa8009900 tx=0x7fcaa802bc60 comp rx=0 tx=0).stop 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 -- 172.21.15.46:0/4301399 shutdown_connections 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 --2- 172.21.15.46:0/4301399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8108620 0x7fcab81101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 --2- 172.21.15.46:0/4301399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103f00 0x7fcab8107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 --2- 172.21.15.46:0/4301399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8103550 0x7fcab8103930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 -- 172.21.15.46:0/4301399 >> 172.21.15.46:0/4301399 conn(0x7fcab80fed90 msgr2=0x7fcab81011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 -- 172.21.15.46:0/4301399 shutdown_connections 2024-03-08T23:10:20.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.562+0000 7fcab7e00640 1 -- 172.21.15.46:0/4301399 wait complete. 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.563+0000 7fcab7e00640 1 Processor -- start 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.563+0000 7fcab7e00640 1 -- start start 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 0x7fcab819f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8199a40 0x7fcab8199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8113700 con 0x7fcab8108620 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8113580 con 0x7fcab8103550 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcab8113880 con 0x7fcab8199a40 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49386/0 (socket says 172.21.15.46:49386) 2024-03-08T23:10:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab6000640 1 -- 172.21.15.46:0/3148369405 learned_addr learned my addr 172.21.15.46:0/3148369405 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab6a00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 0x7fcab819f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.564+0000 7fcab7400640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8199a40 0x7fcab8199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 -- 172.21.15.46:0/3148369405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8199a40 msgr2=0x7fcab8199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8199a40 0x7fcab8199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 -- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 msgr2=0x7fcab819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 -- 172.21.15.46:0/3148369405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab819a560 con 0x7fcab8103550 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6000640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:20.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.565+0000 7fcab6a00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 0x7fcab819f840 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcaa802c270 tx=0x7fcaa802ea10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.566+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8041020 con 0x7fcab8103550 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.566+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaa802ee10 con 0x7fcab8103550 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.566+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaa8004ef0 con 0x7fcab8103550 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.566+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcab819a7f0 con 0x7fcab8103550 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.566+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcab81a68e0 con 0x7fcab8103550 2024-03-08T23:10:20.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.568+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcab8104a20 con 0x7fcab8103550 2024-03-08T23:10:20.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.570+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcaa8044020 con 0x7fcab8103550 2024-03-08T23:10:20.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.571+0000 7fcaaf400640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 0x7fca9c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:20.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.571+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fcaa80f0660 con 0x7fcab8103550 2024-03-08T23:10:20.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.572+0000 7fcab6000640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 0x7fca9c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:20.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.573+0000 7fcab6000640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 0x7fca9c0a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcab8109b70 tx=0x7fcaa4005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:20.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.575+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcaa804b020 con 0x7fcab8103550 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c: In function 'attr_list_path': 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~ 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:In file included from ../src/global.h:25, 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr: from fsstress.c:12: 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~ 2024-03-08T23:10:20.613 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~ 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:In file included from ../src/global.h:25, 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: from fsstress.c:12: 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~ 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c: In function 'attr_set_path': 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~ 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr:In file included from ../src/global.h:25, 2024-03-08T23:10:20.614 INFO:tasks.workunit.client.0.smithi046.stderr: from fsstress.c:12: 2024-03-08T23:10:20.615 INFO:tasks.workunit.client.0.smithi046.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-08T23:10:20.615 INFO:tasks.workunit.client.0.smithi046.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-08T23:10:20.615 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~ 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c: In function 'allocsp_f': 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr: | XFS_IOC_RESVSP64 2024-03-08T23:10:20.626 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-08T23:10:20.646 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c: In function 'freesp_f': 2024-03-08T23:10:20.646 INFO:tasks.workunit.client.0.smithi046.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-08T23:10:20.646 INFO:tasks.workunit.client.0.smithi046.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-08T23:10:20.646 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-08T23:10:20.647 INFO:tasks.workunit.client.0.smithi046.stderr: | XFS_IOC_RESVSP64 2024-03-08T23:10:20.709 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-03-08T23:10:20.709 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-08T23:10:20.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.862+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcab810b880 con 0x7fcab8103550 2024-03-08T23:10:20.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.863+0000 7fcaaf400640 1 -- 172.21.15.46:0/3148369405 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+11338 (secure 0 0 0) 0x7fcaa80bd110 con 0x7fcab8103550 2024-03-08T23:10:20.865 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:10:20.866 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:10:18.830300+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:10:20.866 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 26 2024-03-08T23:10:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 msgr2=0x7fca9c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 0x7fca9c0a4eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fcab8109b70 tx=0x7fcaa4005f90 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 msgr2=0x7fcab819f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 0x7fcab819f840 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcaa802c270 tx=0x7fcaa802ea10 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 shutdown_connections 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fca9c0a29f0 0x7fca9c0a4eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcab8199a40 0x7fcab8199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcab8108620 0x7fcab819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 --2- 172.21.15.46:0/3148369405 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcab8103550 0x7fcab819f840 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 >> 172.21.15.46:0/3148369405 conn(0x7fcab80fed90 msgr2=0x7fcab8101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 shutdown_connections 2024-03-08T23:10:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:20.866+0000 7fcab7e00640 1 -- 172.21.15.46:0/3148369405 wait complete. 2024-03-08T23:10:20.888 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:20.888 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:10:21.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:20 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 -- 172.21.15.46:0/1792904704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e41064a0 msgr2=0x7ff4e410e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 --2- 172.21.15.46:0/1792904704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e41064a0 0x7ff4e410e070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009ca0 tx=0x7ff4dc02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 -- 172.21.15.46:0/1792904704 shutdown_connections 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 --2- 172.21.15.46:0/1792904704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e41064a0 0x7ff4e410e070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 --2- 172.21.15.46:0/1792904704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e4101d80 0x7ff4e4105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 --2- 172.21.15.46:0/1792904704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e41017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 -- 172.21.15.46:0/1792904704 >> 172.21.15.46:0/1792904704 conn(0x7ff4e40fcc90 msgr2=0x7ff4e40ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 -- 172.21.15.46:0/1792904704 shutdown_connections 2024-03-08T23:10:21.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.071+0000 7ff4eae00640 1 -- 172.21.15.46:0/1792904704 wait complete. 2024-03-08T23:10:21.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 Processor -- start 2024-03-08T23:10:21.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 -- start start 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e41064a0 0x7ff4e4073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 0x7ff4e41a5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e4111540 con 0x7ff4e41013d0 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41113c0 con 0x7ff4e4074070 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.073+0000 7ff4eae00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41116c0 con 0x7ff4e41064a0 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49400/0 (socket says 172.21.15.46:49400) 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 -- 172.21.15.46:0/2366214353 learned_addr learned my addr 172.21.15.46:0/2366214353 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9000640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e41064a0 0x7ff4e4073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4ea400640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 0x7ff4e41a5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 -- 172.21.15.46:0/2366214353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e41064a0 msgr2=0x7ff4e4073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e41064a0 0x7ff4e4073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 -- 172.21.15.46:0/2366214353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 msgr2=0x7ff4e41a5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 0x7ff4e41a5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4e9a00640 1 -- 172.21.15.46:0/2366214353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e41a62a0 con 0x7ff4e41013d0 2024-03-08T23:10:21.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.074+0000 7ff4ea400640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 0x7ff4e41a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800e9f0 tx=0x7ff4d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4e2000640 1 -- 172.21.15.46:0/2366214353 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d800cd80 con 0x7ff4e41013d0 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4e2000640 1 -- 172.21.15.46:0/2366214353 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d8010910 con 0x7ff4e41013d0 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e41a6590 con 0x7ff4e41013d0 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4e2000640 1 -- 172.21.15.46:0/2366214353 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8018990 con 0x7ff4e41013d0 2024-03-08T23:10:21.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.075+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e41a6b30 con 0x7ff4e41013d0 2024-03-08T23:10:21.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.081+0000 7ff4e2000640 1 -- 172.21.15.46:0/2366214353 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff4d8010ac0 con 0x7ff4e41013d0 2024-03-08T23:10:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 Processor -- start 2024-03-08T23:10:21.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 -- start start 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x56051699f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169aae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560516903dd0 con 0x56051699fe50 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605168acc70 con 0x5605169a8a30 2024-03-08T23:10:21.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.082+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605168ffe70 con 0x56051699f530 2024-03-08T23:10:21.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169aae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49404/0 (socket says 172.21.15.46:49404) 2024-03-08T23:10:21.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9a00640 1 -- 172.21.15.46:0/1046894330 learned_addr learned my addr 172.21.15.46:0/1046894330 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e2000640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 0x7ff4cc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e2000640 1 -- 172.21.15.46:0/2366214353 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4d80cf530 con 0x7ff4e41013d0 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 -- 172.21.15.46:0/1046894330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 msgr2=0x56051699f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x56051699f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 -- 172.21.15.46:0/1046894330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 msgr2=0x5605169a84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9000640 1 -- 172.21.15.46:0/1046894330 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605169662b0 con 0x5605169a8a30 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.083+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.084+0000 7ff4ea400640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 0x7ff4cc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.084+0000 7ff4e9000640 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169aae20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4069020 tx=0x7ff4d0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6669f2e5d21e94b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.084+0000 7ff4d6a00640 1 -- 172.21.15.46:0/1046894330 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0004050 con 0x5605169a8a30 2024-03-08T23:10:21.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.084+0000 7ff4d6a00640 1 -- 172.21.15.46:0/1046894330 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0005500 con 0x5605169a8a30 2024-03-08T23:10:21.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.084+0000 7ff4d6a00640 1 -- 172.21.15.46:0/1046894330 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0004d40 con 0x5605169a8a30 2024-03-08T23:10:21.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.085+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1046894330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 msgr2=0x5605169aae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.085+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169aae20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4069020 tx=0x7ff4d0011810 comp rx=0 tx=0).stop 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1046894330 shutdown_connections 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169aae20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169a84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1046894330 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x56051699f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1046894330 >> 172.21.15.46:0/1046894330 conn(0x560516999c60 msgr2=0x56051699a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ea400640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 0x7ff4cc0a4eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006010 tx=0x7ff4dc004a40 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1046894330 shutdown_connections 2024-03-08T23:10:21.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1046894330 wait complete. 2024-03-08T23:10:21.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.086+0000 7ff4ef1e2740 1 -- start start 2024-03-08T23:10:21.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 Processor -- start 2024-03-08T23:10:21.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a83d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605169ab820 con 0x56051699f530 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605168c00c0 con 0x56051699fe50 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.087+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5605169ab9a0 con 0x5605169a8a30 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4ef1e2740 10 client.0 _ll_register_callbacks cb 0x5605168faec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53750/0 (socket says 172.21.15.46:53750) 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 -- 172.21.15.46:0/4273860780 learned_addr learned my addr 172.21.15.46:0/4273860780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 -- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 msgr2=0x5605169a5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 -- 172.21.15.46:0/4273860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 msgr2=0x5605169a83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.088+0000 7ff4e9000640 1 -- 172.21.15.46:0/4273860780 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605166183f0 con 0x56051699fe50 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4e9000640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009730 tx=0x7ff4d0005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e60e096daee9176 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4cb400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0003a00 con 0x56051699fe50 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4d0003ba0 con 0x56051699fe50 2024-03-08T23:10:21.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4d0005500 con 0x56051699fe50 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4d0033050 con 0x56051699fe50 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4ef1e2740 10 client.26123 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.089+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5605166e27d0 con 0x56051699fe50 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.090+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0005500 con 0x56051699fe50 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.090+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff4d0004a80 con 0x56051699fe50 2024-03-08T23:10:21.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 10 client.26123 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 10 client.26123 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 20 client.26123 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5605169b8a60 0x5605169bae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 4 client.26123 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5605168d84f0 con 0x5605169b8a60 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.091+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5605169b8a60 0x5605169bae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.092+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5605169b8a60 0x5605169bae50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ec5acfb0cab219b server_cookie=68cff907b88e866c in_seq=0 out_seq=0 2024-03-08T23:10:21.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.092+0000 7ff4cb400640 10 client.26123 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.092+0000 7ff4cb400640 1 -- 172.21.15.46:0/4273860780 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5605168d84f0 con 0x5605169b8a60 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.092+0000 7ff4cb400640 10 client.26123 handle_command_reply: tid=0 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.092+0000 7ff4ef1e2740 1 client.26123 shutdown 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 msgr2=0x5605169a5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0009730 tx=0x7ff4d0005780 comp rx=0 tx=0).stop 2024-03-08T23:10:21.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 shutdown_connections 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5605169b8a60 0x5605169bae50 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x5605169a5760 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/4273860780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4cb400640 0 client.26123 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 >> 172.21.15.46:0/4273860780 conn(0x560516999c60 msgr2=0x56051699b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 shutdown_connections 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/4273860780 wait complete. 2024-03-08T23:10:21.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.093+0000 7ff4ef1e2740 20 client.26123 trim_cache size 0 max 16384 2024-03-08T23:10:21.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.110+0000 7ff4ef1e2740 1 Processor -- start 2024-03-08T23:10:21.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.110+0000 7ff4ef1e2740 1 -- start start 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41b2a20 con 0x56051699f530 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41b0590 con 0x56051699fe50 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41ae630 con 0x5605169a8a30 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53752/0 (socket says 172.21.15.46:53752) 2024-03-08T23:10:21.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 -- 172.21.15.46:0/761360666 learned_addr learned my addr 172.21.15.46:0/761360666 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4ea400640 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 -- 172.21.15.46:0/761360666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 msgr2=0x5605169a7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 -- 172.21.15.46:0/761360666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 msgr2=0x5605169a48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.111+0000 7ff4e9000640 1 -- 172.21.15.46:0/761360666 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5605169ab700 con 0x56051699fe50 2024-03-08T23:10:21.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.112+0000 7ff4e9000640 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0011830 tx=0x7ff4d0005da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d291c64976b5ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.112+0000 7ff4d6000640 1 -- 172.21.15.46:0/761360666 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0004700 con 0x56051699fe50 2024-03-08T23:10:21.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.112+0000 7ff4d6000640 1 -- 172.21.15.46:0/761360666 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0004950 con 0x56051699fe50 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.112+0000 7ff4d6000640 1 -- 172.21.15.46:0/761360666 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0003d40 con 0x56051699fe50 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/761360666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 msgr2=0x560516998b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff4d0011830 tx=0x7ff4d0005da0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/761360666 shutdown_connections 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5605169a8a30 0x5605169a7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56051699fe50 0x560516998b60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/761360666 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699f530 0x5605169a48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/761360666 >> 172.21.15.46:0/761360666 conn(0x560516999c60 msgr2=0x560516998450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/761360666 shutdown_connections 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.113+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/761360666 wait complete. 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 -- start start 2024-03-08T23:10:21.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 Processor -- start 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x5605169afd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41aec30 con 0x56051699fe50 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41aff30 con 0x5605169a8a30 2024-03-08T23:10:21.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.114+0000 7ff4ef1e2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e41af270 con 0x56051699f530 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ef1e2740 10 client.0 _ll_register_callbacks cb 0x7ff4d00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53766/0 (socket says 172.21.15.46:53766) 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 -- 172.21.15.46:0/1254022556 learned_addr learned my addr 172.21.15.46:0/1254022556 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x5605169afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4e9000640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 -- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 msgr2=0x5605169afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x5605169afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 -- 172.21.15.46:0/1254022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 msgr2=0x5605169ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4ea400640 1 -- 172.21.15.46:0/1254022556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5605169aece0 con 0x5605169a8a30 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4e9000640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.115+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x5605169afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4ea400640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc004af0 tx=0x7ff4dc032590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fae8bed4a0f1fc14 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4cb400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc046040 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4dc003c30 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4d0004950 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4dc058050 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4ef1e2740 10 client.26129 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.116+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x560516a0fc70 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4d0004950 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff4dc049070 con 0x5605169a8a30 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 10 client.26129 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 10 client.26129 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 20 client.26129 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560516a103f0 0x560516a12810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 4 client.26129 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:21.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560516911310 con 0x560516a103f0 2024-03-08T23:10:21.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.117+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560516a103f0 0x560516a12810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.123 INFO:tasks.workunit.client.0.smithi046.stderr:fsx.c: In function 'main': 2024-03-08T23:10:21.123 INFO:tasks.workunit.client.0.smithi046.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-08T23:10:21.123 INFO:tasks.workunit.client.0.smithi046.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-08T23:10:21.123 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:21.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4cb400640 10 client.26129 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:21.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4e9a00640 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560516a103f0 0x560516a12810 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b47e0bafac6cf58f server_cookie=a8f5331c500ad497 in_seq=0 out_seq=0 2024-03-08T23:10:21.124 INFO:tasks.workunit.client.0.smithi046.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-08T23:10:21.124 INFO:tasks.workunit.client.0.smithi046.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-08T23:10:21.124 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:21.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4cb400640 1 -- 172.21.15.46:0/1254022556 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560516911310 con 0x560516a103f0 2024-03-08T23:10:21.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4cb400640 10 client.26129 handle_command_reply: tid=0 2024-03-08T23:10:21.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4ef1e2740 1 client.26129 shutdown 2024-03-08T23:10:21.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 msgr2=0x5605169ae6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.118+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc004af0 tx=0x7ff4dc032590 comp rx=0 tx=0).stop 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 shutdown_connections 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560516a103f0 0x560516a12810 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5605169a8a30 0x5605169ae6a0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56051699fe50 0x5605169ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 --2- 172.21.15.46:0/1254022556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56051699f530 0x5605169afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4cb400640 0 client.26129 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 >> 172.21.15.46:0/1254022556 conn(0x560516999c60 msgr2=0x560516998450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 shutdown_connections 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 1 -- 172.21.15.46:0/1254022556 wait complete. 2024-03-08T23:10:21.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.119+0000 7ff4ef1e2740 20 client.26129 trim_cache size 0 max 16384 2024-03-08T23:10:21.127 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:21.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 msgr2=0x7ff4cc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 0x7ff4cc0a4eb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc006010 tx=0x7ff4dc004a40 comp rx=0 tx=0).stop 2024-03-08T23:10:21.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 msgr2=0x7ff4e40735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800e9f0 tx=0x7ff4d800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.127 INFO:tasks.workunit.client.0.smithi046.stderr:doio.c: In function 'parse_cmdline': 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: 3113 | fprintf(stderr, 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~ 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: 3115 | Prog, TagName, tok); 2024-03-08T23:10:21.128 INFO:tasks.workunit.client.0.smithi046.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 shutdown_connections 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4cc0a29f0 0x7ff4cc0a4eb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4e4074070 0x7ff4e41a5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4e41064a0 0x7ff4e4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 --2- 172.21.15.46:0/2366214353 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4e41013d0 0x7ff4e40735f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 >> 172.21.15.46:0/2366214353 conn(0x7ff4e40fcc90 msgr2=0x7ff4e40ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 shutdown_connections 2024-03-08T23:10:21.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.120+0000 7ff4eae00640 1 -- 172.21.15.46:0/2366214353 wait complete. 2024-03-08T23:10:21.137 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:20 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:21.140 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:21.141 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:10:21.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:20 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:21.308 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-08T23:10:21.308 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-08T23:10:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 -- 172.21.15.46:0/1023249626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0103e70 msgr2=0x7f76c0110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 --2- 172.21.15.46:0/1023249626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0103e70 0x7f76c0110330 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009c10 tx=0x7f76b802bef0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 -- 172.21.15.46:0/1023249626 shutdown_connections 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 --2- 172.21.15.46:0/1023249626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0103e70 0x7f76c0110330 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 --2- 172.21.15.46:0/1023249626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c01034b0 0x7f76c0103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 --2- 172.21.15.46:0/1023249626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c0109570 0x7f76c0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.331+0000 7f76c5600640 1 -- 172.21.15.46:0/1023249626 >> 172.21.15.46:0/1023249626 conn(0x7f76c00fef60 msgr2=0x7f76c0101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.332+0000 7f76c5600640 1 -- 172.21.15.46:0/1023249626 shutdown_connections 2024-03-08T23:10:21.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.332+0000 7f76c5600640 1 -- 172.21.15.46:0/1023249626 wait complete. 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.332+0000 7f76c5600640 1 Processor -- start 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 -- start start 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 0x7f76c019f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 0x7f76c019fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 0x7f76c0199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0113880 con 0x7f76c0103e70 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0113700 con 0x7f76c0109570 2024-03-08T23:10:21.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76c5600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0113a00 con 0x7f76c01034b0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76be000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 0x7f76c019fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76be000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 0x7f76c019fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49450/0 (socket says 172.21.15.46:49450) 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76be000640 1 -- 172.21.15.46:0/3830303446 learned_addr learned my addr 172.21.15.46:0/3830303446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.333+0000 7f76bea00640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 0x7f76c019f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bf400640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 0x7f76c0199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 -- 172.21.15.46:0/3830303446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 msgr2=0x7f76c0199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 0x7f76c0199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 -- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 msgr2=0x7f76c019fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 0x7f76c019fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 -- 172.21.15.46:0/3830303446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c019a280 con 0x7f76c01034b0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bf400640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 0x7f76c0199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76bea00640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 0x7f76c019f950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f76b0005fd0 tx=0x7f76b000ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76b7400640 1 -- 172.21.15.46:0/3830303446 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b000cd00 con 0x7f76c01034b0 2024-03-08T23:10:21.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c019a570 con 0x7f76c01034b0 2024-03-08T23:10:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76b7400640 1 -- 172.21.15.46:0/3830303446 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76b00035f0 con 0x7f76c01034b0 2024-03-08T23:10:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.334+0000 7f76b7400640 1 -- 172.21.15.46:0/3830303446 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b0010690 con 0x7f76c01034b0 2024-03-08T23:10:21.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.335+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c01a6810 con 0x7f76c01034b0 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.337+0000 7f76c83b6740 1 Processor -- start 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 -- start start 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd9310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3c34dd0 con 0x5630c3cd0c70 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3bddc70 con 0x5630c3cd0350 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76c83b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3c30e70 con 0x5630c3cd9850 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76b7400640 1 -- 172.21.15.46:0/3830303446 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f76b00037a0 con 0x7f76c01034b0 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76be000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53786/0 (socket says 172.21.15.46:53786) 2024-03-08T23:10:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76bf400640 1 -- 172.21.15.46:0/4018080348 learned_addr learned my addr 172.21.15.46:0/4018080348 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.338+0000 7f76bea00640 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 -- 172.21.15.46:0/4018080348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 msgr2=0x5630c3cd0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 -- 172.21.15.46:0/4018080348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 msgr2=0x5630c3cd9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 -- 172.21.15.46:0/4018080348 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630c3c972b0 con 0x5630c3cd9850 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76bf400640 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76be000640 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdbc40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f76c019ac40 tx=0x7f76ac011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e9a17e912e15b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.339+0000 7f76ab400640 1 -- 172.21.15.46:0/4018080348 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac005020 con 0x5630c3cd9850 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76ab400640 1 -- 172.21.15.46:0/4018080348 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76ac0047a0 con 0x5630c3cd9850 2024-03-08T23:10:21.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76ab400640 1 -- 172.21.15.46:0/4018080348 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac019670 con 0x5630c3cd9850 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76b7400640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 0x7f76a40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76b7400640 1 -- 172.21.15.46:0/3830303446 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f76b00cf470 con 0x7f76c01034b0 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76bf400640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 0x7f76a40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 -- 172.21.15.46:0/4018080348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 msgr2=0x5630c3cdbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdbc40 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f76c019ac40 tx=0x7f76ac011810 comp rx=0 tx=0).stop 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 -- 172.21.15.46:0/4018080348 shutdown_connections 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdbc40 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 --2- 172.21.15.46:0/4018080348 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0350 0x5630c3cd0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.340+0000 7f76c83b6740 1 -- 172.21.15.46:0/4018080348 >> 172.21.15.46:0/4018080348 conn(0x5630c3ccaa80 msgr2=0x5630c3ccae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.341+0000 7f76c83b6740 1 -- 172.21.15.46:0/4018080348 shutdown_connections 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.341+0000 7f76c83b6740 1 -- 172.21.15.46:0/4018080348 wait complete. 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.341+0000 7f76bf400640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 0x7f76a40a4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76b8005f00 tx=0x7f76b8004a40 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.341+0000 7f76c83b6740 1 -- start start 2024-03-08T23:10:21.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 Processor -- start 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3cdc640 con 0x5630c3cd0c70 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3bf10c0 con 0x5630c3cd9850 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630c3cdc7c0 con 0x5630c3cd0350 2024-03-08T23:10:21.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76c83b6740 10 client.0 _ll_register_callbacks cb 0x5630c3c2bec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76bea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76bea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53150/0 (socket says 172.21.15.46:53150) 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76bea00640 1 -- 172.21.15.46:0/1335716366 learned_addr learned my addr 172.21.15.46:0/1335716366 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76be000640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.342+0000 7f76bf400640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 -- 172.21.15.46:0/1335716366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 msgr2=0x5630c3cd69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 -- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 msgr2=0x5630c3cd6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 -- 172.21.15.46:0/1335716366 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630c39493f0 con 0x5630c3cd0350 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76bea00640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76b00cc8c0 tx=0x7f76b000cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4295c38de25d3a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f769fe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:21.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b009b0d0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f76b0097de0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.343+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76ac0047a0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76b0009530 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 10 client.26149 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5630c3a137d0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76ac0047a0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f76b00ccae0 con 0x5630c3cd0350 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 10 client.26149 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 10 client.26149 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 20 client.26149 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3ce9840 0x5630c3cebc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 4 client.26149 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.344+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630c3c094f0 con 0x5630c3ce9840 2024-03-08T23:10:21.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.345+0000 7f76be000640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3ce9840 0x5630c3cebc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.345+0000 7f76be000640 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3ce9840 0x5630c3cebc30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a44c52b00c62ac98 server_cookie=3fb47c6345d3a3e8 in_seq=0 out_seq=0 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.345+0000 7f769fe00640 10 client.26149 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f769fe00640 1 -- 172.21.15.46:0/1335716366 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5630c3c094f0 con 0x5630c3ce9840 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f769fe00640 10 client.26149 handle_command_reply: tid=0 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 client.26149 shutdown 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 msgr2=0x5630c3cd90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f76b00cc8c0 tx=0x7f76b000cf60 comp rx=0 tx=0).stop 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 shutdown_connections 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd69c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3ce9840 0x5630c3cebc30 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 --2- 172.21.15.46:0/1335716366 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd90f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 >> 172.21.15.46:0/1335716366 conn(0x5630c3ccaa80 msgr2=0x5630c3ccc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f769fe00640 0 client.26149 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 shutdown_connections 2024-03-08T23:10:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.346+0000 7f76c83b6740 1 -- 172.21.15.46:0/1335716366 wait complete. 2024-03-08T23:10:21.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.347+0000 7f76c83b6740 20 client.26149 trim_cache size 0 max 16384 2024-03-08T23:10:21.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.359+0000 7f76c83b6740 1 Processor -- start 2024-03-08T23:10:21.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.359+0000 7f76c83b6740 1 -- start start 2024-03-08T23:10:21.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.359+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.359+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76c83b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01b2b60 con 0x5630c3cd0c70 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76c83b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01b06d0 con 0x5630c3cd9850 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76c83b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01ae770 con 0x5630c3cd0350 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53808/0 (socket says 172.21.15.46:53808) 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 -- 172.21.15.46:0/675560391 learned_addr learned my addr 172.21.15.46:0/675560391 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bea00640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76be000640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 -- 172.21.15.46:0/675560391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 msgr2=0x5630c3cd8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 -- 172.21.15.46:0/675560391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 msgr2=0x5630c3cd90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.360+0000 7f76bf400640 1 -- 172.21.15.46:0/675560391 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630c3cd49d0 con 0x5630c3cd9850 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76bea00640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76be000640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76bf400640 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036040 tx=0x7f76b8040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ee115773b436bf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76aaa00640 1 -- 172.21.15.46:0/675560391 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8051070 con 0x5630c3cd9850 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76aaa00640 1 -- 172.21.15.46:0/675560391 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76b802cd70 con 0x5630c3cd9850 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.361+0000 7f76aaa00640 1 -- 172.21.15.46:0/675560391 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b8004440 con 0x5630c3cd9850 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- 172.21.15.46:0/675560391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 msgr2=0x5630c3cd3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f76b8036040 tx=0x7f76b8040000 comp rx=0 tx=0).stop 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- 172.21.15.46:0/675560391 shutdown_connections 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd9850 0x5630c3cd3700 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0c70 0x5630c3cd90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 --2- 172.21.15.46:0/675560391 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd0350 0x5630c3cd8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- 172.21.15.46:0/675560391 >> 172.21.15.46:0/675560391 conn(0x5630c3ccaa80 msgr2=0x5630c3ccae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- 172.21.15.46:0/675560391 shutdown_connections 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- 172.21.15.46:0/675560391 wait complete. 2024-03-08T23:10:21.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.362+0000 7f76c83b6740 1 -- start start 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 Processor -- start 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 0x5630c3cdf8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01aed70 con 0x5630c3cd0350 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01b0070 con 0x5630c3cd0c70 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c01af3b0 con 0x5630c3cd9850 2024-03-08T23:10:21.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76c83b6740 10 client.0 _ll_register_callbacks cb 0x7f76ac009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49508/0 (socket says 172.21.15.46:49508) 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.363+0000 7f76bea00640 1 -- 172.21.15.46:0/3370857406 learned_addr learned my addr 172.21.15.46:0/3370857406 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 0x5630c3cdf8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76bf400640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 -- 172.21.15.46:0/3370857406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 msgr2=0x5630c3cdfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 -- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 msgr2=0x5630c3cdf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 -- 172.21.15.46:0/3370857406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630c3d3fc50 con 0x5630c3cd0c70 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76bea00640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f76be000640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 0x5630c3cdf8a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009560 tx=0x7f76ac009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=552fa78682819e5b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f769fe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:21.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac011d10 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.364+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f76ac004250 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76b802cd70 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76ac004450 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 10 client.26147 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x5630c3d40ef0 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76b802cd70 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f76ac02c420 con 0x5630c3cd0c70 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 10 client.26147 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 10 client.26147 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 20 client.26147 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3d414e0 0x5630c3d43900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 4 client.26147 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.365+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5630c3c42310 con 0x5630c3d414e0 2024-03-08T23:10:21.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.366+0000 7f76bea00640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3d414e0 0x5630c3d43900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.366+0000 7f76bea00640 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3d414e0 0x5630c3d43900 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=af8b2754f35c9ed3 server_cookie=cf8b3e863f420a1c in_seq=0 out_seq=0 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.366+0000 7f769fe00640 10 client.26147 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f769fe00640 1 -- 172.21.15.46:0/3370857406 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5630c3c42310 con 0x5630c3d414e0 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f769fe00640 10 client.26147 handle_command_reply: tid=0 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 client.26147 shutdown 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 msgr2=0x5630c3cdf8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 0x5630c3cdf8a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f76ac009560 tx=0x7f76ac009880 comp rx=0 tx=0).stop 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 shutdown_connections 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5630c3d414e0 0x5630c3d43900 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5630c3cd9850 0x5630c3cdfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5630c3cd0c70 0x5630c3cdf8a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 --2- 172.21.15.46:0/3370857406 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5630c3cd0350 0x5630c3cdf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 >> 172.21.15.46:0/3370857406 conn(0x5630c3ccaa80 msgr2=0x5630c3cd4d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f769fe00640 0 client.26147 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:21.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 shutdown_connections 2024-03-08T23:10:21.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 1 -- 172.21.15.46:0/3370857406 wait complete. 2024-03-08T23:10:21.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.367+0000 7f76c83b6740 20 client.26147 trim_cache size 0 max 16384 2024-03-08T23:10:21.374 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 msgr2=0x7f76a40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 0x7f76a40a4eb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f76b8005f00 tx=0x7f76b8004a40 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 msgr2=0x7f76c019f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 0x7f76c019f950 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f76b0005fd0 tx=0x7f76b000ee50 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 shutdown_connections 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76a40a29f0 0x7f76a40a4eb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f76c0109570 0x7f76c0199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f76c0103e70 0x7f76c019fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 --2- 172.21.15.46:0/3830303446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f76c01034b0 0x7f76c019f950 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 >> 172.21.15.46:0/3830303446 conn(0x7f76c00fef60 msgr2=0x7f76c0100710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 shutdown_connections 2024-03-08T23:10:21.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.368+0000 7f76c5600640 1 -- 172.21.15.46:0/3830303446 wait complete. 2024-03-08T23:10:21.388 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:21.388 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3975327294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 msgr2=0x7fc59c073b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3975327294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c073b00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc58c009900 tx=0x7fc58c02bcc0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3975327294 shutdown_connections 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3975327294 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c10d990 0x7fc59c10fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3975327294 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c074040 0x7fc59c10d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3975327294 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c073b00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3975327294 >> 172.21.15.46:0/3975327294 conn(0x7fc59c0fcc60 msgr2=0x7fc59c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3975327294 shutdown_connections 2024-03-08T23:10:21.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.578+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3975327294 wait complete. 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.579+0000 7fc5a0a00640 1 Processor -- start 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.579+0000 7fc5a0a00640 1 -- start start 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c074040 0x7fc59c19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c10d990 0x7fc59c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c113760 con 0x7fc59c073720 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1135e0 con 0x7fc59c074040 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc5a0a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1138e0 con 0x7fc59c10d990 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49518/0 (socket says 172.21.15.46:49518) 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 -- 172.21.15.46:0/3273500216 learned_addr learned my addr 172.21.15.46:0/3273500216 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59b200640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c10d990 0x7fc59c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc599e00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c074040 0x7fc59c19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 -- 172.21.15.46:0/3273500216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c10d990 msgr2=0x7fc59c199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c10d990 0x7fc59c199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 -- 172.21.15.46:0/3273500216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c074040 msgr2=0x7fc59c19fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.580+0000 7fc59a800640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c074040 0x7fc59c19fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc59a800640 1 -- 172.21.15.46:0/3273500216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59c19a200 con 0x7fc59c073720 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc59a800640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc593400640 1 -- 172.21.15.46:0/3273500216 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c042020 con 0x7fc59c073720 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc59c19a490 con 0x7fc59c073720 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc593400640 1 -- 172.21.15.46:0/3273500216 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc58c005000 con 0x7fc59c073720 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc593400640 1 -- 172.21.15.46:0/3273500216 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c003b00 con 0x7fc59c073720 2024-03-08T23:10:21.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.581+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc59c1a6740 con 0x7fc59c073720 2024-03-08T23:10:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.584+0000 7fc5a4193740 1 Processor -- start 2024-03-08T23:10:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.584+0000 7fc5a4193740 1 -- start start 2024-03-08T23:10:21.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.584+0000 7fc593400640 1 -- 172.21.15.46:0/3273500216 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc58c045020 con 0x7fc59c073720 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673dbe50 0x55d3673e44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36733fdd0 con 0x55d3673db530 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3672e8c70 con 0x55d3673e4a30 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc5a4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d36733be70 con 0x55d3673dbe50 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc59b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc599e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc59b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49530/0 (socket says 172.21.15.46:49530) 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc59b200640 1 -- 172.21.15.46:0/3755810502 learned_addr learned my addr 172.21.15.46:0/3755810502 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.585+0000 7fc59a800640 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673dbe50 0x55d3673e44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 -- 172.21.15.46:0/3755810502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673dbe50 msgr2=0x55d3673e44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673dbe50 0x55d3673e44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 -- 172.21.15.46:0/3755810502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 msgr2=0x55d3673db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 -- 172.21.15.46:0/3755810502 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3673a22b0 con 0x55d3673e4a30 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc59b200640 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc593400640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 0x7fc5800a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc593400640 1 -- 172.21.15.46:0/3273500216 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc58c0f0e60 con 0x7fc59c073720 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc599e00640 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e6e20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc59c19b220 tx=0x7fc58800f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac7ddd7c44f7bd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc59b200640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 0x7fc5800a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc587400640 1 -- 172.21.15.46:0/3755810502 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58800dea0 con 0x55d3673e4a30 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc587400640 1 -- 172.21.15.46:0/3755810502 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc588005500 con 0x55d3673e4a30 2024-03-08T23:10:21.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.586+0000 7fc587400640 1 -- 172.21.15.46:0/3755810502 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880116a0 con 0x55d3673e4a30 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 -- 172.21.15.46:0/3755810502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 msgr2=0x55d3673e6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e6e20 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc59c19b220 tx=0x7fc58800f730 comp rx=0 tx=0).stop 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc59b200640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 0x7fc5800a4eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc594009fd0 tx=0x7fc594009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 -- 172.21.15.46:0/3755810502 shutdown_connections 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e6e20 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673dbe50 0x55d3673e44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 --2- 172.21.15.46:0/3755810502 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673db530 0x55d3673db910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 -- 172.21.15.46:0/3755810502 >> 172.21.15.46:0/3755810502 conn(0x55d3673d5c60 msgr2=0x55d3673d6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 -- 172.21.15.46:0/3755810502 shutdown_connections 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.587+0000 7fc5a4193740 1 -- 172.21.15.46:0/3755810502 wait complete. 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 -- start start 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 Processor -- start 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673e1700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3673e7820 con 0x55d3673dbe50 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3672fc0c0 con 0x55d3673e4a30 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d3673e79a0 con 0x55d3673db530 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.588+0000 7fc5a4193740 10 client.0 _ll_register_callbacks cb 0x55d367336ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc599e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673e1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc599e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673e1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49542/0 (socket says 172.21.15.46:49542) 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc599e00640 1 -- 172.21.15.46:0/307647679 learned_addr learned my addr 172.21.15.46:0/307647679 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59b200640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 -- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 msgr2=0x55d3673e1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 -- 172.21.15.46:0/307647679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 msgr2=0x55d3673e1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673e1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 -- 172.21.15.46:0/307647679 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d3670543f0 con 0x55d3673db530 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc59a800640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e4370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0ee300 tx=0x7fc58c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cdce76be26026c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc57be00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58c0bcd50 con 0x55d3673db530 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.589+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc58c0051b0 con 0x55d3673db530 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc588005500 con 0x55d3673db530 2024-03-08T23:10:21.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc58c0bcb70 con 0x55d3673db530 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 10 client.26167 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d36711e7d0 con 0x55d3673db530 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc588005500 con 0x55d3673db530 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc58c003ca0 con 0x55d3673db530 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 10 client.26167 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 10 client.26167 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 20 client.26167 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d3673f4b00 0x55d3673f6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 4 client.26167 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.590+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d3673144f0 con 0x55d3673f4b00 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.591+0000 7fc599e00640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d3673f4b00 0x55d3673f6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.591+0000 7fc599e00640 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d3673f4b00 0x55d3673f6fa0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32e1a9702ef3990 server_cookie=ecf37a2695e6c7d1 in_seq=0 out_seq=0 2024-03-08T23:10:21.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.591+0000 7fc57be00640 10 client.26167 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc57be00640 1 -- 172.21.15.46:0/307647679 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d3673144f0 con 0x55d3673f4b00 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc57be00640 10 client.26167 handle_command_reply: tid=0 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 client.26167 shutdown 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 msgr2=0x55d3673e4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e4370 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc58c0ee300 tx=0x7fc58c009560 comp rx=0 tx=0).stop 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 shutdown_connections 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d3673f4b00 0x55d3673f6fa0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673e1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673e1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc57be00640 0 client.26167 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 --2- 172.21.15.46:0/307647679 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e4370 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 >> 172.21.15.46:0/307647679 conn(0x55d3673d5c60 msgr2=0x55d3673d79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.592+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 shutdown_connections 2024-03-08T23:10:21.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.593+0000 7fc5a4193740 1 -- 172.21.15.46:0/307647679 wait complete. 2024-03-08T23:10:21.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.593+0000 7fc5a4193740 20 client.26167 trim_cache size 0 max 16384 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.605+0000 7fc5a4193740 1 Processor -- start 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.605+0000 7fc5a4193740 1 -- start start 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.605+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.605+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 0x55d3673ebc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc5a4193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1b2a00 con 0x55d3673e4a30 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc5a4193740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1b0570 con 0x55d3673dbe50 2024-03-08T23:10:21.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc5a4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1ae610 con 0x55d3673db530 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc59b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 0x55d3673ebc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53886/0 (socket says 172.21.15.46:53886) 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc59b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 0x55d3673ebc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49558/0 (socket says 172.21.15.46:49558) 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 -- 172.21.15.46:0/520100549 learned_addr learned my addr 172.21.15.46:0/520100549 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc59a800640 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 -- 172.21.15.46:0/520100549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 msgr2=0x55d3673e0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 -- 172.21.15.46:0/520100549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 msgr2=0x55d3673ebc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 0x55d3673ebc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.606+0000 7fc599e00640 1 -- 172.21.15.46:0/520100549 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d3673dbc80 con 0x55d3673dbe50 2024-03-08T23:10:21.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.607+0000 7fc599e00640 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc58800db30 tx=0x7fc58800d930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0e90abe63e05fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.607+0000 7fc586a00640 1 -- 172.21.15.46:0/520100549 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58800dca0 con 0x55d3673dbe50 2024-03-08T23:10:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.607+0000 7fc586a00640 1 -- 172.21.15.46:0/520100549 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc58800de40 con 0x55d3673dbe50 2024-03-08T23:10:21.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.607+0000 7fc586a00640 1 -- 172.21.15.46:0/520100549 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5880050a0 con 0x55d3673dbe50 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- 172.21.15.46:0/520100549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 msgr2=0x55d3673dca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc58800db30 tx=0x7fc58800d930 comp rx=0 tx=0).stop 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- 172.21.15.46:0/520100549 shutdown_connections 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673e4a30 0x55d3673ebc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673dbe50 0x55d3673dca80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 --2- 172.21.15.46:0/520100549 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673e0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- 172.21.15.46:0/520100549 >> 172.21.15.46:0/520100549 conn(0x55d3673d5c60 msgr2=0x55d3673dc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- 172.21.15.46:0/520100549 shutdown_connections 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- 172.21.15.46:0/520100549 wait complete. 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 -- start start 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.608+0000 7fc5a4193740 1 Processor -- start 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673ea790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673eb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1aec10 con 0x55d3673dbe50 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1aff10 con 0x55d3673e4a30 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc59c1af250 con 0x55d3673db530 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc5a4193740 10 client.0 _ll_register_callbacks cb 0x7fc58800d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc599e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc599e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49572/0 (socket says 172.21.15.46:49572) 2024-03-08T23:10:21.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc599e00640 1 -- 172.21.15.46:0/2984461162 learned_addr learned my addr 172.21.15.46:0/2984461162 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc59a800640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673ea790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.609+0000 7fc59b200640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673eb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 -- 172.21.15.46:0/2984461162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 msgr2=0x55d3673ea790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673ea790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 -- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 msgr2=0x55d3673eb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673eb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 -- 172.21.15.46:0/2984461162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d36744b200 con 0x55d3673dbe50 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc59b200640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc599e00640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc58802d040 tx=0x7fc58800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9579396bbd6f4216 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc57be00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc588038060 con 0x55d3673dbe50 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.610+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc58800de40 con 0x55d3673dbe50 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc588004890 con 0x55d3673dbe50 2024-03-08T23:10:21.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc58803f050 con 0x55d3673dbe50 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 10 client.16038 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55d36744c500 con 0x55d3673dbe50 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc58800de40 con 0x55d3673dbe50 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc58802a230 con 0x55d3673dbe50 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 10 client.16038 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 20 client.16038 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.611+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d36744ca90 0x55d36744eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.612+0000 7fc5a4193740 4 client.16038 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.612+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d36734d310 con 0x55d36744ca90 2024-03-08T23:10:21.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.612+0000 7fc59a800640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d36744ca90 0x55d36744eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.612+0000 7fc59a800640 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d36744ca90 0x55d36744eeb0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c7f1ad1c79c830 server_cookie=70a2e02667ca3a69 in_seq=0 out_seq=0 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.612+0000 7fc57be00640 10 client.16038 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc57be00640 1 -- 172.21.15.46:0/2984461162 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d36734d310 con 0x55d36744ca90 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc57be00640 10 client.16038 handle_command_reply: tid=0 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 client.16038 shutdown 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 msgr2=0x55d3673eacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fc58802d040 tx=0x7fc58800fcb0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 shutdown_connections 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d36744ca90 0x55d36744eeb0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d3673e4a30 0x55d3673eb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d3673dbe50 0x55d3673eacd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 --2- 172.21.15.46:0/2984461162 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d3673db530 0x55d3673ea790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 >> 172.21.15.46:0/2984461162 conn(0x55d3673d5c60 msgr2=0x55d3673dc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc57be00640 0 client.16038 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 shutdown_connections 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 1 -- 172.21.15.46:0/2984461162 wait complete. 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.613+0000 7fc5a4193740 20 client.16038 trim_cache size 0 max 16384 2024-03-08T23:10:21.620 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 msgr2=0x7fc5800a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 0x7fc5800a4eb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc594009fd0 tx=0x7fc594009500 comp rx=0 tx=0).stop 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 msgr2=0x7fc59c19f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc58c000c00 tx=0x7fc58c0043c0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 shutdown_connections 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5800a29f0 0x7fc5800a4eb0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc59c10d990 0x7fc59c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc59c074040 0x7fc59c19fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 --2- 172.21.15.46:0/3273500216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc59c073720 0x7fc59c19f880 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 >> 172.21.15.46:0/3273500216 conn(0x7fc59c0fcc60 msgr2=0x7fc59c0fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 shutdown_connections 2024-03-08T23:10:21.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:21.614+0000 7fc5a0a00640 1 -- 172.21.15.46:0/3273500216 wait complete. 2024-03-08T23:10:22.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:21 smithi192 ceph-mon[36700]: pgmap v80: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 632 KiB/s wr, 47 op/s 2024-03-08T23:10:22.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:21 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3148369405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:22.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:21 smithi062 ceph-mon[35761]: pgmap v80: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 632 KiB/s wr, 47 op/s 2024-03-08T23:10:22.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:21 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3148369405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:22.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:21 smithi046 ceph-mon[31859]: pgmap v80: 129 pgs: 129 active+clean; 74 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 632 KiB/s wr, 47 op/s 2024-03-08T23:10:22.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:21 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3148369405' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:22.633 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:10:22.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:10:22.634 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:10:22.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.853+0000 7f77c2c00640 1 -- 172.21.15.46:0/2192889299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc10d8a0 msgr2=0x7f77bc10dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.853+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2192889299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc10d8a0 0x7f77bc10dc80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009930 tx=0x7f77ac02bc30 comp rx=0 tx=0).stop 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 -- 172.21.15.46:0/2192889299 shutdown_connections 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2192889299 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 0x7f77bc114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2192889299 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc107890 0x7f77bc107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2192889299 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc10d8a0 0x7f77bc10dc80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 -- 172.21.15.46:0/2192889299 >> 172.21.15.46:0/2192889299 conn(0x7f77bc075830 msgr2=0x7f77bc077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 -- 172.21.15.46:0/2192889299 shutdown_connections 2024-03-08T23:10:22.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.854+0000 7f77c2c00640 1 -- 172.21.15.46:0/2192889299 wait complete. 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 Processor -- start 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 -- start start 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 0x7f77bc19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 0x7f77bc1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc117cc0 con 0x7f77bc107890 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc117b40 con 0x7f77bc10d8a0 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77c2c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc117e40 con 0x7f77bc108250 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.855+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49586/0 (socket says 172.21.15.46:49586) 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 -- 172.21.15.46:0/2093486727 learned_addr learned my addr 172.21.15.46:0/2093486727 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:22.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bb400640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 0x7f77bc19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77c0c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 0x7f77bc1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 -- 172.21.15.46:0/2093486727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 msgr2=0x7f77bc19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 0x7f77bc19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 -- 172.21.15.46:0/2093486727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 msgr2=0x7f77bc1a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 0x7f77bc1a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 -- 172.21.15.46:0/2093486727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77bc1a31b0 con 0x7f77bc107890 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77c0c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 0x7f77bc1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:22.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.856+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009900 tx=0x7f77ac004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.857+0000 7f77b8c00640 1 -- 172.21.15.46:0/2093486727 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac043020 con 0x7f77bc107890 2024-03-08T23:10:22.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.857+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc1a3440 con 0x7f77bc107890 2024-03-08T23:10:22.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.857+0000 7f77b8c00640 1 -- 172.21.15.46:0/2093486727 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77ac0054e0 con 0x7f77bc107890 2024-03-08T23:10:22.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.857+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77bc1aad30 con 0x7f77bc107890 2024-03-08T23:10:22.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.858+0000 7f77b8c00640 1 -- 172.21.15.46:0/2093486727 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac003ca0 con 0x7f77bc107890 2024-03-08T23:10:22.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.861+0000 7f77b8c00640 1 -- 172.21.15.46:0/2093486727 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f77ac046020 con 0x7f77bc107890 2024-03-08T23:10:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.862+0000 7f77c59bd740 1 Processor -- start 2024-03-08T23:10:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.862+0000 7f77b8c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 0x7f77a00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.862+0000 7f77c59bd740 1 -- start start 2024-03-08T23:10:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.862+0000 7f77b8c00640 1 -- 172.21.15.46:0/2093486727 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f77ac0f12f0 con 0x7f77bc107890 2024-03-08T23:10:22.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c0c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 0x7f77a00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846bb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846bded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557884616a50 con 0x5578846bbae0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578845bfb00 con 0x5578846b25e0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c59bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557884612af0 con 0x5578846b2f00 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35862/0 (socket says 172.21.15.46:35862) 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77bbe00640 1 -- 172.21.15.46:0/1838585953 learned_addr learned my addr 172.21.15.46:0/1838585953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77bb400640 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846bb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.863+0000 7f77c0c00640 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846bded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 -- 172.21.15.46:0/1838585953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 msgr2=0x5578846bb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846bb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 -- 172.21.15.46:0/1838585953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 msgr2=0x5578846bded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846bded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77c0c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 0x7f77a00a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77b40092f0 tx=0x7f77b4009280 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 -- 172.21.15.46:0/1838585953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578845dd1e0 con 0x5578846b25e0 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0b9b00 tx=0x7f77ac0037d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b73e6643e66645d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77b1200640 1 -- 172.21.15.46:0/1838585953 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac004c80 con 0x5578846b25e0 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77b1200640 1 -- 172.21.15.46:0/1838585953 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77ac0035f0 con 0x5578846b25e0 2024-03-08T23:10:22.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.864+0000 7f77b1200640 1 -- 172.21.15.46:0/1838585953 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0c17f0 con 0x5578846b25e0 2024-03-08T23:10:22.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.865+0000 7f77c59bd740 1 -- 172.21.15.46:0/1838585953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 msgr2=0x5578846b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.865+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0b9b00 tx=0x7f77ac0037d0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.865+0000 7f77c59bd740 1 -- 172.21.15.46:0/1838585953 shutdown_connections 2024-03-08T23:10:22.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.865+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846bded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846bb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1838585953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846b29c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 -- 172.21.15.46:0/1838585953 >> 172.21.15.46:0/1838585953 conn(0x5578846acd10 msgr2=0x5578846ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 -- 172.21.15.46:0/1838585953 shutdown_connections 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 -- 172.21.15.46:0/1838585953 wait complete. 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.866+0000 7f77c59bd740 1 -- start start 2024-03-08T23:10:22.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 Processor -- start 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846b8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846bbae0 0x5578846b94d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578846be690 con 0x5578846b25e0 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578845d2f60 con 0x5578846b2f00 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578846be810 con 0x5578846bbae0 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.867+0000 7f77c59bd740 10 client.0 _ll_register_callbacks cb 0x5578845beb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38140/0 (socket says 172.21.15.46:38140) 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 -- 172.21.15.46:0/3931099786 learned_addr learned my addr 172.21.15.46:0/3931099786 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bb400640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846b8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 -- 172.21.15.46:0/3931099786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846bbae0 msgr2=0x5578846b94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846bbae0 0x5578846b94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 -- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 msgr2=0x5578846b8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846b8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 -- 172.21.15.46:0/3931099786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578843f37e0 con 0x5578846b25e0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77bbe00640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f77ac02bc70 tx=0x7f77ac0ee7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1ed3a83138c7bb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.868+0000 7f77a5600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0bdb00 con 0x5578846b25e0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f77ac0c59b0 con 0x5578846b25e0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557884623f90 con 0x5578846b25e0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77ac003e40 con 0x5578846b25e0 2024-03-08T23:10:22.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77c59bd740 10 client.16050 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557884453f70 con 0x5578846b25e0 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.869+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557884623f90 con 0x5578846b25e0 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f77ac082ce0 con 0x5578846b25e0 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 10 client.16050 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 20 client.16050 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5578846cb950 0x5578846cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 4 client.16050 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55788461e2e0 con 0x5578846cb950 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.870+0000 7f77bb400640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5578846cb950 0x5578846cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.871+0000 7f77bb400640 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5578846cb950 0x5578846cdd40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cd92bfd584ac534 server_cookie=db1657cfc3684358 in_seq=0 out_seq=0 2024-03-08T23:10:22.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.871+0000 7f77a5600640 10 client.16050 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:22.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.871+0000 7f77a5600640 1 -- 172.21.15.46:0/3931099786 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55788461e2e0 con 0x5578846cb950 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.871+0000 7f77a5600640 10 client.16050 handle_command_reply: tid=0 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.871+0000 7f77c59bd740 1 client.16050 shutdown 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 msgr2=0x5578846b8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f77ac02bc70 tx=0x7f77ac0ee7a0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 shutdown_connections 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5578846cb950 0x5578846cdd40 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846bbae0 0x5578846b94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846b8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 --2- 172.21.15.46:0/3931099786 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846b25e0 0x5578846b8a50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77a5600640 0 client.16050 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 >> 172.21.15.46:0/3931099786 conn(0x5578846acd10 msgr2=0x5578846aea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 shutdown_connections 2024-03-08T23:10:22.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 1 -- 172.21.15.46:0/3931099786 wait complete. 2024-03-08T23:10:22.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.872+0000 7f77c59bd740 20 client.16050 trim_cache size 0 max 16384 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 Processor -- start 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 -- start start 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846b6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b6fe0 con 0x5578846bbae0 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b4b50 con 0x5578846b25e0 2024-03-08T23:10:22.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.889+0000 7f77c59bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b2bf0 con 0x5578846b2f00 2024-03-08T23:10:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bb400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77c0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35870/0 (socket says 172.21.15.46:35870) 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 -- 172.21.15.46:0/1772324587 learned_addr learned my addr 172.21.15.46:0/1772324587 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bb400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53836/0 (socket says 172.21.15.46:53836) 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 -- 172.21.15.46:0/1772324587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 msgr2=0x5578846b5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846b5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 -- 172.21.15.46:0/1772324587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 msgr2=0x5578846b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 -- 172.21.15.46:0/1772324587 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5578842eb170 con 0x5578846b25e0 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.890+0000 7f77bbe00640 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0ba070 tx=0x7f77ac004ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a9938b2785c63f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.891+0000 7f77a7e00640 1 -- 172.21.15.46:0/1772324587 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac005690 con 0x5578846b25e0 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.891+0000 7f77a7e00640 1 -- 172.21.15.46:0/1772324587 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77ac0c5c00 con 0x5578846b25e0 2024-03-08T23:10:22.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.891+0000 7f77a7e00640 1 -- 172.21.15.46:0/1772324587 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77ac0c5e40 con 0x5578846b25e0 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.891+0000 7f77c59bd740 1 -- 172.21.15.46:0/1772324587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 msgr2=0x5578846bb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.891+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f77ac0ba070 tx=0x7f77ac004ee0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 -- 172.21.15.46:0/1772324587 shutdown_connections 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x5578846b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b2f00 0x5578846b5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 --2- 172.21.15.46:0/1772324587 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b25e0 0x5578846bb180 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 -- 172.21.15.46:0/1772324587 >> 172.21.15.46:0/1772324587 conn(0x5578846acd10 msgr2=0x5578846b7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 -- 172.21.15.46:0/1772324587 shutdown_connections 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 -- 172.21.15.46:0/1772324587 wait complete. 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 -- start start 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.892+0000 7f77c59bd740 1 Processor -- start 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b25e0 0x5578846c1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x557884721cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b31f0 con 0x5578846bbae0 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b44f0 con 0x5578846b2f00 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77bc1b3830 con 0x5578846b25e0 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c59bd740 10 client.0 _ll_register_callbacks cb 0x7f77ac0b9300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77c0c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x557884721cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35880/0 (socket says 172.21.15.46:35880) 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 -- 172.21.15.46:0/2040863089 learned_addr learned my addr 172.21.15.46:0/2040863089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b25e0 0x5578846c1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 -- 172.21.15.46:0/2040863089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b25e0 msgr2=0x5578846c1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b25e0 0x5578846c1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 -- 172.21.15.46:0/2040863089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 msgr2=0x557884721cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x557884721cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.893+0000 7f77bb400640 1 -- 172.21.15.46:0/2040863089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578845e2050 con 0x5578846b2f00 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77bb400640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77bc077e20 tx=0x7f77a801f4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1289401617781d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77a5600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77a8004220 con 0x5578846b2f00 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77ac0c5c00 con 0x5578846b2f00 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f77a80043c0 con 0x5578846b2f00 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77a8031050 con 0x5578846b2f00 2024-03-08T23:10:22.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77c59bd740 10 client.26183 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.894+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x557884723250 con 0x5578846b2f00 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77ac0c5c00 con 0x5578846b2f00 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f77a8026030 con 0x5578846b2f00 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 10 client.26183 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 20 client.26183 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557884723a10 0x557884725e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 4 client.26183 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55788459ca40 con 0x557884723a10 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.895+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557884723a10 0x557884725e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.896+0000 7f77bbe00640 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557884723a10 0x557884725e30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbd40bd06d5a7d56 server_cookie=c7c1b332dcb4017c in_seq=0 out_seq=0 2024-03-08T23:10:22.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.896+0000 7f77a5600640 10 client.26183 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.897+0000 7f77a5600640 1 -- 172.21.15.46:0/2040863089 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55788459ca40 con 0x557884723a10 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.897+0000 7f77a5600640 10 client.26183 handle_command_reply: tid=0 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.897+0000 7f77c59bd740 1 client.26183 shutdown 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.897+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 msgr2=0x5578846c1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.897+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77bc077e20 tx=0x7f77a801f4f0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 shutdown_connections 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557884723a10 0x557884725e30 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5578846bbae0 0x557884721cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5578846b2f00 0x5578846c1c80 secure :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f77bc077e20 tx=0x7f77a801f4f0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 --2- 172.21.15.46:0/2040863089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5578846b25e0 0x5578846c1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77a5600640 0 client.26183 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 >> 172.21.15.46:0/2040863089 conn(0x5578846acd10 msgr2=0x5578846ab580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 shutdown_connections 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 1 -- 172.21.15.46:0/2040863089 wait complete. 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.898+0000 7f77c59bd740 20 client.26183 trim_cache size 0 max 16384 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "526b45d4-faf3-4fc9-bb88-e244e407364b", 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:10:22.904 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 msgr2=0x7f77a00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 0x7f77a00a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f77b40092f0 tx=0x7f77b4009280 comp rx=0 tx=0).stop 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 msgr2=0x7f77bc19e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f77ac009900 tx=0x7f77ac004430 comp rx=0 tx=0).stop 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 shutdown_connections 2024-03-08T23:10:22.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77a00a2970 0x7f77a00a4e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f77bc10d8a0 0x7f77bc1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77bc108250 0x7f77bc19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 --2- 172.21.15.46:0/2093486727 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f77bc107890 0x7f77bc19e1c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 >> 172.21.15.46:0/2093486727 conn(0x7f77bc075830 msgr2=0x7f77bc075fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 shutdown_connections 2024-03-08T23:10:22.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:22.899+0000 7f77c2c00640 1 -- 172.21.15.46:0/2093486727 wait complete. 2024-03-08T23:10:22.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:10:22.926 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 -- 172.21.15.46:0/3111761819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc101e80 msgr2=0x7fe5cc105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3111761819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc101e80 0x7fe5cc105ec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009860 tx=0x7fe5b802bb70 comp rx=0 tx=0).stop 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 -- 172.21.15.46:0/3111761819 shutdown_connections 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3111761819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3111761819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc101e80 0x7fe5cc105ec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3111761819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc1018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.189+0000 7fe5d4200640 1 -- 172.21.15.46:0/3111761819 >> 172.21.15.46:0/3111761819 conn(0x7fe5cc0fcd50 msgr2=0x7fe5cc0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.190+0000 7fe5d4200640 1 -- 172.21.15.46:0/3111761819 shutdown_connections 2024-03-08T23:10:23.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.190+0000 7fe5d4200640 1 -- 172.21.15.46:0/3111761819 wait complete. 2024-03-08T23:10:23.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 Processor -- start 2024-03-08T23:10:23.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 -- start start 2024-03-08T23:10:23.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc112580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc10fcc0 0x7fe5cc1aa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1138c0 con 0x7fe5cc10fcc0 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc113740 con 0x7fe5cc1065a0 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.192+0000 7fe5d4200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc113a40 con 0x7fe5cc1014d0 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.193+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc10fcc0 0x7fe5cc1aa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.193+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc112580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.193+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc112580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53872/0 (socket says 172.21.15.46:53872) 2024-03-08T23:10:23.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.193+0000 7fe5d1800640 1 -- 172.21.15.46:0/3854421959 learned_addr learned my addr 172.21.15.46:0/3854421959 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.193+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 -- 172.21.15.46:0/3854421959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 msgr2=0x7fe5cc112580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc112580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 -- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc10fcc0 msgr2=0x7fe5cc1aa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc10fcc0 0x7fe5cc1aa5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 -- 172.21.15.46:0/3854421959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc1aaae0 con 0x7fe5cc1065a0 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.194+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10f620 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8037040 tx=0x7fe5b8004530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.195+0000 7fe5ca000640 1 -- 172.21.15.46:0/3854421959 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8041020 con 0x7fe5cc1065a0 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.195+0000 7fe5ca000640 1 -- 172.21.15.46:0/3854421959 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5b802c9f0 con 0x7fe5cc1065a0 2024-03-08T23:10:23.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.195+0000 7fe5ca000640 1 -- 172.21.15.46:0/3854421959 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b8039040 con 0x7fe5cc1065a0 2024-03-08T23:10:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.196+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc1aacb0 con 0x7fe5cc1065a0 2024-03-08T23:10:23.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.196+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc1ab120 con 0x7fe5cc1065a0 2024-03-08T23:10:23.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.200+0000 7fe5d6e5d740 1 Processor -- start 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.200+0000 7fe5d6e5d740 1 -- start start 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.200+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.200+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ec540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.200+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fe9eee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fe9a5410 con 0x55f7fe9e3580 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fe947d50 con 0x55f7fe9eca80 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fe8f0b80 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38180/0 (socket says 172.21.15.46:38180) 2024-03-08T23:10:23.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.201+0000 7fe5d2200640 1 -- 172.21.15.46:0/1336923645 learned_addr learned my addr 172.21.15.46:0/1336923645 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:23.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d2200640 1 -- 172.21.15.46:0/1336923645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 msgr2=0x55f7fe9ec540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:10:23.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d2200640 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ec540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d2200640 1 -- 172.21.15.46:0/1336923645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 msgr2=0x55f7fe9eee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fe9eee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d2200640 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fe9eee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d2200640 1 -- 172.21.15.46:0/1336923645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7fe7267d0 con 0x55f7fe9e3580 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.202+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fe9eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.203+0000 7fe5d2200640 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009f50 tx=0x7fe5c40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d969d003fe1140f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.203+0000 7fe5ca000640 1 -- 172.21.15.46:0/3854421959 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe5b8044020 con 0x7fe5cc1065a0 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.205+0000 7fe5bea00640 1 -- 172.21.15.46:0/1336923645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4004d20 con 0x55f7fe9e3580 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.205+0000 7fe5bea00640 1 -- 172.21.15.46:0/1336923645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5c4004050 con 0x55f7fe9e3580 2024-03-08T23:10:23.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.205+0000 7fe5bea00640 1 -- 172.21.15.46:0/1336923645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4005280 con 0x55f7fe9e3580 2024-03-08T23:10:23.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.206+0000 7fe5ca000640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 0x7fe5b40a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/1336923645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 msgr2=0x55f7fe9e3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe5c4009f50 tx=0x7fe5c40118d0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/1336923645 shutdown_connections 2024-03-08T23:10:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fe9eee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ec540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/1336923645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e3960 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/1336923645 >> 172.21.15.46:0/1336923645 conn(0x55f7fe9ddcb0 msgr2=0x55f7fe9de090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d1800640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 0x7fe5b40a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/1336923645 shutdown_connections 2024-03-08T23:10:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.207+0000 7fe5ca000640 1 -- 172.21.15.46:0/3854421959 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fe5b80f07d0 con 0x7fe5cc1065a0 2024-03-08T23:10:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.208+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/1336923645 wait complete. 2024-03-08T23:10:23.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.208+0000 7fe5d6e5d740 1 -- start start 2024-03-08T23:10:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.209+0000 7fe5d1800640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 0x7fe5b40a4f70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00096c0 tx=0x7fe5c0009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.209+0000 7fe5d6e5d740 1 Processor -- start 2024-03-08T23:10:23.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 0x55f7fe9ea5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fe903ee0 con 0x55f7fe9eca80 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fe943df0 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7fea00880 con 0x55f7fe9e3580 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d6e5d740 10 client.0 _ll_register_callbacks cb 0x55f7fe90a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:23.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53880/0 (socket says 172.21.15.46:53880) 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.210+0000 7fe5d0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35924/0 (socket says 172.21.15.46:35924) 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 -- 172.21.15.46:0/4287990745 learned_addr learned my addr 172.21.15.46:0/4287990745 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d2200640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 0x55f7fe9ea5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 -- 172.21.15.46:0/4287990745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 msgr2=0x55f7fe9e9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 -- 172.21.15.46:0/4287990745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 msgr2=0x55f7fe9ea5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 0x55f7fe9ea5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.211+0000 7fe5d0e00640 1 -- 172.21.15.46:0/4287990745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7fe65c3f0 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5d1800640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5d2200640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 0x55f7fe9ea5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009780 tx=0x7fe5b80057e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=445d9a45290133ab server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5b3400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b80bd280 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fe5b8032de0 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.212+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5c4004050 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5b8034700 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5d6e5d740 10 client.26192 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f7fe61c0f0 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5c4004050 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe5b803c030 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5d6e5d740 10 client.26192 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5d6e5d740 10 client.26192 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.213+0000 7fe5d6e5d740 20 client.26192 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea0dc60 0x55f7fea100a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5d6e5d740 4 client.26192 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7fe955290 con 0x55f7fea0dc60 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5d1800640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea0dc60 0x55f7fea100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5d1800640 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea0dc60 0x55f7fea100a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0e6b43510eaef26 server_cookie=7b369f20113925e1 in_seq=0 out_seq=0 2024-03-08T23:10:23.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.214+0000 7fe5b3400640 10 client.26192 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:23.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.215+0000 7fe5b3400640 1 -- 172.21.15.46:0/4287990745 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f7fe955290 con 0x55f7fea0dc60 2024-03-08T23:10:23.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.215+0000 7fe5b3400640 10 client.26192 handle_command_reply: tid=0 2024-03-08T23:10:23.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.215+0000 7fe5d6e5d740 1 client.26192 shutdown 2024-03-08T23:10:23.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.215+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 msgr2=0x55f7fe9ea060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.215+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8009780 tx=0x7fe5b80057e0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 shutdown_connections 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea0dc60 0x55f7fea100a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9eca80 0x55f7fe9ea5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9e3ea0 0x55f7fe9ea060 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/4287990745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fe9e9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 >> 172.21.15.46:0/4287990745 conn(0x55f7fe9ddcb0 msgr2=0x55f7fe9df170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5b3400640 0 client.26192 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 shutdown_connections 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/4287990745 wait complete. 2024-03-08T23:10:23.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.216+0000 7fe5d6e5d740 20 client.26192 trim_cache size 0 max 16384 2024-03-08T23:10:23.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 Processor -- start 2024-03-08T23:10:23.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 -- start start 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea04ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea05000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1ae900 con 0x55f7fe9e3580 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1ac470 con 0x55f7fe9eca80 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.242+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1a0760 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea04ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea05000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea04ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53892/0 (socket says 172.21.15.46:53892) 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d0e00640 1 -- 172.21.15.46:0/382745150 learned_addr learned my addr 172.21.15.46:0/382745150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d1800640 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 -- 172.21.15.46:0/382745150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 msgr2=0x55f7fea04ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea04ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 -- 172.21.15.46:0/382745150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 msgr2=0x55f7fe9ea3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9ea3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.243+0000 7fe5d2200640 1 -- 172.21.15.46:0/382745150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7fe9d98f0 con 0x55f7fe9eca80 2024-03-08T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.244+0000 7fe5d2200640 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea05000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe5c401b040 tx=0x7fe5c4003ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adcc94796d1fd716 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.244+0000 7fe5be000640 1 -- 172.21.15.46:0/382745150 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4027030 con 0x55f7fe9eca80 2024-03-08T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.245+0000 7fe5be000640 1 -- 172.21.15.46:0/382745150 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5c4004530 con 0x55f7fe9eca80 2024-03-08T23:10:23.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.245+0000 7fe5be000640 1 -- 172.21.15.46:0/382745150 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c4004e50 con 0x55f7fe9eca80 2024-03-08T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/382745150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 msgr2=0x55f7fea05000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea05000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fe5c401b040 tx=0x7fe5c4003ac0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/382745150 shutdown_connections 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea05000 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea04ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/382745150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3580 0x55f7fe9ea3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/382745150 >> 172.21.15.46:0/382745150 conn(0x55f7fe9ddcb0 msgr2=0x55f7fea0a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/382745150 shutdown_connections 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.246+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/382745150 wait complete. 2024-03-08T23:10:23.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.247+0000 7fe5d6e5d740 1 -- start start 2024-03-08T23:10:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.247+0000 7fe5d6e5d740 1 Processor -- start 2024-03-08T23:10:23.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea58880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea631d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1a0d60 con 0x55f7fe9e3ea0 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1abe10 con 0x55f7fe9eca80 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc1a13a0 con 0x55f7fe9e3580 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d6e5d740 10 client.0 _ll_register_callbacks cb 0x7fe5c4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea631d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53894/0 (socket says 172.21.15.46:53894) 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d1800640 1 -- 172.21.15.46:0/2151355350 learned_addr learned my addr 172.21.15.46:0/2151355350 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:23.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.248+0000 7fe5d2200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea631d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:35946/0 (socket says 172.21.15.46:35946) 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea58880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d1800640 1 -- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 msgr2=0x55f7fea631d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d1800640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea631d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d1800640 1 -- 172.21.15.46:0/2151355350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 msgr2=0x55f7fea58880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d1800640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea58880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d1800640 1 -- 172.21.15.46:0/2151355350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7fea63790 con 0x55f7fe9e3580 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.249+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea58880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:10:23.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.250+0000 7fe5d1800640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55f7fea109c0 tx=0x7fe5c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63f9c27234b439bf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:23.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.250+0000 7fe5b3400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:23.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5c00037f0 con 0x55f7fe9e3580 2024-03-08T23:10:23.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe5c4004530 con 0x55f7fe9e3580 2024-03-08T23:10:23.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fe5c0003990 con 0x55f7fe9e3580 2024-03-08T23:10:23.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5c003a050 con 0x55f7fe9e3580 2024-03-08T23:10:23.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5d6e5d740 10 client.26203 fetch_fsmap learned FSMap version 26 2024-03-08T23:10:23.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55f7fea64ae0 con 0x55f7fe9e3580 2024-03-08T23:10:23.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.251+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe5c4004530 con 0x55f7fe9e3580 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 26) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fe5c0018040 con 0x55f7fe9e3580 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 10 client.26203 fetch_fsmap finished waiting for FSMap version 26 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 10 client.26203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 20 client.26203 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea65160 0x55f7fea67580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 4 client.26203 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f7fe94f5e0 con 0x55f7fea65160 2024-03-08T23:10:23.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.252+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea65160 0x55f7fea67580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:23.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.253+0000 7fe5b3400640 10 client.26203 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:23.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.253+0000 7fe5d0e00640 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea65160 0x55f7fea67580 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf23206e59c35067 server_cookie=b2fb580d5ab9212f in_seq=0 out_seq=0 2024-03-08T23:10:23.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.253+0000 7fe5b3400640 1 -- 172.21.15.46:0/2151355350 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f7fe94f5e0 con 0x55f7fea65160 2024-03-08T23:10:23.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.253+0000 7fe5b3400640 10 client.26203 handle_command_reply: tid=0 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.253+0000 7fe5d6e5d740 1 client.26203 shutdown 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 msgr2=0x55f7fea58340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55f7fea109c0 tx=0x7fe5c0004730 comp rx=0 tx=0).stop 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 shutdown_connections 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f7fea65160 0x55f7fea67580 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f7fe9eca80 0x55f7fea631d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f7fe9e3ea0 0x55f7fea58880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 --2- 172.21.15.46:0/2151355350 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f7fe9e3580 0x55f7fea58340 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 >> 172.21.15.46:0/2151355350 conn(0x55f7fe9ddcb0 msgr2=0x55f7fe9e83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5b3400640 0 client.26203 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:23.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.254+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 shutdown_connections 2024-03-08T23:10:23.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.255+0000 7fe5d6e5d740 1 -- 172.21.15.46:0/2151355350 wait complete. 2024-03-08T23:10:23.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.255+0000 7fe5d6e5d740 20 client.26203 trim_cache size 0 max 16384 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "526b45d4-faf3-4fc9-bb88-e244e407364b": { 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "526b45d4-faf3-4fc9-bb88-e244e407364b", 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:10:23.260 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.255+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 msgr2=0x7fe5b40a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.255+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 0x7fe5b40a4f70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe5c00096c0 tx=0x7fe5c0009260 comp rx=0 tx=0).stop 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 msgr2=0x7fe5cc10f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10f620 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8037040 tx=0x7fe5b8004530 comp rx=0 tx=0).stop 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 shutdown_connections 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe5b40a2ab0 0x7fe5b40a4f70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe5cc10fcc0 0x7fe5cc1aa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe5cc1065a0 0x7fe5cc10f620 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe5b8037040 tx=0x7fe5b8004530 comp rx=0 tx=0).stop 2024-03-08T23:10:23.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 --2- 172.21.15.46:0/3854421959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe5cc1014d0 0x7fe5cc112580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 >> 172.21.15.46:0/3854421959 conn(0x7fe5cc0fcd50 msgr2=0x7fe5cc0fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 shutdown_connections 2024-03-08T23:10:23.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:23.256+0000 7fe5d4200640 1 -- 172.21.15.46:0/3854421959 wait complete. 2024-03-08T23:10:23.285 INFO:tasks.cephfs.filesystem:scrub status for tag:526b45d4-faf3-4fc9-bb88-e244e407364b - {'path': '/', 'tag': '526b45d4-faf3-4fc9-bb88-e244e407364b', 'options': 'recursive,force'} 2024-03-08T23:10:23.551 INFO:tasks.workunit.client.0.smithi046.stdout: [DEP] 2024-03-08T23:10:23.555 INFO:tasks.workunit.client.0.smithi046.stdout: [DEP] 2024-03-08T23:10:23.556 INFO:tasks.workunit.client.0.smithi046.stdout: [DEP] 2024-03-08T23:10:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:23 smithi192 ceph-mon[36700]: pgmap v81: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 687 KiB/s wr, 91 op/s 2024-03-08T23:10:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:23 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:23 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:23 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:23 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.121 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] dirstress 2024-03-08T23:10:24.126 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fill 2024-03-08T23:10:24.127 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fill2 2024-03-08T23:10:24.132 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] getpagesize 2024-03-08T23:10:24.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:23 smithi062 ceph-mon[35761]: pgmap v81: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 687 KiB/s wr, 91 op/s 2024-03-08T23:10:24.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:23 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:24.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:23 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:23 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:24.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:23 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:23 smithi046 ceph-mon[31859]: pgmap v81: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 687 KiB/s wr, 91 op/s 2024-03-08T23:10:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:23 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:23 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:23 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:23 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:24.447 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] holes 2024-03-08T23:10:24.464 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] lstat64 2024-03-08T23:10:24.580 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] nametest 2024-03-08T23:10:24.586 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] permname 2024-03-08T23:10:24.812 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] randholes 2024-03-08T23:10:24.838 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] runas 2024-03-08T23:10:24.968 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] truncfile 2024-03-08T23:10:25.002 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] usemem 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:24 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:24 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.181 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] mmapcat 2024-03-08T23:10:25.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:10:25.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:25.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:25.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:24 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:25.265 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] append_reader 2024-03-08T23:10:25.294 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] append_writer 2024-03-08T23:10:25.340 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] dirperf 2024-03-08T23:10:25.479 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] metaperf 2024-03-08T23:10:25.570 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] devzero 2024-03-08T23:10:25.599 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] feature 2024-03-08T23:10:25.765 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] alloc 2024-03-08T23:10:25.933 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fault 2024-03-08T23:10:25.939 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] fstest 2024-03-08T23:10:25.967 INFO:tasks.workunit.client.0.smithi046.stdout: [CC] t_access_root 2024-03-08T23:10:26.042 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c: In function 'main': 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: 154 | bozo! 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~ 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: 154 | bozo! 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: | ^ 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: | ; 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: 184 | rflag = 1; 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~ 2024-03-08T23:10:26.043 INFO:tasks.workunit.client.0.smithi046.stderr: | nflag 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr: 320 | bozo! 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr: | ^ 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr: | ; 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr: 128 | struct flock64 f; 2024-03-08T23:10:26.044 INFO:tasks.workunit.client.0.smithi046.stderr: | ^ 2024-03-08T23:10:26.052 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-08T23:10:26.052 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-08T23:10:26.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:25 smithi192 ceph-mon[36700]: pgmap v82: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:26.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:25 smithi062 ceph-mon[35761]: pgmap v82: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c: In function 'run_child': 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~ 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~ 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:26.220 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~ 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~ 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-08T23:10:26.221 INFO:tasks.workunit.client.0.smithi046.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-08T23:10:26.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:25 smithi046 ceph-mon[31859]: pgmap v82: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:26.329 INFO:tasks.workunit.client.0.smithi046.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-08T23:10:26.329 INFO:tasks.workunit.client.0.smithi046.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-08T23:10:26.330 INFO:tasks.workunit.client.0.smithi046.stderr:make: *** [Makefile:51: default] Error 2 2024-03-08T23:10:26.332 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-08T23:10:26.332 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-08T23:10:26.332 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-08T23:10:27.382 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi046 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-08T23:10:27.636 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=ff70c05f0c8346a8a9b94f3f856a0109 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi046 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-08T23:10:27.638 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-08T23:10:27.649 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-08T23:10:27.649 INFO:teuthology.task.exec:Executing custom commands... 2024-03-08T23:10:27.649 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:10:27.649 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-08T23:10:27.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-08T23:10:28.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:27 smithi192 ceph-mon[36700]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:28.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:27 smithi062 ceph-mon[35761]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:27 smithi046 ceph-mon[31859]: pgmap v83: 129 pgs: 129 active+clean; 73 MiB data, 515 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 530 KiB/s wr, 81 op/s 2024-03-08T23:10:30.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:29 smithi192 ceph-mon[36700]: pgmap v84: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 553 KiB/s wr, 90 op/s 2024-03-08T23:10:30.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:29 smithi062 ceph-mon[35761]: pgmap v84: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 553 KiB/s wr, 90 op/s 2024-03-08T23:10:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:29 smithi046 ceph-mon[31859]: pgmap v84: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 553 KiB/s wr, 90 op/s 2024-03-08T23:10:32.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:31 smithi192 ceph-mon[36700]: pgmap v85: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-03-08T23:10:32.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:31 smithi062 ceph-mon[35761]: pgmap v85: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-03-08T23:10:32.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:31 smithi046 ceph-mon[31859]: pgmap v85: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 78 KiB/s wr, 52 op/s 2024-03-08T23:10:34.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:33 smithi192 ceph-mon[36700]: pgmap v86: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 108 KiB/s wr, 56 op/s 2024-03-08T23:10:34.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:33 smithi062 ceph-mon[35761]: pgmap v86: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 108 KiB/s wr, 56 op/s 2024-03-08T23:10:34.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:33 smithi046 ceph-mon[31859]: pgmap v86: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 108 KiB/s wr, 56 op/s 2024-03-08T23:10:36.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:35 smithi062 ceph-mon[35761]: pgmap v87: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:36.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:35 smithi046 ceph-mon[31859]: pgmap v87: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:36.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:35 smithi192 ceph-mon[36700]: pgmap v87: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:37 smithi062 ceph-mon[35761]: pgmap v88: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:37 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:10:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:37 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:10:38.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:37 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:10:38.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:37 smithi046 ceph-mon[31859]: pgmap v88: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:38.417 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:37 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:10:38.418 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:37 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:10:38.418 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:37 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:10:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:37 smithi192 ceph-mon[36700]: pgmap v88: 129 pgs: 129 active+clean; 74 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 12 op/s 2024-03-08T23:10:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:37 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:10:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:37 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:10:38.419 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:37 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:10:40.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:39 smithi062 ceph-mon[35761]: pgmap v89: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 294 KiB/s wr, 23 op/s 2024-03-08T23:10:40.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:39 smithi046 ceph-mon[31859]: pgmap v89: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 294 KiB/s wr, 23 op/s 2024-03-08T23:10:40.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:39 smithi192 ceph-mon[36700]: pgmap v89: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 294 KiB/s wr, 23 op/s 2024-03-08T23:10:42.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:41 smithi062 ceph-mon[35761]: pgmap v90: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 271 KiB/s wr, 15 op/s 2024-03-08T23:10:42.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:41 smithi046 ceph-mon[31859]: pgmap v90: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 271 KiB/s wr, 15 op/s 2024-03-08T23:10:42.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:41 smithi192 ceph-mon[36700]: pgmap v90: 129 pgs: 129 active+clean; 76 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 271 KiB/s wr, 15 op/s 2024-03-08T23:10:44.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:43 smithi062 ceph-mon[35761]: pgmap v91: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 363 KiB/s wr, 22 op/s 2024-03-08T23:10:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:43 smithi046 ceph-mon[31859]: pgmap v91: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 363 KiB/s wr, 22 op/s 2024-03-08T23:10:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:43 smithi192 ceph-mon[36700]: pgmap v91: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 363 KiB/s wr, 22 op/s 2024-03-08T23:10:45.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:44 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:45.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:44 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:45.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:44 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:10:46.127 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:45 smithi046 ceph-mon[31859]: pgmap v92: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:46.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:45 smithi062 ceph-mon[35761]: pgmap v92: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:46.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:45 smithi192 ceph-mon[36700]: pgmap v92: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:48.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:47 smithi062 ceph-mon[35761]: pgmap v93: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:48.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:47 smithi046 ceph-mon[31859]: pgmap v93: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:48.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:47 smithi192 ceph-mon[36700]: pgmap v93: 129 pgs: 129 active+clean; 77 MiB data, 508 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 334 KiB/s wr, 18 op/s 2024-03-08T23:10:50.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:49 smithi062 ceph-mon[35761]: pgmap v94: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 337 KiB/s wr, 22 op/s 2024-03-08T23:10:50.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:49 smithi046 ceph-mon[31859]: pgmap v94: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 337 KiB/s wr, 22 op/s 2024-03-08T23:10:50.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:49 smithi192 ceph-mon[36700]: pgmap v94: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 337 KiB/s wr, 22 op/s 2024-03-08T23:10:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:51 smithi062 ceph-mon[35761]: pgmap v95: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 96 KiB/s wr, 10 op/s 2024-03-08T23:10:52.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:51 smithi046 ceph-mon[31859]: pgmap v95: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 96 KiB/s wr, 10 op/s 2024-03-08T23:10:52.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:51 smithi192 ceph-mon[36700]: pgmap v95: 129 pgs: 129 active+clean; 77 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 96 KiB/s wr, 10 op/s 2024-03-08T23:10:53.285 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:10:53.286 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:10:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.517+0000 7fb069800640 1 -- 172.21.15.46:0/2320421718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 msgr2=0x7fb064103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.517+0000 7fb069800640 1 --2- 172.21.15.46:0/2320421718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064103930 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb04c009890 tx=0x7fb04c02bb70 comp rx=0 tx=0).stop 2024-03-08T23:10:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 -- 172.21.15.46:0/2320421718 shutdown_connections 2024-03-08T23:10:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 --2- 172.21.15.46:0/2320421718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064103e70 0x7fb064110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 --2- 172.21.15.46:0/2320421718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064103930 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 --2- 172.21.15.46:0/2320421718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064109570 0x7fb064109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 -- 172.21.15.46:0/2320421718 >> 172.21.15.46:0/2320421718 conn(0x7fb0640fef40 msgr2=0x7fb064101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 -- 172.21.15.46:0/2320421718 shutdown_connections 2024-03-08T23:10:53.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.518+0000 7fb069800640 1 -- 172.21.15.46:0/2320421718 wait complete. 2024-03-08T23:10:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.519+0000 7fb069800640 1 Processor -- start 2024-03-08T23:10:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.519+0000 7fb069800640 1 -- start start 2024-03-08T23:10:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 0x7fb06419a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb064113850 con 0x7fb064103e70 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641136d0 con 0x7fb064109570 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb069800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641139d0 con 0x7fb0641034b0 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 0x7fb06419a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb063400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb063400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40240/0 (socket says 172.21.15.46:40240) 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb063400640 1 -- 172.21.15.46:0/1220855790 learned_addr learned my addr 172.21.15.46:0/1220855790 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.520+0000 7fb062a00640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 -- 172.21.15.46:0/1220855790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 msgr2=0x7fb064199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 -- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 msgr2=0x7fb06419a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 0x7fb06419a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 -- 172.21.15.46:0/1220855790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06419ed20 con 0x7fb064109570 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb05a000640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 0x7fb06419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb063400640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00eb10 tx=0x7fb05c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb05be00640 1 -- 172.21.15.46:0/1220855790 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c00ce20 con 0x7fb064109570 2024-03-08T23:10:53.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb05be00640 1 -- 172.21.15.46:0/1220855790 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb05c005020 con 0x7fb064109570 2024-03-08T23:10:53.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb06419f010 con 0x7fb064109570 2024-03-08T23:10:53.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb05be00640 1 -- 172.21.15.46:0/1220855790 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05c010690 con 0x7fb064109570 2024-03-08T23:10:53.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.521+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0641a68a0 con 0x7fb064109570 2024-03-08T23:10:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.524+0000 7fb06c3f0740 1 Processor -- start 2024-03-08T23:10:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb05be00640 1 -- 172.21.15.46:0/1220855790 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb05c005540 con 0x7fb064109570 2024-03-08T23:10:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 -- start start 2024-03-08T23:10:53.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 0x55d4078ea540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078ece70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4078a3410 con 0x55d4078e1580 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407845d50 con 0x55d4078eaa80 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4077eeb80 con 0x55d4078e1ea0 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.525+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38324/0 (socket says 172.21.15.46:38324) 2024-03-08T23:10:53.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb05a000640 1 -- 172.21.15.46:0/3403052524 learned_addr learned my addr 172.21.15.46:0/3403052524 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 0x55d4078ea540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb05a000640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 -- 172.21.15.46:0/3403052524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 msgr2=0x55d4078ece70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078ece70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 -- 172.21.15.46:0/3403052524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 msgr2=0x55d4078e1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 -- 172.21.15.46:0/3403052524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4076247d0 con 0x55d4078e1ea0 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb05a000640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:53.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb05be00640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 0x7fb0480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.526+0000 7fb062a00640 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 0x55d4078ea540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb054009c30 tx=0x7fb054011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a778239ce2f65b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb05be00640 1 -- 172.21.15.46:0/1220855790 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb05c0cf8f0 con 0x7fb064109570 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb05a000640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 0x7fb0480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb053e00640 1 -- 172.21.15.46:0/3403052524 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05401e070 con 0x55d4078e1ea0 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb053e00640 1 -- 172.21.15.46:0/3403052524 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0540035f0 con 0x55d4078e1ea0 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb053e00640 1 -- 172.21.15.46:0/3403052524 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054019690 con 0x55d4078e1ea0 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3403052524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 msgr2=0x55d4078ea540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb05a000640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 0x7fb0480a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb04c000c00 tx=0x7fb04c005da0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.527+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 0x55d4078ea540 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb054009c30 tx=0x7fb054011810 comp rx=0 tx=0).stop 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3403052524 shutdown_connections 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078ece70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1ea0 0x55d4078ea540 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3403052524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078e1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3403052524 >> 172.21.15.46:0/3403052524 conn(0x55d4078dbcb0 msgr2=0x55d4078dc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3403052524 shutdown_connections 2024-03-08T23:10:53.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3403052524 wait complete. 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 -- start start 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.528+0000 7fb06c3f0740 1 Processor -- start 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1580 0x55d4078e7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078e84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407801ee0 con 0x55d4078e1ea0 2024-03-08T23:10:53.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d407841df0 con 0x55d4078eaa80 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4078ed870 con 0x55d4078e1580 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb06c3f0740 10 client.0 _ll_register_callbacks cb 0x55d4078088d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38326/0 (socket says 172.21.15.46:38326) 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb05a000640 1 -- 172.21.15.46:0/3241758018 learned_addr learned my addr 172.21.15.46:0/3241758018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb062a00640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1580 0x55d4078e7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.529+0000 7fb063400640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078e84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 -- 172.21.15.46:0/3241758018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1580 msgr2=0x55d4078e7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1580 0x55d4078e7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 -- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 msgr2=0x55d4078e84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078e84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 -- 172.21.15.46:0/3241758018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d40755a3f0 con 0x55d4078e1ea0 2024-03-08T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb05a000640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0057c0 tx=0x7fb04c0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0e648e839ebe9e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb050c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:53.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c050070 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb04c02c920 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.530+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb0540035f0 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb04c05c050 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 10 client.16083 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d40751a0f0 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0540035f0 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fb04c04b070 con 0x55d4078e1ea0 2024-03-08T23:10:53.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 10 client.16083 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 10 client.16083 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 20 client.16083 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d4078facd0 0x55d4078fd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 4 client.16083 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.531+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d407853290 con 0x55d4078facd0 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.532+0000 7fb062a00640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d4078facd0 0x55d4078fd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.532+0000 7fb062a00640 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d4078facd0 0x55d4078fd130 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baa42f6df7bdcc6f server_cookie=f04bc6f6bcbb6f5f in_seq=0 out_seq=0 2024-03-08T23:10:53.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.532+0000 7fb050c00640 10 client.16083 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb050c00640 1 -- 172.21.15.46:0/3241758018 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d407853290 con 0x55d4078facd0 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb050c00640 10 client.16083 handle_command_reply: tid=0 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 client.16083 shutdown 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 msgr2=0x55d4078e7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0057c0 tx=0x7fb04c0056d0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 shutdown_connections 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d4078facd0 0x55d4078fd130 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078eaa80 0x55d4078e84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d4078e7fa0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/3241758018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078e1580 0x55d4078e7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb050c00640 0 client.16083 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.533+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 >> 172.21.15.46:0/3241758018 conn(0x55d4078dbcb0 msgr2=0x55d4078dda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.534+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 shutdown_connections 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.534+0000 7fb06c3f0740 1 -- 172.21.15.46:0/3241758018 wait complete. 2024-03-08T23:10:53.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.534+0000 7fb06c3f0740 20 client.16083 trim_cache size 0 max 16384 2024-03-08T23:10:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 Processor -- start 2024-03-08T23:10:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 -- start start 2024-03-08T23:10:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 0x55d407901ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4078f1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641b2cc0 con 0x55d4078e1580 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.550+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641b0800 con 0x55d4078e1ea0 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641ae8a0 con 0x55d4078eaa80 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38342/0 (socket says 172.21.15.46:38342) 2024-03-08T23:10:53.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 -- 172.21.15.46:0/864161078 learned_addr learned my addr 172.21.15.46:0/864161078 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb05a000640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 0x55d407901ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb063400640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4078f1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 -- 172.21.15.46:0/864161078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 msgr2=0x55d4078f1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4078f1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 -- 172.21.15.46:0/864161078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 msgr2=0x55d407901ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 0x55d407901ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb062a00640 1 -- 172.21.15.46:0/864161078 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4078d78f0 con 0x55d4078e1580 2024-03-08T23:10:53.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.551+0000 7fb05a000640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 0x55d407901ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:53.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.552+0000 7fb062a00640 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb054009b80 tx=0x7fb054009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e360e21e4dde9a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.552+0000 7fb053400640 1 -- 172.21.15.46:0/864161078 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05401e040 con 0x55d4078e1580 2024-03-08T23:10:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.552+0000 7fb053400640 1 -- 172.21.15.46:0/864161078 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb054029030 con 0x55d4078e1580 2024-03-08T23:10:53.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.552+0000 7fb053400640 1 -- 172.21.15.46:0/864161078 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0540291e0 con 0x55d4078e1580 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- 172.21.15.46:0/864161078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 msgr2=0x55d4078ea190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb054009b80 tx=0x7fb054009d20 comp rx=0 tx=0).stop 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- 172.21.15.46:0/864161078 shutdown_connections 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4078f1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1ea0 0x55d407901ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/864161078 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1580 0x55d4078ea190 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- 172.21.15.46:0/864161078 >> 172.21.15.46:0/864161078 conn(0x55d4078dbcb0 msgr2=0x55d4078e2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- 172.21.15.46:0/864161078 shutdown_connections 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- 172.21.15.46:0/864161078 wait complete. 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 -- start start 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.553+0000 7fb06c3f0740 1 Processor -- start 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d40794b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4079451e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641aeea0 con 0x55d4078e1ea0 2024-03-08T23:10:53.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641b01a0 con 0x55d4078e1580 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641af4e0 con 0x55d4078eaa80 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb06c3f0740 10 client.0 _ll_register_callbacks cb 0x7fb0540097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb05a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d40794b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb062a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb062a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40290/0 (socket says 172.21.15.46:40290) 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb062a00640 1 -- 172.21.15.46:0/508879127 learned_addr learned my addr 172.21.15.46:0/508879127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.554+0000 7fb063400640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4079451e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 -- 172.21.15.46:0/508879127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 msgr2=0x55d4079451e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4079451e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 -- 172.21.15.46:0/508879127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 msgr2=0x55d40794b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d40794b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 -- 172.21.15.46:0/508879127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4079458d0 con 0x55d4078e1580 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb062a00640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb054003970 tx=0x7fb054011dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc14453f5d4ee4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb050c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05401e040 con 0x55d4078e1580 2024-03-08T23:10:53.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb054029030 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb0540035f0 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb054003a10 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb06c3f0740 10 client.26216 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.555+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d407952320 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb054029030 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fb054029600 con 0x55d4078e1580 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 10 client.26216 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:53.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 10 client.26216 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 20 client.26216 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d407952800 0x55d407954c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 4 client.26216 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d40784d5e0 con 0x55d407952800 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.556+0000 7fb05a000640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d407952800 0x55d407954c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb05a000640 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d407952800 0x55d407954c20 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1ead5761920e064 server_cookie=4da2a1aba9bed44e in_seq=0 out_seq=0 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb050c00640 10 client.26216 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb050c00640 1 -- 172.21.15.46:0/508879127 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55d40784d5e0 con 0x55d407952800 2024-03-08T23:10:53.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb050c00640 10 client.26216 handle_command_reply: tid=0 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 client.26216 shutdown 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 msgr2=0x55d40794ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb054003970 tx=0x7fb054011dc0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 shutdown_connections 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d407952800 0x55d407954c20 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d4078eaa80 0x55d4079451e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d4078e1ea0 0x55d40794b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 --2- 172.21.15.46:0/508879127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d4078e1580 0x55d40794ab70 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb050c00640 0 client.26216 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.557+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 >> 172.21.15.46:0/508879127 conn(0x55d4078dbcb0 msgr2=0x55d4078e5f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 shutdown_connections 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb06c3f0740 1 -- 172.21.15.46:0/508879127 wait complete. 2024-03-08T23:10:53.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb06c3f0740 20 client.26216 trim_cache size 0 max 16384 2024-03-08T23:10:53.571 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:53.571 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:10:53.571 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:10:53.571 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 msgr2=0x7fb0480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 0x7fb0480a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb04c000c00 tx=0x7fb04c005da0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 msgr2=0x7fb06419e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fb05c00eb10 tx=0x7fb05c00ef10 comp rx=0 tx=0).stop 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 shutdown_connections 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb0480a2970 0x7fb0480a4e30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb064109570 0x7fb06419e5f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb064103e70 0x7fb06419a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 --2- 172.21.15.46:0/1220855790 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0641034b0 0x7fb064199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.558+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 >> 172.21.15.46:0/1220855790 conn(0x7fb0640fef40 msgr2=0x7fb0641008f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.559+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 shutdown_connections 2024-03-08T23:10:53.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.559+0000 7fb069800640 1 -- 172.21.15.46:0/1220855790 wait complete. 2024-03-08T23:10:53.580 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:10:53.580 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:10:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 -- 172.21.15.46:0/3310166046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0103540 msgr2=0x7faab0103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 --2- 172.21.15.46:0/3310166046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0103540 0x7faab0103920 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7faaa0009900 tx=0x7faaa002bc30 comp rx=0 tx=0).stop 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 -- 172.21.15.46:0/3310166046 shutdown_connections 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 --2- 172.21.15.46:0/3310166046 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0108610 0x7faab01101e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 --2- 172.21.15.46:0/3310166046 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0103ef0 0x7faab0107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 --2- 172.21.15.46:0/3310166046 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0103540 0x7faab0103920 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.803+0000 7faaaf200640 1 -- 172.21.15.46:0/3310166046 >> 172.21.15.46:0/3310166046 conn(0x7faab00fed80 msgr2=0x7faab01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.804+0000 7faaaf200640 1 -- 172.21.15.46:0/3310166046 shutdown_connections 2024-03-08T23:10:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.804+0000 7faaaf200640 1 -- 172.21.15.46:0/3310166046 wait complete. 2024-03-08T23:10:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.805+0000 7faaaf200640 1 Processor -- start 2024-03-08T23:10:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.805+0000 7faaaf200640 1 -- start start 2024-03-08T23:10:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 0x7faab0199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab0113710 con 0x7faab0103540 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab0113590 con 0x7faab0108610 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaaf200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab0113890 con 0x7faab0199a70 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaade00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaa7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaa7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40308/0 (socket says 172.21.15.46:40308) 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaa7e00640 1 -- 172.21.15.46:0/848751334 learned_addr learned my addr 172.21.15.46:0/848751334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:53.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaade00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38368/0 (socket says 172.21.15.46:38368) 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.806+0000 7faaae800640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 0x7faab0199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 -- 172.21.15.46:0/848751334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 msgr2=0x7faab019fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 -- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 msgr2=0x7faab019f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 -- 172.21.15.46:0/848751334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab019a590 con 0x7faab0199a70 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaa7e00640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:53.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaade00640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:53.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaae800640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 0x7faab0199f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faaa800ec50 tx=0x7faaa800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8003b70 con 0x7faab0199a70 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaa8003d10 con 0x7faab0199a70 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8010890 con 0x7faab0199a70 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.807+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab019a880 con 0x7faab0199a70 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.808+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab01a6a80 con 0x7faab0199a70 2024-03-08T23:10:53.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.809+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab019af50 con 0x7faab0199a70 2024-03-08T23:10:53.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.811+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7faaa8004b40 con 0x7faab0199a70 2024-03-08T23:10:53.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.812+0000 7faaa6a00640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 0x7faa940a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:53.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.812+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7faaa80cf640 con 0x7faab0199a70 2024-03-08T23:10:53.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.812+0000 7faaade00640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 0x7faa940a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:53.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.813+0000 7faaade00640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 0x7faa940a4eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faaa0000c00 tx=0x7faaa002df90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:53.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:53.814+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7faaa809c040 con 0x7faab0199a70 2024-03-08T23:10:53.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:53 smithi046 ceph-mon[31859]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-03-08T23:10:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.061+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faab0068430 con 0x7faab0199a70 2024-03-08T23:10:54.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.062+0000 7faaa6a00640 1 -- 172.21.15.46:0/848751334 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11340 (secure 0 0 0) 0x7faab0068430 con 0x7faab0199a70 2024-03-08T23:10:54.066 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:10:54.066 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:10:43.831330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:10:54.067 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-03-08T23:10:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 msgr2=0x7faa940a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 0x7faa940a4eb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faaa0000c00 tx=0x7faaa002df90 comp rx=0 tx=0).stop 2024-03-08T23:10:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 msgr2=0x7faab0199f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 0x7faab0199f20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faaa800ec50 tx=0x7faaa800c6a0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 shutdown_connections 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faa940a29f0 0x7faa940a4eb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faab0199a70 0x7faab0199f20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faab0108610 0x7faab019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 --2- 172.21.15.46:0/848751334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faab0103540 0x7faab019f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 >> 172.21.15.46:0/848751334 conn(0x7faab00fed80 msgr2=0x7faab0101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 shutdown_connections 2024-03-08T23:10:54.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.065+0000 7faaaf200640 1 -- 172.21.15.46:0/848751334 wait complete. 2024-03-08T23:10:54.090 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:54.090 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:10:54.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:53 smithi192 ceph-mon[36700]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 -- 172.21.15.46:0/1313789048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe8073720 msgr2=0x7f2fe8073b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 --2- 172.21.15.46:0/1313789048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe8073720 0x7f2fe8073b00 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009900 tx=0x7f2fd802bcd0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 -- 172.21.15.46:0/1313789048 shutdown_connections 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 --2- 172.21.15.46:0/1313789048 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe810fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 --2- 172.21.15.46:0/1313789048 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe810d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 --2- 172.21.15.46:0/1313789048 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe8073720 0x7f2fe8073b00 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 -- 172.21.15.46:0/1313789048 >> 172.21.15.46:0/1313789048 conn(0x7f2fe80fcc50 msgr2=0x7f2fe80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 -- 172.21.15.46:0/1313789048 shutdown_connections 2024-03-08T23:10:54.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.311+0000 7f2feca00640 1 -- 172.21.15.46:0/1313789048 wait complete. 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.312+0000 7f2feca00640 1 Processor -- start 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.312+0000 7f2feca00640 1 -- start start 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.312+0000 7f2feca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe8199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2feca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2feca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe819e700 0x7f2fe819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2feca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe8113730 con 0x7f2fe819e700 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2feca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81135b0 con 0x7f2fe810d9a0 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2feca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81138b0 con 0x7f2fe8074040 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe8199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe819e700 0x7f2fe819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40324/0 (socket says 172.21.15.46:40324) 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe8199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49194/0 (socket says 172.21.15.46:49194) 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 -- 172.21.15.46:0/2956470903 learned_addr learned my addr 172.21.15.46:0/2956470903 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 -- 172.21.15.46:0/2956470903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 msgr2=0x7f2fe8199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe8199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 -- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe819e700 msgr2=0x7f2fe819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe819e700 0x7f2fe819ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.313+0000 7f2fdea00640 1 -- 172.21.15.46:0/2956470903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe819f1f0 con 0x7f2fe810d9a0 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2fdea00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e9f0 tx=0x7f2fd000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2fe4c00640 1 -- 172.21.15.46:0/2956470903 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd000cd80 con 0x7f2fe810d9a0 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2fe4c00640 1 -- 172.21.15.46:0/2956470903 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fd0005500 con 0x7f2fe810d9a0 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe819f4e0 con 0x7f2fe810d9a0 2024-03-08T23:10:54.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2fe4c00640 1 -- 172.21.15.46:0/2956470903 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0010690 con 0x7f2fe810d9a0 2024-03-08T23:10:54.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.314+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe81a6c80 con 0x7f2fe810d9a0 2024-03-08T23:10:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.317+0000 7f2fe4c00640 1 -- 172.21.15.46:0/2956470903 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2fd00046c0 con 0x7f2fe810d9a0 2024-03-08T23:10:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.317+0000 7f2ff0da4740 1 Processor -- start 2024-03-08T23:10:54.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.317+0000 7f2ff0da4740 1 -- start start 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 0x5612c8537910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 0x5612c85404f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c8542e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c849bdd0 con 0x5612c8537e50 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c8444c70 con 0x5612c8540a30 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c8497e70 con 0x5612c8537530 2024-03-08T23:10:54.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 0x5612c85404f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 0x5612c85404f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38402/0 (socket says 172.21.15.46:38402) 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 0x5612c8537910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2fe7400640 1 -- 172.21.15.46:0/789398206 learned_addr learned my addr 172.21.15.46:0/789398206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.318+0000 7f2fdea00640 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c8542e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 -- 172.21.15.46:0/789398206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 msgr2=0x5612c8542e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c8542e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 -- 172.21.15.46:0/789398206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 msgr2=0x5612c85404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 0x5612c85404f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 -- 172.21.15.46:0/789398206 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612c84fe2b0 con 0x5612c8537530 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe4c00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 0x7f2fcc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 0x5612c8537910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009f00 tx=0x7f2fe0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69487788841f1b32 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe4c00640 1 -- 172.21.15.46:0/2956470903 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2fd00cf620 con 0x7f2fe810d9a0 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.319+0000 7f2fe7400640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 0x7f2fcc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2fd7e00640 1 -- 172.21.15.46:0/789398206 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0004d20 con 0x5612c8537530 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2fd7e00640 1 -- 172.21.15.46:0/789398206 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fe0004050 con 0x5612c8537530 2024-03-08T23:10:54.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2fd7e00640 1 -- 172.21.15.46:0/789398206 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0005240 con 0x5612c8537530 2024-03-08T23:10:54.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 -- 172.21.15.46:0/789398206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 msgr2=0x5612c8537910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 0x5612c8537910 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009f00 tx=0x7f2fe0011810 comp rx=0 tx=0).stop 2024-03-08T23:10:54.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2fe7400640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 0x7f2fcc0a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5612c8537000 tx=0x7f2fd802cf90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 -- 172.21.15.46:0/789398206 shutdown_connections 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c8542e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537e50 0x5612c85404f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/789398206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537530 0x5612c8537910 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 -- 172.21.15.46:0/789398206 >> 172.21.15.46:0/789398206 conn(0x5612c8531c60 msgr2=0x5612c8532040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 -- 172.21.15.46:0/789398206 shutdown_connections 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.320+0000 7f2ff0da4740 1 -- 172.21.15.46:0/789398206 wait complete. 2024-03-08T23:10:54.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.321+0000 7f2ff0da4740 1 -- start start 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.321+0000 7f2ff0da4740 1 Processor -- start 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c85403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c853d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c853dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c8543820 con 0x5612c8537530 2024-03-08T23:10:54.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c84580c0 con 0x5612c8540a30 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612c85439a0 con 0x5612c8537e50 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2ff0da4740 10 client.0 _ll_register_callbacks cb 0x5612c8492ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c853d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c85403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c853dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c853d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49208/0 (socket says 172.21.15.46:49208) 2024-03-08T23:10:54.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fdea00640 1 -- 172.21.15.46:0/443104704 learned_addr learned my addr 172.21.15.46:0/443104704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 -- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 msgr2=0x5612c853d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c853d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 -- 172.21.15.46:0/443104704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 msgr2=0x5612c85403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c85403b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.322+0000 7f2fe7e00640 1 -- 172.21.15.46:0/443104704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612c81b03f0 con 0x5612c8540a30 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c853dcc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009730 tx=0x7f2fe0004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaea93147114bdef server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2fd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0029020 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2fe0025470 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fe0004050 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fe0034050 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2ff0da4740 10 client.26234 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.323+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5612c827a7d0 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fe0004050 con 0x5612c8540a30 2024-03-08T23:10:54.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f2fe0019700 con 0x5612c8540a30 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 10 client.26234 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 10 client.26234 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 20 client.26234 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c8550aa0 0x5612c8552e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 4 client.26234 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5612c84704f0 con 0x5612c8550aa0 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.324+0000 7f2fe7400640 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c8550aa0 0x5612c8552e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.325+0000 7f2fe7400640 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c8550aa0 0x5612c8552e90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f184ae5cb5ab037 server_cookie=c1bc2f29cc8b35b5 in_seq=0 out_seq=0 2024-03-08T23:10:54.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.325+0000 7f2fd4c00640 10 client.26234 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2fd4c00640 1 -- 172.21.15.46:0/443104704 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5612c84704f0 con 0x5612c8550aa0 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2fd4c00640 10 client.26234 handle_command_reply: tid=0 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 client.26234 shutdown 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 msgr2=0x5612c853dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c853dcc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009730 tx=0x7f2fe0004430 comp rx=0 tx=0).stop 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 shutdown_connections 2024-03-08T23:10:54.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c8550aa0 0x5612c8552e90 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8540a30 0x5612c853dcc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c853d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/443104704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c85403b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 >> 172.21.15.46:0/443104704 conn(0x5612c8531c60 msgr2=0x5612c85339b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2fd4c00640 0 client.26234 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:54.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 shutdown_connections 2024-03-08T23:10:54.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.326+0000 7f2ff0da4740 1 -- 172.21.15.46:0/443104704 wait complete. 2024-03-08T23:10:54.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.327+0000 7f2ff0da4740 20 client.26234 trim_cache size 0 max 16384 2024-03-08T23:10:54.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 Processor -- start 2024-03-08T23:10:54.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 -- start start 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c853fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8540a30 0x5612c853a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81b2fa0 con 0x5612c8537530 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81b0ae0 con 0x5612c8537e50 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81aeb80 con 0x5612c8540a30 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c853fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40356/0 (socket says 172.21.15.46:40356) 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.340+0000 7f2fdea00640 1 -- 172.21.15.46:0/3885196591 learned_addr learned my addr 172.21.15.46:0/3885196591 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 -- 172.21.15.46:0/3885196591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8540a30 msgr2=0x5612c853a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8540a30 0x5612c853a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 -- 172.21.15.46:0/3885196591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 msgr2=0x5612c853fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c853fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 -- 172.21.15.46:0/3885196591 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5612c853bc90 con 0x5612c8537e50 2024-03-08T23:10:54.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fdea00640 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00049d0 tx=0x7f2fd0086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc6971846b946c41 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fd7400640 1 -- 172.21.15.46:0/3885196591 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd009b220 con 0x5612c8537e50 2024-03-08T23:10:54.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fd7400640 1 -- 172.21.15.46:0/3885196591 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fd009bfc0 con 0x5612c8537e50 2024-03-08T23:10:54.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.341+0000 7f2fd7400640 1 -- 172.21.15.46:0/3885196591 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00cf220 con 0x5612c8537e50 2024-03-08T23:10:54.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3885196591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 msgr2=0x5612c85403b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00049d0 tx=0x7f2fd0086040 comp rx=0 tx=0).stop 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3885196591 shutdown_connections 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8540a30 0x5612c853a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537e50 0x5612c85403b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3885196591 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8537530 0x5612c853fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3885196591 >> 172.21.15.46:0/3885196591 conn(0x5612c8531c60 msgr2=0x5612c853c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3885196591 shutdown_connections 2024-03-08T23:10:54.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3885196591 wait complete. 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.342+0000 7f2ff0da4740 1 -- start start 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 Processor -- start 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 0x5612c8546600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c8546b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81af180 con 0x5612c8540a30 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81b0480 con 0x5612c8537530 2024-03-08T23:10:54.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe81af7c0 con 0x5612c8537e50 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2ff0da4740 10 client.0 _ll_register_callbacks cb 0x7f2fe0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 0x5612c8546600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fe7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38422/0 (socket says 172.21.15.46:38422) 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fe7e00640 1 -- 172.21.15.46:0/3434942833 learned_addr learned my addr 172.21.15.46:0/3434942833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fe7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 0x5612c8546600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40362/0 (socket says 172.21.15.46:40362) 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.343+0000 7f2fdea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c8546b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 -- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 msgr2=0x5612c8546b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c8546b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 -- 172.21.15.46:0/3434942833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 msgr2=0x5612c8546600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 0x5612c8546600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 -- 172.21.15.46:0/3434942833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612c85a6e20 con 0x5612c8540a30 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fe7e00640 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009560 tx=0x7f2fe0005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dbbea45814a9c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fd4c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe003d060 con 0x5612c8540a30 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2fe0003f10 con 0x5612c8540a30 2024-03-08T23:10:54.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.344+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2fd009bfc0 con 0x5612c8540a30 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2fe0044050 con 0x5612c8540a30 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 10 client.16113 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5612c85a80b0 con 0x5612c8540a30 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2fd009bfc0 con 0x5612c8540a30 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f2fe00257a0 con 0x5612c8540a30 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 10 client.16113 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 10 client.16113 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 20 client.16113 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c85a87e0 0x5612c85aac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 4 client.16113 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.345+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5612c84a9310 con 0x5612c85a87e0 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2fe7400640 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c85a87e0 0x5612c85aac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2fe7400640 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c85a87e0 0x5612c85aac00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=192b7ce4077b2160 server_cookie=7067ebf52bca4f5d in_seq=0 out_seq=0 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2fd4c00640 10 client.16113 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2fd4c00640 1 -- 172.21.15.46:0/3434942833 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5612c84a9310 con 0x5612c85a87e0 2024-03-08T23:10:54.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2fd4c00640 10 client.16113 handle_command_reply: tid=0 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.346+0000 7f2ff0da4740 1 client.16113 shutdown 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 msgr2=0x5612c8547080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009560 tx=0x7f2fe0005c50 comp rx=0 tx=0).stop 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 shutdown_connections 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5612c85a87e0 0x5612c85aac00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5612c8540a30 0x5612c8547080 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5612c8537e50 0x5612c8546b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 --2- 172.21.15.46:0/3434942833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5612c8537530 0x5612c8546600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 >> 172.21.15.46:0/3434942833 conn(0x5612c8531c60 msgr2=0x5612c8530450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2fd4c00640 0 client.16113 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 shutdown_connections 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 1 -- 172.21.15.46:0/3434942833 wait complete. 2024-03-08T23:10:54.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.347+0000 7f2ff0da4740 20 client.16113 trim_cache size 0 max 16384 2024-03-08T23:10:54.360 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:54.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 msgr2=0x7f2fcc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 0x7f2fcc0a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x5612c8537000 tx=0x7f2fd802cf90 comp rx=0 tx=0).stop 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 msgr2=0x7f2fe819a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2fd000e9f0 tx=0x7f2fd000edf0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 shutdown_connections 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2fcc0a2970 0x7f2fcc0a4e30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2fe819e700 0x7f2fe819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2fe810d9a0 0x7f2fe819a380 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 --2- 172.21.15.46:0/2956470903 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2fe8074040 0x7f2fe8199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 >> 172.21.15.46:0/2956470903 conn(0x7f2fe80fcc50 msgr2=0x7f2fe80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 shutdown_connections 2024-03-08T23:10:54.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.348+0000 7f2feca00640 1 -- 172.21.15.46:0/2956470903 wait complete. 2024-03-08T23:10:54.371 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:54.371 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:10:54.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:53 smithi062 ceph-mon[35761]: pgmap v96: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 102 KiB/s wr, 14 op/s 2024-03-08T23:10:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 -- 172.21.15.46:0/818151128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f726406a1e0 msgr2=0x7f726410e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 --2- 172.21.15.46:0/818151128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f726406a1e0 0x7f726410e170 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f725c009cd0 tx=0x7f725c02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 -- 172.21.15.46:0/818151128 shutdown_connections 2024-03-08T23:10:54.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 --2- 172.21.15.46:0/818151128 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f726406a1e0 0x7f726410e170 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 --2- 172.21.15.46:0/818151128 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7264069860 0x7f7264069ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 --2- 172.21.15.46:0/818151128 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 -- 172.21.15.46:0/818151128 >> 172.21.15.46:0/818151128 conn(0x7f72640771d0 msgr2=0x7f72640775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 -- 172.21.15.46:0/818151128 shutdown_connections 2024-03-08T23:10:54.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.581+0000 7f726a200640 1 -- 172.21.15.46:0/818151128 wait complete. 2024-03-08T23:10:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.582+0000 7f726a200640 1 Processor -- start 2024-03-08T23:10:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.582+0000 7f726a200640 1 -- start start 2024-03-08T23:10:54.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726406a1e0 0x7f726419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264111650 con 0x7f7264069860 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641114d0 con 0x7f726406a1e0 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f726a200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641117d0 con 0x7f7264107340 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f7263400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f7263400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38448/0 (socket says 172.21.15.46:38448) 2024-03-08T23:10:54.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f7263400640 1 -- 172.21.15.46:0/259022283 learned_addr learned my addr 172.21.15.46:0/259022283 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f725aa00640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726406a1e0 0x7f726419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.583+0000 7f7263e00640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 -- 172.21.15.46:0/259022283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 msgr2=0x7f7264197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 -- 172.21.15.46:0/259022283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726406a1e0 msgr2=0x7f726419dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726406a1e0 0x7f726419dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 -- 172.21.15.46:0/259022283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264198110 con 0x7f7264069860 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263e00640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:54.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7263400640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f725400e9f0 tx=0x7f725400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7260c00640 1 -- 172.21.15.46:0/259022283 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725400cd80 con 0x7f7264069860 2024-03-08T23:10:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7260c00640 1 -- 172.21.15.46:0/259022283 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7254010910 con 0x7f7264069860 2024-03-08T23:10:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f7260c00640 1 -- 172.21.15.46:0/259022283 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7254018990 con 0x7f7264069860 2024-03-08T23:10:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7264198400 con 0x7f7264069860 2024-03-08T23:10:54.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.584+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72641a46a0 con 0x7f7264069860 2024-03-08T23:10:54.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.587+0000 7f7260c00640 1 -- 172.21.15.46:0/259022283 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7254010ac0 con 0x7f7264069860 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 Processor -- start 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 -- start start 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f99629a0 0x5580f996f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 0x5580f9972050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f98c6dd0 con 0x5580f9962080 2024-03-08T23:10:54.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f986fc70 con 0x5580f996fc60 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.588+0000 7f726cfca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f98c2e70 con 0x5580f99629a0 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38456/0 (socket says 172.21.15.46:38456) 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 -- 172.21.15.46:0/687377580 learned_addr learned my addr 172.21.15.46:0/687377580 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263400640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f99629a0 0x5580f996f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f725aa00640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 0x5580f9972050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 -- 172.21.15.46:0/687377580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f99629a0 msgr2=0x5580f996f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f99629a0 0x5580f996f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 -- 172.21.15.46:0/687377580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 msgr2=0x5580f9972050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 0x5580f9972050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7263e00640 1 -- 172.21.15.46:0/687377580 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580f99292b0 con 0x5580f9962080 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f725aa00640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 0x5580f9972050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7260c00640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 0x7f72480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.589+0000 7f7260c00640 1 -- 172.21.15.46:0/259022283 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f72540cf4e0 con 0x7f7264069860 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.590+0000 7f7263400640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 0x7f72480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.590+0000 7f7263e00640 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f725c009ca0 tx=0x7f725c0323c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bacb82fa22f94fb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.590+0000 7f7253e00640 1 -- 172.21.15.46:0/687377580 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c043020 con 0x5580f9962080 2024-03-08T23:10:54.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.590+0000 7f7253e00640 1 -- 172.21.15.46:0/687377580 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c02cca0 con 0x5580f9962080 2024-03-08T23:10:54.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.590+0000 7f7253e00640 1 -- 172.21.15.46:0/687377580 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c039040 con 0x5580f9962080 2024-03-08T23:10:54.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- 172.21.15.46:0/687377580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 msgr2=0x5580f9962460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f725c009ca0 tx=0x7f725c0323c0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f7263400640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 0x7f72480a4eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5580f9963260 tx=0x7f72540d2f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- 172.21.15.46:0/687377580 shutdown_connections 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f996fc60 0x5580f9972050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f99629a0 0x5580f996f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 --2- 172.21.15.46:0/687377580 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f9962080 0x5580f9962460 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- 172.21.15.46:0/687377580 >> 172.21.15.46:0/687377580 conn(0x5580f995ca80 msgr2=0x5580f995ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- 172.21.15.46:0/687377580 shutdown_connections 2024-03-08T23:10:54.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- 172.21.15.46:0/687377580 wait complete. 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.591+0000 7f726cfca740 1 -- start start 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 Processor -- start 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f996c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f99729c0 con 0x5580f996fc60 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f98830c0 con 0x5580f99629a0 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580f9972b40 con 0x5580f9962080 2024-03-08T23:10:54.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f726cfca740 10 client.0 _ll_register_callbacks cb 0x5580f98bdec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f7263e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.592+0000 7f725aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f996c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f7263e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38462/0 (socket says 172.21.15.46:38462) 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f7263e00640 1 -- 172.21.15.46:0/1485685363 learned_addr learned my addr 172.21.15.46:0/1485685363 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f7263e00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 -- 172.21.15.46:0/1485685363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 msgr2=0x5580f996f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 -- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 msgr2=0x5580f996ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 -- 172.21.15.46:0/1485685363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580f95db3f0 con 0x5580f99629a0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f7263e00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f725aa00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f996c920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7264106a10 tx=0x7f724c012a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ecf0ceedb8e378 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.593+0000 7f7250c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c004030 con 0x5580f99629a0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f724c0041d0 con 0x5580f99629a0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f725c02cca0 con 0x5580f99629a0 2024-03-08T23:10:54.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724c025050 con 0x5580f99629a0 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f726cfca740 10 client.26252 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5580f96a57d0 con 0x5580f99629a0 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c02cca0 con 0x5580f99629a0 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f724c00f780 con 0x5580f99629a0 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f726cfca740 10 client.26252 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.594+0000 7f726cfca740 10 client.26252 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:10:54.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f726cfca740 20 client.26252 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f997fd50 0x5580f9982160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f726cfca740 4 client.26252 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580f989b4f0 con 0x5580f997fd50 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f7263e00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f997fd50 0x5580f9982160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f7263e00640 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f997fd50 0x5580f9982160 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=76f7fe064aae83a8 server_cookie=2d3ca8119d8a3b42 in_seq=0 out_seq=0 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.595+0000 7f7250c00640 10 client.26252 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f7250c00640 1 -- 172.21.15.46:0/1485685363 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5580f989b4f0 con 0x5580f997fd50 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f7250c00640 10 client.26252 handle_command_reply: tid=0 2024-03-08T23:10:54.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 client.26252 shutdown 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 msgr2=0x5580f996c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f996c920 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7264106a10 tx=0x7f724c012a80 comp rx=0 tx=0).stop 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 shutdown_connections 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f996c920 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f997fd50 0x5580f9982160 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.608 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 --2- 172.21.15.46:0/1485685363 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 >> 172.21.15.46:0/1485685363 conn(0x5580f995ca80 msgr2=0x5580f995e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.596+0000 7f7250c00640 0 client.26252 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:54.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.597+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 shutdown_connections 2024-03-08T23:10:54.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.597+0000 7f726cfca740 1 -- 172.21.15.46:0/1485685363 wait complete. 2024-03-08T23:10:54.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.597+0000 7f726cfca740 20 client.26252 trim_cache size 0 max 16384 2024-03-08T23:10:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 Processor -- start 2024-03-08T23:10:54.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 -- start start 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f995b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ae880 con 0x5580f996fc60 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ac3c0 con 0x5580f99629a0 2024-03-08T23:10:54.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f726cfca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641aa460 con 0x5580f9962080 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.610+0000 7f7263e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49266/0 (socket says 172.21.15.46:49266) 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 -- 172.21.15.46:0/3938499351 learned_addr learned my addr 172.21.15.46:0/3938499351 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f725aa00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f995b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 -- 172.21.15.46:0/3938499351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 msgr2=0x5580f995b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f995b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 -- 172.21.15.46:0/3938499351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 msgr2=0x5580f996f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 -- 172.21.15.46:0/3938499351 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580f996d440 con 0x5580f9962080 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f725aa00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f995b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.611+0000 7f7263e00640 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f725c009430 tx=0x7f725c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d7a1d0a8229246c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f7253400640 1 -- 172.21.15.46:0/3938499351 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c039030 con 0x5580f9962080 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f7253400640 1 -- 172.21.15.46:0/3938499351 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c055790 con 0x5580f9962080 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f7253400640 1 -- 172.21.15.46:0/3938499351 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c043040 con 0x5580f9962080 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 -- 172.21.15.46:0/3938499351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 msgr2=0x5580f996ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f725c009430 tx=0x7f725c004a40 comp rx=0 tx=0).stop 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 -- 172.21.15.46:0/3938499351 shutdown_connections 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f996fc60 0x5580f996f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f99629a0 0x5580f995b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 --2- 172.21.15.46:0/3938499351 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f9962080 0x5580f996ba80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 -- 172.21.15.46:0/3938499351 >> 172.21.15.46:0/3938499351 conn(0x5580f995ca80 msgr2=0x5580f995b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.612+0000 7f726cfca740 1 -- 172.21.15.46:0/3938499351 shutdown_connections 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 -- 172.21.15.46:0/3938499351 wait complete. 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 -- start start 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 Processor -- start 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 0x5580f99772e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f996fc60 0x5580f9975dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.613+0000 7f726cfca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641aaa60 con 0x5580f99629a0 2024-03-08T23:10:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f726cfca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641abd60 con 0x5580f9962080 2024-03-08T23:10:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f726cfca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ab0a0 con 0x5580f996fc60 2024-03-08T23:10:54.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f726cfca740 10 client.0 _ll_register_callbacks cb 0x7f725c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38472/0 (socket says 172.21.15.46:38472) 2024-03-08T23:10:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 -- 172.21.15.46:0/2260099315 learned_addr learned my addr 172.21.15.46:0/2260099315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f7263e00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 0x5580f99772e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f7263e00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f996fc60 0x5580f9975dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 -- 172.21.15.46:0/2260099315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f996fc60 msgr2=0x5580f9975dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f996fc60 0x5580f9975dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 -- 172.21.15.46:0/2260099315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 msgr2=0x5580f99772e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 0x5580f99772e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f725aa00640 1 -- 172.21.15.46:0/2260099315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580f9972050 con 0x5580f99629a0 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.614+0000 7f7263e00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 0x5580f99772e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f725aa00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5580f996def0 tx=0x7f724c00fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=660c3ea9439c48c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f7250c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c02d070 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f724c00f7d0 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f725c055790 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f724c022040 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f726cfca740 10 client.16131 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.615+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5580f99d7250 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c055790 con 0x5580f99629a0 2024-03-08T23:10:54.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f724c0038c0 con 0x5580f99629a0 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 10 client.16131 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 10 client.16131 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 20 client.16131 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f99d7a00 0x5580f99d9e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 4 client.16131 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5580f98d4310 con 0x5580f99d7a00 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.616+0000 7f7263e00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f99d7a00 0x5580f99d9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f7263e00640 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f99d7a00 0x5580f99d9e20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39c1ab184c89bd20 server_cookie=10becd8a7a559c5d in_seq=0 out_seq=0 2024-03-08T23:10:54.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f7250c00640 10 client.16131 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f7250c00640 1 -- 172.21.15.46:0/2260099315 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5580f98d4310 con 0x5580f99d7a00 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f7250c00640 10 client.16131 handle_command_reply: tid=0 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 client.16131 shutdown 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 msgr2=0x5580f9975680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5580f996def0 tx=0x7f724c00fd40 comp rx=0 tx=0).stop 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 shutdown_connections 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5580f99d7a00 0x5580f99d9e20 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580f996fc60 0x5580f9975dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5580f99629a0 0x5580f9975680 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f7250c00640 0 client.16131 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 --2- 172.21.15.46:0/2260099315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580f9962080 0x5580f99772e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.617+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 >> 172.21.15.46:0/2260099315 conn(0x5580f995ca80 msgr2=0x5580f9967f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 shutdown_connections 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726cfca740 1 -- 172.21.15.46:0/2260099315 wait complete. 2024-03-08T23:10:54.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726cfca740 20 client.16131 trim_cache size 0 max 16384 2024-03-08T23:10:54.630 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 msgr2=0x7f72480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 0x7f72480a4eb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x5580f9963260 tx=0x7f72540d2f90 comp rx=0 tx=0).stop 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 msgr2=0x7f726419d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f725400e9f0 tx=0x7f725400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 shutdown_connections 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f72480a29f0 0x7f72480a4eb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7264107340 0x7f7264197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f726406a1e0 0x7f726419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 --2- 172.21.15.46:0/259022283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7264069860 0x7f726419d7e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.618+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 >> 172.21.15.46:0/259022283 conn(0x7f72640771d0 msgr2=0x7f72640ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.619+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 shutdown_connections 2024-03-08T23:10:54.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.619+0000 7f726a200640 1 -- 172.21.15.46:0/259022283 wait complete. 2024-03-08T23:10:54.640 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:10:54.641 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:10:54.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 -- 172.21.15.46:0/1751616393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102320 msgr2=0x7faed8102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 --2- 172.21.15.46:0/1751616393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102320 0x7faed8102700 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7faecc009900 tx=0x7faecc02bcd0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 -- 172.21.15.46:0/1751616393 shutdown_connections 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 --2- 172.21.15.46:0/1751616393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 0x7faed810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 --2- 172.21.15.46:0/1751616393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102c40 0x7faed810b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 --2- 172.21.15.46:0/1751616393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102320 0x7faed8102700 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 -- 172.21.15.46:0/1751616393 >> 172.21.15.46:0/1751616393 conn(0x7faed80fbb40 msgr2=0x7faed80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 -- 172.21.15.46:0/1751616393 shutdown_connections 2024-03-08T23:10:54.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.855+0000 7faede800640 1 -- 172.21.15.46:0/1751616393 wait complete. 2024-03-08T23:10:54.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.856+0000 7faede800640 1 Processor -- start 2024-03-08T23:10:54.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 -- start start 2024-03-08T23:10:54.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 0x7faed80fff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 0x7faed81009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed8111570 con 0x7faed8102c40 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81113f0 con 0x7faed8102320 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.857+0000 7faede800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81116f0 con 0x7faed810b740 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38476/0 (socket says 172.21.15.46:38476) 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 -- 172.21.15.46:0/655117550 learned_addr learned my addr 172.21.15.46:0/655117550 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedde00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 0x7faed81009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedd400640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 0x7faed80fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 -- 172.21.15.46:0/655117550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 msgr2=0x7faed81009f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 0x7faed81009f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 -- 172.21.15.46:0/655117550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 msgr2=0x7faed80fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 0x7faed80fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 -- 172.21.15.46:0/655117550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed81a5f90 con 0x7faed8102c40 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedd400640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 0x7faed80fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.858+0000 7faedca00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faec400e9f0 tx=0x7faec400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.859+0000 7faed6000640 1 -- 172.21.15.46:0/655117550 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec400ccc0 con 0x7faed8102c40 2024-03-08T23:10:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.859+0000 7faed6000640 1 -- 172.21.15.46:0/655117550 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faec4010910 con 0x7faed8102c40 2024-03-08T23:10:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.859+0000 7faed6000640 1 -- 172.21.15.46:0/655117550 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4018920 con 0x7faed8102c40 2024-03-08T23:10:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.859+0000 7faede800640 1 -- 172.21.15.46:0/655117550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed81a6280 con 0x7faed8102c40 2024-03-08T23:10:54.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.859+0000 7faede800640 1 -- 172.21.15.46:0/655117550 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed81a6780 con 0x7faed8102c40 2024-03-08T23:10:54.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.862+0000 7faed6000640 1 -- 172.21.15.46:0/655117550 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7faec4010430 con 0x7faed8102c40 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 Processor -- start 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 -- start start 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadadede20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadad46dd0 con 0x55aadade2530 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadacefc70 con 0x55aadade2e50 2024-03-08T23:10:54.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faee2bc6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadad42e70 con 0x55aadadeba30 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faedde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faedde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38488/0 (socket says 172.21.15.46:38488) 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faedde00640 1 -- 172.21.15.46:0/2657806278 learned_addr learned my addr 172.21.15.46:0/2657806278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.863+0000 7faedca00640 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadadede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedd400640 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 -- 172.21.15.46:0/2657806278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 msgr2=0x55aadadede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadadede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 -- 172.21.15.46:0/2657806278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 msgr2=0x55aadadeb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faed6000640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 0x7faec00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 -- 172.21.15.46:0/2657806278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aadada92b0 con 0x55aadade2530 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faed6000640 1 -- 172.21.15.46:0/655117550 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7faec40cf3a0 con 0x7faed8102c40 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 0x7faec00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.864+0000 7faedde00640 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faed0009f50 tx=0x7faed00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e1a3a0b1f77b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.865+0000 7faeca600640 1 -- 172.21.15.46:0/2657806278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0004050 con 0x55aadade2530 2024-03-08T23:10:54.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.865+0000 7faeca600640 1 -- 172.21.15.46:0/2657806278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faed0004750 con 0x55aadade2530 2024-03-08T23:10:54.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.865+0000 7faeca600640 1 -- 172.21.15.46:0/2657806278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0019400 con 0x55aadade2530 2024-03-08T23:10:54.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- 172.21.15.46:0/2657806278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 msgr2=0x55aadade2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faed0009f50 tx=0x7faed00118d0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faedde00640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 0x7faec00a4eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faed0012fd0 tx=0x7faed0019680 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- 172.21.15.46:0/2657806278 shutdown_connections 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadadede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 --2- 172.21.15.46:0/2657806278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadade2910 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- 172.21.15.46:0/2657806278 >> 172.21.15.46:0/2657806278 conn(0x55aadaddcc60 msgr2=0x55aadaddd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- 172.21.15.46:0/2657806278 shutdown_connections 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- 172.21.15.46:0/2657806278 wait complete. 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.866+0000 7faee2bc6740 1 -- start start 2024-03-08T23:10:54.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 Processor -- start 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadade8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadade8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadadee820 con 0x55aadade2530 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadad030c0 con 0x55aadade2e50 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aadadee9a0 con 0x55aadadeba30 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faee2bc6740 10 client.0 _ll_register_callbacks cb 0x55aadad3dec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faedd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faedd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38502/0 (socket says 172.21.15.46:38502) 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faedd400640 1 -- 172.21.15.46:0/996687600 learned_addr learned my addr 172.21.15.46:0/996687600 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faedde00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadade8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.867+0000 7faedca00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadade8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 -- 172.21.15.46:0/996687600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 msgr2=0x55aadade8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadade8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 -- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 msgr2=0x55aadade8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadade8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 -- 172.21.15.46:0/996687600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aadaa5b3f0 con 0x55aadade2530 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedde00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadade8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedca00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadade8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faedd400640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faecc006010 tx=0x7faecc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e011e1a16ba45383 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.868+0000 7faebea00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc044070 con 0x55aadade2530 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7faecc004440 con 0x55aadade2530 2024-03-08T23:10:54.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faed0004750 con 0x55aadade2530 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faecc02c920 con 0x55aadade2530 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faee2bc6740 10 client.16143 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55aadab257d0 con 0x55aadade2530 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.869+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faed0004750 con 0x55aadade2530 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7faecc0037b0 con 0x55aadade2530 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 10 client.16143 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:54.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 20 client.16143 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadadfb9e0 0x55aadadfde10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 4 client.16143 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aadad1b4f0 con 0x55aadadfb9e0 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.870+0000 7faedca00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadadfb9e0 0x55aadadfde10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faedca00640 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadadfb9e0 0x55aadadfde10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcb81b3dc32c6f49 server_cookie=9fbb2e09f4acdc47 in_seq=0 out_seq=0 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faebea00640 10 client.16143 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faebea00640 1 -- 172.21.15.46:0/996687600 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aadad1b4f0 con 0x55aadadfb9e0 2024-03-08T23:10:54.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faebea00640 10 client.16143 handle_command_reply: tid=0 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faee2bc6740 1 client.16143 shutdown 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 msgr2=0x55aadadeb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.871+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faecc006010 tx=0x7faecc004040 comp rx=0 tx=0).stop 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 shutdown_connections 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadadfb9e0 0x55aadadfde10 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadadeba30 0x55aadade8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadade8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 --2- 172.21.15.46:0/996687600 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadeb2f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faebea00640 0 client.16143 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 >> 172.21.15.46:0/996687600 conn(0x55aadaddcc60 msgr2=0x55aadadde9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 shutdown_connections 2024-03-08T23:10:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 1 -- 172.21.15.46:0/996687600 wait complete. 2024-03-08T23:10:54.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.872+0000 7faee2bc6740 20 client.16143 trim_cache size 0 max 16384 2024-03-08T23:10:54.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.885+0000 7faee2bc6740 1 Processor -- start 2024-03-08T23:10:54.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.885+0000 7faee2bc6740 1 -- start start 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.885+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadadeba30 0x55aadadeb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faee2bc6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81b2b20 con 0x55aadadeba30 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faee2bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81b0660 con 0x55aadade2e50 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faee2bc6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81ae700 con 0x55aadade2530 2024-03-08T23:10:54.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadadeba30 0x55aadadeb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49316/0 (socket says 172.21.15.46:49316) 2024-03-08T23:10:54.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 -- 172.21.15.46:0/3992125791 learned_addr learned my addr 172.21.15.46:0/3992125791 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedca00640 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 -- 172.21.15.46:0/3992125791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 msgr2=0x55aadadeadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 -- 172.21.15.46:0/3992125791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadadeba30 msgr2=0x55aadadeb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadadeba30 0x55aadadeb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.886+0000 7faedd400640 1 -- 172.21.15.46:0/3992125791 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aadade2c80 con 0x55aadade2530 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.887+0000 7faedd400640 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faecc02c210 tx=0x7faecc004e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53ba32c8fbd0e595 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.887+0000 7faec9c00640 1 -- 172.21.15.46:0/3992125791 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc05a070 con 0x55aadade2530 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.887+0000 7faec9c00640 1 -- 172.21.15.46:0/3992125791 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faecc009520 con 0x55aadade2530 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.887+0000 7faec9c00640 1 -- 172.21.15.46:0/3992125791 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faecc044040 con 0x55aadade2530 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 -- 172.21.15.46:0/3992125791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 msgr2=0x55aadade7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7faecc02c210 tx=0x7faecc004e10 comp rx=0 tx=0).stop 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 -- 172.21.15.46:0/3992125791 shutdown_connections 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadadeba30 0x55aadadeb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadade2e50 0x55aadadeadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 --2- 172.21.15.46:0/3992125791 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2530 0x55aadade7660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 -- 172.21.15.46:0/3992125791 >> 172.21.15.46:0/3992125791 conn(0x55aadaddcc60 msgr2=0x55aadade6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 -- 172.21.15.46:0/3992125791 shutdown_connections 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.888+0000 7faee2bc6740 1 -- 172.21.15.46:0/3992125791 wait complete. 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 -- start start 2024-03-08T23:10:54.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 Processor -- start 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadf2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81aed00 con 0x55aadade2530 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81b0000 con 0x55aadadeba30 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.889+0000 7faee2bc6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed81af340 con 0x55aadade2e50 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faee2bc6740 10 client.0 _ll_register_callbacks cb 0x7faed0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:54.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadf2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49328/0 (socket says 172.21.15.46:49328) 2024-03-08T23:10:54.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 -- 172.21.15.46:0/563541862 learned_addr learned my addr 172.21.15.46:0/563541862 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:54.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40494/0 (socket says 172.21.15.46:40494) 2024-03-08T23:10:54.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 -- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 msgr2=0x55aadadf1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 -- 172.21.15.46:0/563541862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 msgr2=0x55aadadf2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadf2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 -- 172.21.15.46:0/563541862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aadadf1bf0 con 0x55aadade2e50 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedde00640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedd400640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadf2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faedca00640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faec40ccd90 tx=0x7faec400cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1b85b69d614c031 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.890+0000 7faebea00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:54.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.893+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec409b750 con 0x55aadade2e50 2024-03-08T23:10:54.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.893+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7faec409f520 con 0x55aadade2e50 2024-03-08T23:10:54.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.893+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faecc009520 con 0x55aadade2e50 2024-03-08T23:10:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.896+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7faec409f7f0 con 0x55aadade2e50 2024-03-08T23:10:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.896+0000 7faee2bc6740 10 client.26278 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.896+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55aadae52d80 con 0x55aadade2e50 2024-03-08T23:10:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faecc009520 con 0x55aadade2e50 2024-03-08T23:10:54.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7faec40cf080 con 0x55aadade2e50 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 10 client.26278 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 10 client.26278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 20 client.26278 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadae53510 0x55aadae55930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 4 client.26278 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55aadad54310 con 0x55aadae53510 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.897+0000 7faedd400640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadae53510 0x55aadae55930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:54.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faedd400640 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadae53510 0x55aadae55930 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efcfc383a31e96a2 server_cookie=a6a8047526dbe0f5 in_seq=0 out_seq=0 2024-03-08T23:10:54.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faebea00640 10 client.26278 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faebea00640 1 -- 172.21.15.46:0/563541862 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55aadad54310 con 0x55aadae53510 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faebea00640 10 client.26278 handle_command_reply: tid=0 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 client.26278 shutdown 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 msgr2=0x55aadadf10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7faec40ccd90 tx=0x7faec400cf60 comp rx=0 tx=0).stop 2024-03-08T23:10:54.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 shutdown_connections 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55aadae53510 0x55aadae55930 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55aadadeba30 0x55aadadf1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55aadade2e50 0x55aadadf10c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 --2- 172.21.15.46:0/563541862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55aadade2530 0x55aadadf2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faebea00640 0 client.26278 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.898+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 >> 172.21.15.46:0/563541862 conn(0x55aadaddcc60 msgr2=0x55aadade77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 shutdown_connections 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faee2bc6740 1 -- 172.21.15.46:0/563541862 wait complete. 2024-03-08T23:10:54.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faee2bc6740 20 client.26278 trim_cache size 0 max 16384 2024-03-08T23:10:54.906 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 -- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 msgr2=0x7faec00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 0x7faec00a4eb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faed0012fd0 tx=0x7faed0019680 comp rx=0 tx=0).stop 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 -- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 msgr2=0x7faed81004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7faec400e9f0 tx=0x7faec400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 -- 172.21.15.46:0/655117550 shutdown_connections 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7faec00a29f0 0x7faec00a4eb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.899+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7faed810b740 0x7faed81009f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.900+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7faed8102c40 0x7faed81004b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.900+0000 7faede800640 1 --2- 172.21.15.46:0/655117550 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7faed8102320 0x7faed80fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.900+0000 7faede800640 1 -- 172.21.15.46:0/655117550 >> 172.21.15.46:0/655117550 conn(0x7faed80fbb40 msgr2=0x7faed80fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.900+0000 7faede800640 1 -- 172.21.15.46:0/655117550 shutdown_connections 2024-03-08T23:10:54.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:54.900+0000 7faede800640 1 -- 172.21.15.46:0/655117550 wait complete. 2024-03-08T23:10:55.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/848751334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:55.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/848751334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:55.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/848751334' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:10:55.921 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:10:55.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:10:55.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 -- 172.21.15.46:0/26138212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 msgr2=0x7f819c103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 --2- 172.21.15.46:0/26138212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c103930 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8184009930 tx=0x7f818402bc30 comp rx=0 tx=0).stop 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 -- 172.21.15.46:0/26138212 shutdown_connections 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 --2- 172.21.15.46:0/26138212 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 --2- 172.21.15.46:0/26138212 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c103930 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 --2- 172.21.15.46:0/26138212 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 0x7f819c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 -- 172.21.15.46:0/26138212 >> 172.21.15.46:0/26138212 conn(0x7f819c0fef40 msgr2=0x7f819c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 -- 172.21.15.46:0/26138212 shutdown_connections 2024-03-08T23:10:56.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.160+0000 7f81a1a00640 1 -- 172.21.15.46:0/26138212 wait complete. 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.161+0000 7f81a1a00640 1 Processor -- start 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.161+0000 7f81a1a00640 1 -- start start 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 0x7f819c19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c113880 con 0x7f819c103e70 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c113700 con 0x7f819c109570 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f81a1a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c113a00 con 0x7f819c1034b0 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f8193e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f819aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f819aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49348/0 (socket says 172.21.15.46:49348) 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f8193e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38532/0 (socket says 172.21.15.46:38532) 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f819aa00640 1 -- 172.21.15.46:0/2002741403 learned_addr learned my addr 172.21.15.46:0/2002741403 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.162+0000 7f819b400640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 0x7f819c19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 -- 172.21.15.46:0/2002741403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 msgr2=0x7f819c199e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c199e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 -- 172.21.15.46:0/2002741403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 msgr2=0x7f819c19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 0x7f819c19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 -- 172.21.15.46:0/2002741403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c19ede0 con 0x7f819c103e70 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193e00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8184036040 tx=0x7f8184005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193400640 1 -- 172.21.15.46:0/2002741403 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184043020 con 0x7f819c103e70 2024-03-08T23:10:56.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c19f070 con 0x7f819c103e70 2024-03-08T23:10:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.163+0000 7f8193400640 1 -- 172.21.15.46:0/2002741403 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8184005000 con 0x7f819c103e70 2024-03-08T23:10:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.164+0000 7f8193400640 1 -- 172.21.15.46:0/2002741403 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8184003af0 con 0x7f819c103e70 2024-03-08T23:10:56.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.164+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c1a6960 con 0x7f819c103e70 2024-03-08T23:10:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 Processor -- start 2024-03-08T23:10:56.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 -- start start 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f8193400640 1 -- 172.21.15.46:0/2002741403 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8184046020 con 0x7f819c103e70 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c688085a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c68808ae0 0x564c6880aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c68763a50 con 0x564c687ff5e0 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6870cb00 con 0x564c68808ae0 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.167+0000 7f81a47f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6875faf0 con 0x564c687fff00 2024-03-08T23:10:56.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f8193e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c68808ae0 0x564c6880aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38540/0 (socket says 172.21.15.46:38540) 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 -- 172.21.15.46:0/1847136643 learned_addr learned my addr 172.21.15.46:0/1847136643 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819aa00640 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c688085a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 -- 172.21.15.46:0/1847136643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 msgr2=0x564c688085a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c688085a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 -- 172.21.15.46:0/1847136643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c68808ae0 msgr2=0x564c6880aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c68808ae0 0x564c6880aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.168+0000 7f819b400640 1 -- 172.21.15.46:0/1847136643 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c6872a1e0 con 0x564c687ff5e0 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f819b400640 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8194009fd0 tx=0x7f81940118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b679cd5314163f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f8193400640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 0x7f81800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f818be00640 1 -- 172.21.15.46:0/1847136643 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194004d20 con 0x564c687ff5e0 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f818be00640 1 -- 172.21.15.46:0/1847136643 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8194004050 con 0x564c687ff5e0 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f818be00640 1 -- 172.21.15.46:0/1847136643 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194005280 con 0x564c687ff5e0 2024-03-08T23:10:56.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f8193400640 1 -- 172.21.15.46:0/2002741403 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f81840f0fb0 con 0x7f819c103e70 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.169+0000 7f819aa00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 0x7f81800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- 172.21.15.46:0/1847136643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 msgr2=0x564c687ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8194009fd0 tx=0x7f81940118d0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f819aa00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 0x7f81800a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x564c687ff0b0 tx=0x7f818c005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- 172.21.15.46:0/1847136643 shutdown_connections 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c68808ae0 0x564c6880aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c688085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1847136643 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687ff5e0 0x564c687ff9c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- 172.21.15.46:0/1847136643 >> 172.21.15.46:0/1847136643 conn(0x564c687f9d10 msgr2=0x564c687fa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- 172.21.15.46:0/1847136643 shutdown_connections 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- 172.21.15.46:0/1847136643 wait complete. 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.170+0000 7f81a47f3740 1 -- start start 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 Processor -- start 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c68805b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c68806040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6880b8d0 con 0x564c68808ae0 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6871ff60 con 0x564c687ff5e0 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564c6880ba50 con 0x564c687fff00 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f81a47f3740 10 client.0 _ll_register_callbacks cb 0x564c6870bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38548/0 (socket says 172.21.15.46:38548) 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f819b400640 1 -- 172.21.15.46:0/3257006302 learned_addr learned my addr 172.21.15.46:0/3257006302 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.171+0000 7f819aa00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c68805b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f8193e00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c68806040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 -- 172.21.15.46:0/3257006302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 msgr2=0x564c68806040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c68806040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 -- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 msgr2=0x564c68805b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c68805b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 -- 172.21.15.46:0/3257006302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c685407e0 con 0x564c68808ae0 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f8193e00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c68806040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819aa00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c68805b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:56.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f819b400640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8194005c10 tx=0x7f8194004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7128bfb60fe348ed server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.172+0000 7f8188c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194029020 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f819400f4e0 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8194004050 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8194034050 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f81a47f3740 10 client.16161 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564c685a0f70 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8194004050 con 0x564c68808ae0 2024-03-08T23:10:56.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8194025490 con 0x564c68808ae0 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f81a47f3740 10 client.16161 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.173+0000 7f81a47f3740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f81a47f3740 20 client.16161 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68818ed0 0x564c6881b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f81a47f3740 4 client.16161 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564c68770f90 con 0x564c68818ed0 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f819aa00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68818ed0 0x564c6881b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f819aa00640 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68818ed0 0x564c6881b2c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc266f9c0e3be547 server_cookie=77d93f2bd02189e6 in_seq=0 out_seq=0 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.174+0000 7f8188c00640 10 client.16161 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f8188c00640 1 -- 172.21.15.46:0/3257006302 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564c68770f90 con 0x564c68818ed0 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f8188c00640 10 client.16161 handle_command_reply: tid=0 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f81a47f3740 1 client.16161 shutdown 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 msgr2=0x564c68806580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8194005c10 tx=0x7f8194004430 comp rx=0 tx=0).stop 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 shutdown_connections 2024-03-08T23:10:56.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.175+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68818ed0 0x564c6881b2c0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c68808ae0 0x564c68806580 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f8188c00640 0 client.16161 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c68806040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3257006302 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c68805b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 >> 172.21.15.46:0/3257006302 conn(0x564c687f9d10 msgr2=0x564c687fba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 shutdown_connections 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 1 -- 172.21.15.46:0/3257006302 wait complete. 2024-03-08T23:10:56.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.176+0000 7f81a47f3740 20 client.16161 trim_cache size 0 max 16384 2024-03-08T23:10:56.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.192+0000 7f81a47f3740 1 Processor -- start 2024-03-08T23:10:56.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 -- start start 2024-03-08T23:10:56.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687ff5e0 0x564c688043d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 0x564c68804910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1b2d20 con 0x564c687fff00 2024-03-08T23:10:56.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1b0860 con 0x564c687f9d10 2024-03-08T23:10:56.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f81a47f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1ae900 con 0x564c687ff5e0 2024-03-08T23:10:56.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 0x564c68804910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f819aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f819b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 0x564c68804910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38560/0 (socket says 172.21.15.46:38560) 2024-03-08T23:10:56.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f8193e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687ff5e0 0x564c688043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f819aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40558/0 (socket says 172.21.15.46:40558) 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.193+0000 7f819b400640 1 -- 172.21.15.46:0/3196373563 learned_addr learned my addr 172.21.15.46:0/3196373563 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 -- 172.21.15.46:0/3196373563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687ff5e0 msgr2=0x564c688043d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687ff5e0 0x564c688043d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 -- 172.21.15.46:0/3196373563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 msgr2=0x564c68804910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 0x564c68804910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 -- 172.21.15.46:0/3196373563 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564c687ffd30 con 0x564c687f9d10 2024-03-08T23:10:56.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f819aa00640 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x564c6881ba10 tx=0x7f818c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21e9df32fad243d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f818b400640 1 -- 172.21.15.46:0/3196373563 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c02a520 con 0x564c687f9d10 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f818b400640 1 -- 172.21.15.46:0/3196373563 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f818c02a6c0 con 0x564c687f9d10 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.194+0000 7f818b400640 1 -- 172.21.15.46:0/3196373563 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f818c01e650 con 0x564c687f9d10 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 -- 172.21.15.46:0/3196373563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 msgr2=0x564c68808230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x564c6881ba10 tx=0x7f818c01b000 comp rx=0 tx=0).stop 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 -- 172.21.15.46:0/3196373563 shutdown_connections 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687fff00 0x564c68804910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687ff5e0 0x564c688043d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 --2- 172.21.15.46:0/3196373563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687f9d10 0x564c68808230 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 -- 172.21.15.46:0/3196373563 >> 172.21.15.46:0/3196373563 conn(0x564c6881b630 msgr2=0x564c68800e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 -- 172.21.15.46:0/3196373563 shutdown_connections 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.195+0000 7f81a47f3740 1 -- 172.21.15.46:0/3196373563 wait complete. 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 -- start start 2024-03-08T23:10:56.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 Processor -- start 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c6880eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c6880f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1aef00 con 0x564c687f9d10 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1b0200 con 0x564c687ff5e0 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c1af540 con 0x564c687fff00 2024-03-08T23:10:56.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.196+0000 7f81a47f3740 10 client.0 _ll_register_callbacks cb 0x7f8194009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f819aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f819aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38564/0 (socket says 172.21.15.46:38564) 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f819aa00640 1 -- 172.21.15.46:0/1319935361 learned_addr learned my addr 172.21.15.46:0/1319935361 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c6880eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f819b400640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c6880f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 -- 172.21.15.46:0/1319935361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 msgr2=0x564c6880f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c6880f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 -- 172.21.15.46:0/1319935361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 msgr2=0x564c6880e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 -- 172.21.15.46:0/1319935361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564c68438170 con 0x564c687ff5e0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f819aa00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8193e00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c6880eb80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f81840ee450 tx=0x7f8184000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c683ea2462f57685 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.197+0000 7f8188c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81840dbe60 con 0x564c687ff5e0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f818c02a6c0 con 0x564c687ff5e0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f81840c1170 con 0x564c687ff5e0 2024-03-08T23:10:56.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81840bcea0 con 0x564c687ff5e0 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 10 client.26294 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x564c6872f050 con 0x564c687ff5e0 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f818c02a6c0 con 0x564c687ff5e0 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f81840ee5c0 con 0x564c687ff5e0 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 10 client.26294 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 10 client.26294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 20 client.26294 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:56.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68808ae0 0x564c68872900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 4 client.26294 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.198+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564c6876b2e0 con 0x564c68808ae0 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.199+0000 7f819aa00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68808ae0 0x564c68872900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.199+0000 7f819aa00640 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68808ae0 0x564c68872900 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=196da26073c8ad21 server_cookie=4266363aff4d28c3 in_seq=0 out_seq=0 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.199+0000 7f8188c00640 10 client.26294 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.200+0000 7f8188c00640 1 -- 172.21.15.46:0/1319935361 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564c6876b2e0 con 0x564c68808ae0 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.200+0000 7f8188c00640 10 client.26294 handle_command_reply: tid=0 2024-03-08T23:10:56.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.200+0000 7f81a47f3740 1 client.26294 shutdown 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 msgr2=0x564c6880eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c6880eb80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f81840ee450 tx=0x7f8184000bc0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 shutdown_connections 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564c68808ae0 0x564c68872900 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564c687fff00 0x564c6880f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564c687ff5e0 0x564c6880eb80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 --2- 172.21.15.46:0/1319935361 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564c687f9d10 0x564c6880e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f8188c00640 0 client.26294 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 >> 172.21.15.46:0/1319935361 conn(0x564c6881b630 msgr2=0x564c688042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 shutdown_connections 2024-03-08T23:10:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 1 -- 172.21.15.46:0/1319935361 wait complete. 2024-03-08T23:10:56.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.201+0000 7f81a47f3740 20 client.26294 trim_cache size 0 max 16384 2024-03-08T23:10:56.212 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:56.213 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:10:56.213 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "303b546f-f81e-4a1b-a4a7-d218ba50d273", 2024-03-08T23:10:56.213 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:10:56.213 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 msgr2=0x7f81800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 0x7f81800a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x564c687ff0b0 tx=0x7f818c005f90 comp rx=0 tx=0).stop 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 msgr2=0x7f819c19a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8184036040 tx=0x7f8184005e80 comp rx=0 tx=0).stop 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 shutdown_connections 2024-03-08T23:10:56.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f81800a2970 0x7f81800a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f819c109570 0x7f819c19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f819c103e70 0x7f819c19a380 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 --2- 172.21.15.46:0/2002741403 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f819c1034b0 0x7f819c199e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 >> 172.21.15.46:0/2002741403 conn(0x7f819c0fef40 msgr2=0x7f819c100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 shutdown_connections 2024-03-08T23:10:56.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.202+0000 7f81a1a00640 1 -- 172.21.15.46:0/2002741403 wait complete. 2024-03-08T23:10:56.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:55 smithi046 ceph-mon[31859]: pgmap v97: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:56.229 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:10:56.229 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:10:56.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:55 smithi192 ceph-mon[36700]: pgmap v97: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:56.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:55 smithi062 ceph-mon[35761]: pgmap v97: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:56.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 -- 172.21.15.46:0/3422073459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc780109560 msgr2=0x7fc780109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 --2- 172.21.15.46:0/3422073459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc780109560 0x7fc780109940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc778009cd0 tx=0x7fc77802c100 comp rx=0 tx=0).stop 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 -- 172.21.15.46:0/3422073459 shutdown_connections 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 --2- 172.21.15.46:0/3422073459 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 0x7fc780110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 --2- 172.21.15.46:0/3422073459 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7801034a0 0x7fc780103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 --2- 172.21.15.46:0/3422073459 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc780109560 0x7fc780109940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 -- 172.21.15.46:0/3422073459 >> 172.21.15.46:0/3422073459 conn(0x7fc7800fef70 msgr2=0x7fc780101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 -- 172.21.15.46:0/3422073459 shutdown_connections 2024-03-08T23:10:56.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.496+0000 7fc786a00640 1 -- 172.21.15.46:0/3422073459 wait complete. 2024-03-08T23:10:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.497+0000 7fc786a00640 1 Processor -- start 2024-03-08T23:10:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.497+0000 7fc786a00640 1 -- start start 2024-03-08T23:10:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 0x7fc78019f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc780109560 0x7fc7801a9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780077510 con 0x7fc780109560 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780077390 con 0x7fc780103e60 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc786a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc780077690 con 0x7fc7801034a0 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49396/0 (socket says 172.21.15.46:49396) 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc77fe00640 1 -- 172.21.15.46:0/3830509886 learned_addr learned my addr 172.21.15.46:0/3830509886 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.498+0000 7fc77f400640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 0x7fc78019f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc784a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc780109560 0x7fc7801a9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 -- 172.21.15.46:0/3830509886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 msgr2=0x7fc78019f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 0x7fc78019f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 -- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc780109560 msgr2=0x7fc7801a9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc780109560 0x7fc7801a9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 -- 172.21.15.46:0/3830509886 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7801aa520 con 0x7fc7801034a0 2024-03-08T23:10:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.499+0000 7fc77fe00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc77802c710 tx=0x7fc778035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.500+0000 7fc77cc00640 1 -- 172.21.15.46:0/3830509886 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc778039040 con 0x7fc7801034a0 2024-03-08T23:10:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.500+0000 7fc77cc00640 1 -- 172.21.15.46:0/3830509886 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc778044070 con 0x7fc7801034a0 2024-03-08T23:10:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.500+0000 7fc77cc00640 1 -- 172.21.15.46:0/3830509886 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7780043a0 con 0x7fc7801034a0 2024-03-08T23:10:56.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.500+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7801aa6f0 con 0x7fc7801034a0 2024-03-08T23:10:56.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.501+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7801aab00 con 0x7fc7801034a0 2024-03-08T23:10:56.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.505+0000 7fc789681740 1 Processor -- start 2024-03-08T23:10:56.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.505+0000 7fc789681740 1 -- start start 2024-03-08T23:10:56.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ff580 0x561adf4ff960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf508540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc77cc00640 1 -- 172.21.15.46:0/3830509886 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc778004540 con 0x7fc7801034a0 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf4c1410 con 0x561adf4ffea0 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf463d50 con 0x561adf4ff580 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc789681740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf40cb80 con 0x561adf508a80 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc77f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc77f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49400/0 (socket says 172.21.15.46:49400) 2024-03-08T23:10:56.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.506+0000 7fc77f400640 1 -- 172.21.15.46:0/3909929043 learned_addr learned my addr 172.21.15.46:0/3909929043 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77fe00640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf508540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc784a00640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ff580 0x561adf4ff960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 -- 172.21.15.46:0/3909929043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ff580 msgr2=0x561adf4ff960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ff580 0x561adf4ff960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 -- 172.21.15.46:0/3909929043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 msgr2=0x561adf508540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf508540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 -- 172.21.15.46:0/3909929043 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561adf2427d0 con 0x561adf508a80 2024-03-08T23:10:56.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77fe00640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf508540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:10:56.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc77f400640 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009c90 tx=0x7fc76c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7418521e810f17a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc775200640 1 -- 172.21.15.46:0/3909929043 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c01e070 con 0x561adf508a80 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc775200640 1 -- 172.21.15.46:0/3909929043 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc76c019b10 con 0x561adf508a80 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.507+0000 7fc775200640 1 -- 172.21.15.46:0/3909929043 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c00ea80 con 0x561adf508a80 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.508+0000 7fc77cc00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 0x7fc7640a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 -- 172.21.15.46:0/3909929043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 msgr2=0x561adf50ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009c90 tx=0x7fc76c011810 comp rx=0 tx=0).stop 2024-03-08T23:10:56.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc77cc00640 1 -- 172.21.15.46:0/3830509886 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc7780f1500 con 0x7fc7801034a0 2024-03-08T23:10:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 -- 172.21.15.46:0/3909929043 shutdown_connections 2024-03-08T23:10:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf50ae70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf508540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 --2- 172.21.15.46:0/3909929043 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ff580 0x561adf4ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc789681740 1 -- 172.21.15.46:0/3909929043 >> 172.21.15.46:0/3909929043 conn(0x561adf4f9cb0 msgr2=0x561adf4fa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.509+0000 7fc784a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 0x7fc7640a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.510+0000 7fc789681740 1 -- 172.21.15.46:0/3909929043 shutdown_connections 2024-03-08T23:10:56.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.510+0000 7fc789681740 1 -- 172.21.15.46:0/3909929043 wait complete. 2024-03-08T23:10:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.510+0000 7fc784a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 0x7fc7640a5000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc770005fd0 tx=0x7fc77000e480 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.510+0000 7fc789681740 1 -- start start 2024-03-08T23:10:56.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.511+0000 7fc789681740 1 Processor -- start 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.511+0000 7fc789681740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.511+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 0x561adf505f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38604/0 (socket says 172.21.15.46:38604) 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf506480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf41fee0 con 0x561adf4ff580 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77fe00640 1 -- 172.21.15.46:0/62098706 learned_addr learned my addr 172.21.15.46:0/62098706 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf45fdf0 con 0x561adf4ffea0 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561adf50b8c0 con 0x561adf508a80 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc789681740 10 client.0 _ll_register_callbacks cb 0x561adf4268d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 0x561adf505f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc784a00640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf506480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 -- 172.21.15.46:0/62098706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 msgr2=0x561adf506480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf506480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 -- 172.21.15.46:0/62098706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 msgr2=0x561adf505a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77f400640 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561adf1783f0 con 0x561adf4ffea0 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc784a00640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf506480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.512+0000 7fc77fe00640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.513+0000 7fc77f400640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 0x561adf505f40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x561adf4fae40 tx=0x7fc76c019a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b21cd4f83d65b25 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.513+0000 7fc769600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.513+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c01e050 con 0x561adf4ffea0 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.513+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc76c005450 con 0x561adf4ffea0 2024-03-08T23:10:56.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.513+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc76c019b10 con 0x561adf4ffea0 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc76c033050 con 0x561adf4ffea0 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 10 client.26306 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561adf1380f0 con 0x561adf4ffea0 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc76c019b10 con 0x561adf4ffea0 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc76c004750 con 0x561adf4ffea0 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 10 client.26306 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 10 client.26306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:56.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 20 client.26306 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf518af0 0x561adf51aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.514+0000 7fc789681740 4 client.26306 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:10:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.515+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561adf471290 con 0x561adf518af0 2024-03-08T23:10:56.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.515+0000 7fc77fe00640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf518af0 0x561adf51aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.515+0000 7fc77fe00640 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf518af0 0x561adf51aee0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f9d1b5be86c5b73 server_cookie=8cdbe62a3e5a622b in_seq=0 out_seq=0 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.515+0000 7fc769600640 10 client.26306 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.516+0000 7fc769600640 1 -- 172.21.15.46:0/62098706 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561adf471290 con 0x561adf518af0 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.516+0000 7fc769600640 10 client.26306 handle_command_reply: tid=0 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.516+0000 7fc789681740 1 client.26306 shutdown 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 msgr2=0x561adf505f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 0x561adf505f40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x561adf4fae40 tx=0x7fc76c019a60 comp rx=0 tx=0).stop 2024-03-08T23:10:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 shutdown_connections 2024-03-08T23:10:56.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf518af0 0x561adf51aee0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf508a80 0x561adf506480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf4ffea0 0x561adf505f40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 --2- 172.21.15.46:0/62098706 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf505a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 >> 172.21.15.46:0/62098706 conn(0x561adf4f9cb0 msgr2=0x561adf4fba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc769600640 0 client.26306 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 shutdown_connections 2024-03-08T23:10:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 1 -- 172.21.15.46:0/62098706 wait complete. 2024-03-08T23:10:56.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.517+0000 7fc789681740 20 client.26306 trim_cache size 0 max 16384 2024-03-08T23:10:56.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.545+0000 7fc789681740 1 Processor -- start 2024-03-08T23:10:56.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.545+0000 7fc789681740 1 -- start start 2024-03-08T23:10:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf50fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf510070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b7000 con 0x561adf4ffea0 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b4b40 con 0x561adf508a80 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc789681740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b2be0 con 0x561adf4ff580 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc77fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49414/0 (socket says 172.21.15.46:49414) 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc77fe00640 1 -- 172.21.15.46:0/2574771527 learned_addr learned my addr 172.21.15.46:0/2574771527 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc77f400640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf50fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.546+0000 7fc784a00640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf510070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 -- 172.21.15.46:0/2574771527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 msgr2=0x561adf510070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf510070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 -- 172.21.15.46:0/2574771527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 msgr2=0x561adf50fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf50fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 -- 172.21.15.46:0/2574771527 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561adf4ffc90 con 0x561adf4ff580 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77f400640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf50fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc77fe00640 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7780b9d10 tx=0x7fc7780f3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6c38ce965a500a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc76be00640 1 -- 172.21.15.46:0/2574771527 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc778004830 con 0x561adf4ff580 2024-03-08T23:10:56.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc76be00640 1 -- 172.21.15.46:0/2574771527 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7780bdfb0 con 0x561adf4ff580 2024-03-08T23:10:56.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.547+0000 7fc76be00640 1 -- 172.21.15.46:0/2574771527 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7780bebe0 con 0x561adf4ff580 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.548+0000 7fc789681740 1 -- 172.21.15.46:0/2574771527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 msgr2=0x561adf508130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.548+0000 7fc789681740 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc7780b9d10 tx=0x7fc7780f3040 comp rx=0 tx=0).stop 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 -- 172.21.15.46:0/2574771527 shutdown_connections 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf510070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ffea0 0x561adf50fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 --2- 172.21.15.46:0/2574771527 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ff580 0x561adf508130 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 -- 172.21.15.46:0/2574771527 >> 172.21.15.46:0/2574771527 conn(0x561adf4f9cb0 msgr2=0x561adf51aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 -- 172.21.15.46:0/2574771527 shutdown_connections 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 -- 172.21.15.46:0/2574771527 wait complete. 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.549+0000 7fc789681740 1 -- start start 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.550+0000 7fc789681740 1 Processor -- start 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf568580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf562e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b31e0 con 0x561adf4ff580 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b44e0 con 0x561adf508a80 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7801b3820 con 0x561adf4ffea0 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc789681740 10 client.0 _ll_register_callbacks cb 0x7fc76c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc77f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc784a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf562e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc77f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49416/0 (socket says 172.21.15.46:49416) 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc77f400640 1 -- 172.21.15.46:0/4183717850 learned_addr learned my addr 172.21.15.46:0/4183717850 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.551+0000 7fc77fe00640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf568580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 -- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 msgr2=0x561adf562e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf562e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 -- 172.21.15.46:0/4183717850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 msgr2=0x561adf568580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf568580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 -- 172.21.15.46:0/4183717850 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561adf563570 con 0x561adf4ffea0 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77fe00640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf568580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc77f400640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc76c004b20 tx=0x7fc76c004d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c59c399010a0aed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:10:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.552+0000 7fc769600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:10:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc76c01e040 con 0x561adf4ffea0 2024-03-08T23:10:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7780bdfb0 con 0x561adf4ffea0 2024-03-08T23:10:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc76c005460 con 0x561adf4ffea0 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc76c019ad0 con 0x561adf4ffea0 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc789681740 10 client.26311 fetch_fsmap learned FSMap version 27 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.553+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x561adf56fdb0 con 0x561adf4ffea0 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.554+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7780bdfb0 con 0x561adf4ffea0 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc76c025590 con 0x561adf4ffea0 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 10 client.26311 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:10:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 10 client.26311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 20 client.26311 populate_metadata read hostname 'smithi046' 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf570500 0x561adf572920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 4 client.26311 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561adf46b5e0 con 0x561adf570500 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.555+0000 7fc77fe00640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf570500 0x561adf572920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc77fe00640 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf570500 0x561adf572920 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a651735fc197efe5 server_cookie=2a023bccc93f9d00 in_seq=0 out_seq=0 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc769600640 10 client.26311 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc769600640 1 -- 172.21.15.46:0/4183717850 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x561adf46b5e0 con 0x561adf570500 2024-03-08T23:10:56.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc769600640 10 client.26311 handle_command_reply: tid=0 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 client.26311 shutdown 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 msgr2=0x561adf568ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc76c004b20 tx=0x7fc76c004d90 comp rx=0 tx=0).stop 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 shutdown_connections 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561adf570500 0x561adf572920 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561adf508a80 0x561adf562e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561adf4ffea0 0x561adf568ac0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 --2- 172.21.15.46:0/4183717850 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561adf4ff580 0x561adf568580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 >> 172.21.15.46:0/4183717850 conn(0x561adf4f9cb0 msgr2=0x561adf500cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.556+0000 7fc769600640 0 client.26311 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.557+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 shutdown_connections 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.557+0000 7fc789681740 1 -- 172.21.15.46:0/4183717850 wait complete. 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.557+0000 7fc789681740 20 client.26311 trim_cache size 0 max 16384 2024-03-08T23:10:56.565 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "303b546f-f81e-4a1b-a4a7-d218ba50d273": { 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "303b546f-f81e-4a1b-a4a7-d218ba50d273", 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:10:56.566 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:10:56.567 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:10:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 msgr2=0x7fc7640a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 0x7fc7640a5000 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc770005fd0 tx=0x7fc77000e480 comp rx=0 tx=0).stop 2024-03-08T23:10:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 msgr2=0x7fc78019ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:10:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc77802c710 tx=0x7fc778035d10 comp rx=0 tx=0).stop 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 shutdown_connections 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc7640a2b40 0x7fc7640a5000 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc780109560 0x7fc7801a9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc780103e60 0x7fc78019f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 --2- 172.21.15.46:0/3830509886 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7801034a0 0x7fc78019ef60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 >> 172.21.15.46:0/3830509886 conn(0x7fc7800fef70 msgr2=0x7fc780100b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 shutdown_connections 2024-03-08T23:10:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:10:56.558+0000 7fc786a00640 1 -- 172.21.15.46:0/3830509886 wait complete. 2024-03-08T23:10:56.583 INFO:tasks.cephfs.filesystem:scrub status for tag:303b546f-f81e-4a1b-a4a7-d218ba50d273 - {'path': '/', 'tag': '303b546f-f81e-4a1b-a4a7-d218ba50d273', 'options': 'recursive,force'} 2024-03-08T23:10:57.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:10:57.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.215 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:56 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:10:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:10:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:56 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:10:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:10:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:57.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:56 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:10:58.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:57 smithi046 ceph-mon[31859]: pgmap v98: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:57 smithi192 ceph-mon[36700]: pgmap v98: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:57 smithi062 ceph-mon[35761]: pgmap v98: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 9.1 KiB/s wr, 6 op/s 2024-03-08T23:10:59.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:10:59.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:10:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:11:00.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:10:59 smithi046 ceph-mon[31859]: pgmap v99: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-03-08T23:11:00.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:10:59 smithi192 ceph-mon[36700]: pgmap v99: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-03-08T23:11:00.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:10:59 smithi062 ceph-mon[35761]: pgmap v99: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 17 KiB/s wr, 10 op/s 2024-03-08T23:11:01.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:00 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:11:01.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:00 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:11:01.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:00 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:11:01.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:00 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:11:01.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:00 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:11:01.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:00 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:11:01.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:00 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:11:01.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:00 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:11:01.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:00 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:11:02.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:02 smithi192 ceph-mon[36700]: pgmap v100: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-03-08T23:11:02.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:02 smithi062 ceph-mon[35761]: pgmap v100: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-03-08T23:11:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:02 smithi046 ceph-mon[31859]: pgmap v100: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 14 KiB/s wr, 7 op/s 2024-03-08T23:11:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:04 smithi192 ceph-mon[36700]: pgmap v101: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-03-08T23:11:04.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:04 smithi062 ceph-mon[35761]: pgmap v101: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-03-08T23:11:04.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:04 smithi046 ceph-mon[31859]: pgmap v101: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 36 KiB/s wr, 24 op/s 2024-03-08T23:11:06.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:06 smithi192 ceph-mon[36700]: pgmap v102: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:06.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:06 smithi062 ceph-mon[35761]: pgmap v102: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:06.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:06 smithi046 ceph-mon[31859]: pgmap v102: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:07.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:07 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:07.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:07 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:07.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:07 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:07.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:07 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:07.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:07 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:07.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:07 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:08 smithi192 ceph-mon[36700]: pgmap v103: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:08 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:08.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:08 smithi062 ceph-mon[35761]: pgmap v103: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:08.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:08 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:08.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:08 smithi046 ceph-mon[31859]: pgmap v103: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 30 KiB/s wr, 21 op/s 2024-03-08T23:11:08.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:08 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:10.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:10 smithi192 ceph-mon[36700]: pgmap v104: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-03-08T23:11:10.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:10 smithi062 ceph-mon[35761]: pgmap v104: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-03-08T23:11:10.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:10 smithi046 ceph-mon[31859]: pgmap v104: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 30 KiB/s wr, 24 op/s 2024-03-08T23:11:12.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:12 smithi192 ceph-mon[36700]: pgmap v105: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:11:12.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:12 smithi062 ceph-mon[35761]: pgmap v105: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:11:12.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:12 smithi046 ceph-mon[31859]: pgmap v105: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:11:14.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:14 smithi192 ceph-mon[36700]: pgmap v106: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:11:14.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:14 smithi062 ceph-mon[35761]: pgmap v106: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:11:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:14 smithi046 ceph-mon[31859]: pgmap v106: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:11:16.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:16 smithi062 ceph-mon[35761]: pgmap v107: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:16.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:16 smithi046 ceph-mon[31859]: pgmap v107: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:16.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:16 smithi192 ceph-mon[36700]: pgmap v107: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:18.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:18 smithi062 ceph-mon[35761]: pgmap v108: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:18.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:18 smithi046 ceph-mon[31859]: pgmap v108: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:18.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:18 smithi192 ceph-mon[36700]: pgmap v108: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-03-08T23:11:20.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:20 smithi062 ceph-mon[35761]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:11:20.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:20 smithi046 ceph-mon[31859]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:11:20.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:20 smithi192 ceph-mon[36700]: pgmap v109: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 170 B/s wr, 7 op/s 2024-03-08T23:11:22.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:22 smithi062 ceph-mon[35761]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-08T23:11:22.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:22 smithi046 ceph-mon[31859]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-08T23:11:22.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:22 smithi192 ceph-mon[36700]: pgmap v110: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-08T23:11:24.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:24 smithi062 ceph-mon[35761]: pgmap v111: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:11:24.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:24 smithi046 ceph-mon[31859]: pgmap v111: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:11:24.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:24 smithi192 ceph-mon[36700]: pgmap v111: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:11:26.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:26 smithi062 ceph-mon[35761]: pgmap v112: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:26.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:26 smithi046 ceph-mon[31859]: pgmap v112: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:26.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:26 smithi192 ceph-mon[36700]: pgmap v112: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:26.584 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:11:26.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:11:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 -- 172.21.15.46:0/2205490632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 msgr2=0x7f3bac1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2205490632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac1017b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009900 tx=0x7f3b9c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:11:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 -- 172.21.15.46:0/2205490632 shutdown_connections 2024-03-08T23:11:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2205490632 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 0x7f3bac10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2205490632 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac101d80 0x7f3bac105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2205490632 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac1017b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 -- 172.21.15.46:0/2205490632 >> 172.21.15.46:0/2205490632 conn(0x7f3bac0fcc50 msgr2=0x7f3bac0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 -- 172.21.15.46:0/2205490632 shutdown_connections 2024-03-08T23:11:26.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.807+0000 7f3bb1800640 1 -- 172.21.15.46:0/2205490632 wait complete. 2024-03-08T23:11:26.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.808+0000 7f3bb1800640 1 Processor -- start 2024-03-08T23:11:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.808+0000 7f3bb1800640 1 -- start start 2024-03-08T23:11:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 0x7f3bac073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 0x7f3bac1a5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac111580 con 0x7f3bac1013d0 2024-03-08T23:11:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac111400 con 0x7f3bac1064a0 2024-03-08T23:11:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb1800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac111700 con 0x7f3bac074070 2024-03-08T23:11:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3babe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3babe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49444/0 (socket says 172.21.15.46:49444) 2024-03-08T23:11:26.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3babe00640 1 -- 172.21.15.46:0/2698114843 learned_addr learned my addr 172.21.15.46:0/2698114843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 0x7f3bac1a5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 0x7f3bac073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.809+0000 7f3babe00640 1 -- 172.21.15.46:0/2698114843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 msgr2=0x7f3bac1a5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3babe00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 0x7f3bac1a5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3babe00640 1 -- 172.21.15.46:0/2698114843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 msgr2=0x7f3bac073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3babe00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 0x7f3bac073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3babe00640 1 -- 172.21.15.46:0/2698114843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bac1a6310 con 0x7f3bac1013d0 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 0x7f3bac1a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3babe00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c02c240 tx=0x7f3b9c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3ba9600640 1 -- 172.21.15.46:0/2698114843 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c043020 con 0x7f3bac1013d0 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3ba9600640 1 -- 172.21.15.46:0/2698114843 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b9c0054e0 con 0x7f3bac1013d0 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bac1a65a0 con 0x7f3bac1013d0 2024-03-08T23:11:26.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3ba9600640 1 -- 172.21.15.46:0/2698114843 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c003b30 con 0x7f3bac1013d0 2024-03-08T23:11:26.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.810+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bac1a6af0 con 0x7f3bac1013d0 2024-03-08T23:11:26.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 Processor -- start 2024-03-08T23:11:26.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3ba9600640 1 -- 172.21.15.46:0/2698114843 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3b9c046020 con 0x7f3bac1013d0 2024-03-08T23:11:26.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 -- start start 2024-03-08T23:11:26.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d50eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d51a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d4d0410 con 0x55776d50f0b0 2024-03-08T23:11:26.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d472d50 con 0x55776d517c90 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.814+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d41bb80 con 0x55776d50e790 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3babe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3babe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49460/0 (socket says 172.21.15.46:49460) 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3babe00640 1 -- 172.21.15.46:0/1137452847 learned_addr learned my addr 172.21.15.46:0/1137452847 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d50eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d51a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 -- 172.21.15.46:0/1137452847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 msgr2=0x55776d51a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d51a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 -- 172.21.15.46:0/1137452847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 msgr2=0x55776d517750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3bb0e00640 1 -- 172.21.15.46:0/1137452847 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55776d2517d0 con 0x55776d50e790 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.815+0000 7f3babe00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d50eb70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009f10 tx=0x7f3ba40107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=756fab3f16ea7dea server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3ba9600640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 0x7f3b900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3ba1600640 1 -- 172.21.15.46:0/1137452847 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba401c680 con 0x55776d50e790 2024-03-08T23:11:26.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3ba1600640 1 -- 172.21.15.46:0/1137452847 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ba401cd00 con 0x55776d50e790 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3ba1600640 1 -- 172.21.15.46:0/1137452847 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba401a380 con 0x55776d50e790 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3ba9600640 1 -- 172.21.15.46:0/2698114843 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3b9c0f10b0 con 0x7f3bac1013d0 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.816+0000 7f3babe00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 0x7f3b900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/1137452847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 msgr2=0x55776d50eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d50eb70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009f10 tx=0x7f3ba40107a0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/1137452847 shutdown_connections 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d51a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d517750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/1137452847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d50eb70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/1137452847 >> 172.21.15.46:0/1137452847 conn(0x55776d508ec0 msgr2=0x55776d5092a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/1137452847 shutdown_connections 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3babe00640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 0x7f3b900a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0bd3f0 tx=0x7f3b9c0b9040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/1137452847 wait complete. 2024-03-08T23:11:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.817+0000 7f3bb5bf2740 1 -- start start 2024-03-08T23:11:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 Processor -- start 2024-03-08T23:11:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d514cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d515770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d42eee0 con 0x55776d517c90 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d46edf0 con 0x55776d50f0b0 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776d51aa80 con 0x55776d50e790 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb5bf2740 10 client.0 _ll_register_callbacks cb 0x55776d4358d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d514cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3ba3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3ba3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46208/0 (socket says 172.21.15.46:46208) 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d514cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45304/0 (socket says 172.21.15.46:45304) 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.818+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918870019 learned_addr learned my addr 172.21.15.46:0/918870019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d515770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918870019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 msgr2=0x55776d514cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d514cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918870019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 msgr2=0x55776d515770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d515770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918870019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55776d1873f0 con 0x55776d50f0b0 2024-03-08T23:11:26.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3b94012730 tx=0x7f3b94012bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9811e3cc9cbfe624 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3b9a000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.819+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9400f920 con 0x55776d50f0b0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3ba401cd00 con 0x55776d50f0b0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3b9400fac0 con 0x55776d50f0b0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b9400fcc0 con 0x55776d50f0b0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3bb5bf2740 10 client.26324 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55776d1470f0 con 0x55776d50f0b0 2024-03-08T23:11:26.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ba401cd00 con 0x55776d50f0b0 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3b9401c420 con 0x55776d50f0b0 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3bb5bf2740 10 client.26324 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.820+0000 7f3bb5bf2740 10 client.26324 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb5bf2740 20 client.26324 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d527f90 0x55776d52a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb5bf2740 4 client.26324 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55776d480290 con 0x55776d527f90 2024-03-08T23:11:26.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d527f90 0x55776d52a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d527f90 0x55776d52a380 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6270d9da34556a7 server_cookie=cbb737f12499e01d in_seq=0 out_seq=0 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.821+0000 7f3b9a000640 10 client.26324 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.822+0000 7f3b9a000640 1 -- 172.21.15.46:0/918870019 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55776d480290 con 0x55776d527f90 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.822+0000 7f3b9a000640 10 client.26324 handle_command_reply: tid=0 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.822+0000 7f3bb5bf2740 1 client.26324 shutdown 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 msgr2=0x55776d515230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3b94012730 tx=0x7f3b94012bf0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 shutdown_connections 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d527f90 0x55776d52a380 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d515770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d515230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3b9a000640 0 client.26324 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918870019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d514cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 >> 172.21.15.46:0/918870019 conn(0x55776d508ec0 msgr2=0x55776d50ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 shutdown_connections 2024-03-08T23:11:26.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918870019 wait complete. 2024-03-08T23:11:26.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.823+0000 7f3bb5bf2740 20 client.26324 trim_cache size 0 max 16384 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 Processor -- start 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 -- start start 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d517420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d51ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1b2a50 con 0x55776d517c90 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1b0590 con 0x55776d50f0b0 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.840+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1ae630 con 0x55776d50e790 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49482/0 (socket says 172.21.15.46:49482) 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d51ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3bb0e00640 1 -- 172.21.15.46:0/918632246 learned_addr learned my addr 172.21.15.46:0/918632246 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:26.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d517420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918632246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 msgr2=0x55776d517420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d517420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918632246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 msgr2=0x55776d51f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 -- 172.21.15.46:0/918632246 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55776d504570 con 0x55776d50f0b0 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.841+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d51ecf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b9401c860 tx=0x7f3b940123e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=530e794e6a4397fc server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.842+0000 7f3ba0c00640 1 -- 172.21.15.46:0/918632246 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94009ea0 con 0x55776d50f0b0 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.842+0000 7f3ba0c00640 1 -- 172.21.15.46:0/918632246 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b9400fe60 con 0x55776d50f0b0 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.842+0000 7f3ba0c00640 1 -- 172.21.15.46:0/918632246 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b940035f0 con 0x55776d50f0b0 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918632246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 msgr2=0x55776d51ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d51ecf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f3b9401c860 tx=0x7f3b940123e0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918632246 shutdown_connections 2024-03-08T23:11:26.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d517c90 0x55776d51f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d50f0b0 0x55776d51ecf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/918632246 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d517420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918632246 >> 172.21.15.46:0/918632246 conn(0x55776d508ec0 msgr2=0x55776d5135e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918632246 shutdown_connections 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/918632246 wait complete. 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 -- start start 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.843+0000 7f3bb5bf2740 1 Processor -- start 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d578580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d572410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1aec30 con 0x55776d50f0b0 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1aff30 con 0x55776d517c90 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bac1af270 con 0x55776d50e790 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb5bf2740 10 client.0 _ll_register_callbacks cb 0x7f3ba40099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3ba3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d578580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb0e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45312/0 (socket says 172.21.15.46:45312) 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.844+0000 7f3bb0e00640 1 -- 172.21.15.46:0/2874743900 learned_addr learned my addr 172.21.15.46:0/2874743900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d572410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 -- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 msgr2=0x55776d572410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d572410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 -- 172.21.15.46:0/2874743900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 msgr2=0x55776d578580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d578580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 -- 172.21.15.46:0/2874743900 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55776d572be0 con 0x55776d50e790 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb0e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009cd0 tx=0x7f3ba401e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c6b0b1b5da2a407 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3b9a000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba402d760 con 0x55776d50e790 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f3ba401c480 con 0x55776d50e790 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.845+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b9400fe60 con 0x55776d50e790 2024-03-08T23:11:26.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3ba403f050 con 0x55776d50e790 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 10 client.26329 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55776d57f7e0 con 0x55776d50e790 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b9400fe60 con 0x55776d50e790 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3ba4038060 con 0x55776d50e790 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 10 client.26329 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 10 client.26329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 20 client.26329 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:26.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d57fcf0 0x55776d582110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 4 client.26329 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:11:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.846+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55776d47a5e0 con 0x55776d57fcf0 2024-03-08T23:11:26.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d57fcf0 0x55776d582110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:26.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3ba3e00640 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d57fcf0 0x55776d582110 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65f3a152a3fb8a2b server_cookie=22af74884f59ee30 in_seq=0 out_seq=0 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3b9a000640 10 client.26329 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3b9a000640 1 -- 172.21.15.46:0/2874743900 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55776d47a5e0 con 0x55776d57fcf0 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3b9a000640 10 client.26329 handle_command_reply: tid=0 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3bb5bf2740 1 client.26329 shutdown 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 msgr2=0x55776d578040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.847+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009cd0 tx=0x7f3ba401e040 comp rx=0 tx=0).stop 2024-03-08T23:11:26.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 shutdown_connections 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55776d57fcf0 0x55776d582110 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55776d517c90 0x55776d572410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55776d50f0b0 0x55776d578580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 --2- 172.21.15.46:0/2874743900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55776d50e790 0x55776d578040 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3b9a000640 0 client.26329 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 >> 172.21.15.46:0/2874743900 conn(0x55776d508ec0 msgr2=0x55776d50a2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 shutdown_connections 2024-03-08T23:11:26.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 1 -- 172.21.15.46:0/2874743900 wait complete. 2024-03-08T23:11:26.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.848+0000 7f3bb5bf2740 20 client.26329 trim_cache size 0 max 16384 2024-03-08T23:11:26.862 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:11:26.862 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:11:26.862 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:11:26.863 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:11:26.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 msgr2=0x7f3b900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 0x7f3b900a4e30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c0bd3f0 tx=0x7f3b9c0b9040 comp rx=0 tx=0).stop 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 msgr2=0x7f3bac0735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c02c240 tx=0x7f3b9c0042f0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 shutdown_connections 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f3b900a2970 0x7f3b900a4e30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3bac074070 0x7f3bac1a5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f3bac1064a0 0x7f3bac073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 --2- 172.21.15.46:0/2698114843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3bac1013d0 0x7f3bac0735f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 >> 172.21.15.46:0/2698114843 conn(0x7f3bac0fcc50 msgr2=0x7f3bac0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 shutdown_connections 2024-03-08T23:11:26.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:26.849+0000 7f3bb1800640 1 -- 172.21.15.46:0/2698114843 wait complete. 2024-03-08T23:11:26.871 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:11:26.871 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:11:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 -- 172.21.15.46:0/4031841548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c80736c0 msgr2=0x7ff8c8073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 --2- 172.21.15.46:0/4031841548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c80736c0 0x7ff8c8073aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc009900 tx=0x7ff8bc02bcd0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 -- 172.21.15.46:0/4031841548 shutdown_connections 2024-03-08T23:11:27.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 --2- 172.21.15.46:0/4031841548 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c810db20 0x7ff8c810ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 --2- 172.21.15.46:0/4031841548 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c8073fe0 0x7ff8c810d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 --2- 172.21.15.46:0/4031841548 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c80736c0 0x7ff8c8073aa0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 -- 172.21.15.46:0/4031841548 >> 172.21.15.46:0/4031841548 conn(0x7ff8c80fcd90 msgr2=0x7ff8c80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 -- 172.21.15.46:0/4031841548 shutdown_connections 2024-03-08T23:11:27.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.095+0000 7ff8d0600640 1 -- 172.21.15.46:0/4031841548 wait complete. 2024-03-08T23:11:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.096+0000 7ff8d0600640 1 Processor -- start 2024-03-08T23:11:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.096+0000 7ff8d0600640 1 -- start start 2024-03-08T23:11:27.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c80736c0 0x7ff8c8199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 0x7ff8c819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c81138e0 con 0x7ff8c810db20 2024-03-08T23:11:27.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8113760 con 0x7ff8c8073fe0 2024-03-08T23:11:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8d0600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8c8113a60 con 0x7ff8c80736c0 2024-03-08T23:11:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8ce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8ce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49500/0 (socket says 172.21.15.46:49500) 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8ce600640 1 -- 172.21.15.46:0/2038876749 learned_addr learned my addr 172.21.15.46:0/2038876749 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8cd200640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 0x7ff8c819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.097+0000 7ff8cdc00640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c80736c0 0x7ff8c8199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 -- 172.21.15.46:0/2038876749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c80736c0 msgr2=0x7ff8c8199e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c80736c0 0x7ff8c8199e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 -- 172.21.15.46:0/2038876749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 msgr2=0x7ff8c819a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 0x7ff8c819a3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 -- 172.21.15.46:0/2038876749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c819ee20 con 0x7ff8c810db20 2024-03-08T23:11:27.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8cd200640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 0x7ff8c819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8ce600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000ec50 tx=0x7ff8c000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0003b70 con 0x7ff8c810db20 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c0003d10 con 0x7ff8c810db20 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c00109e0 con 0x7ff8c810db20 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.098+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8c819f110 con 0x7ff8c810db20 2024-03-08T23:11:27.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.099+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8c81a69a0 con 0x7ff8c810db20 2024-03-08T23:11:27.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.100+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8c8068530 con 0x7ff8c810db20 2024-03-08T23:11:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.102+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff8c0003ec0 con 0x7ff8c810db20 2024-03-08T23:11:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.104+0000 7ff8c6000640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 0x7ff8b00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.104+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff8c00cf710 con 0x7ff8c810db20 2024-03-08T23:11:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.104+0000 7ff8cdc00640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 0x7ff8b00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.105+0000 7ff8cdc00640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 0x7ff8b00a4eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005ab0 tx=0x7ff8bc005a20 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.109+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff8c009c0b0 con 0x7ff8c810db20 2024-03-08T23:11:27.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.354+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8c802ce70 con 0x7ff8c810db20 2024-03-08T23:11:27.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.356+0000 7ff8c6000640 1 -- 172.21.15.46:0/2038876749 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11340 (secure 0 0 0) 0x7ff8c009b9b0 con 0x7ff8c810db20 2024-03-08T23:11:27.359 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:11:27.359 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:10:43.831330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:11:27.359 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-03-08T23:11:27.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.358+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 msgr2=0x7ff8b00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.358+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 0x7ff8b00a4eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc005ab0 tx=0x7ff8bc005a20 comp rx=0 tx=0).stop 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 msgr2=0x7ff8c819e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000ec50 tx=0x7ff8c000c6a0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 shutdown_connections 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b00a29f0 0x7ff8b00a4eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8c810db20 0x7ff8c819e6f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8c8073fe0 0x7ff8c819a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 --2- 172.21.15.46:0/2038876749 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8c80736c0 0x7ff8c8199e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 >> 172.21.15.46:0/2038876749 conn(0x7ff8c80fcd90 msgr2=0x7ff8c80fe5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 shutdown_connections 2024-03-08T23:11:27.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.359+0000 7ff8d0600640 1 -- 172.21.15.46:0/2038876749 wait complete. 2024-03-08T23:11:27.381 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:11:27.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:11:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 -- 172.21.15.46:0/1267338171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f236010d8a0 msgr2=0x7f236010dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 --2- 172.21.15.46:0/1267338171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f236010d8a0 0x7f236010dc80 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2354009930 tx=0x7f235402bc30 comp rx=0 tx=0).stop 2024-03-08T23:11:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 -- 172.21.15.46:0/1267338171 shutdown_connections 2024-03-08T23:11:27.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 --2- 172.21.15.46:0/1267338171 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2360108250 0x7f2360114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 --2- 172.21.15.46:0/1267338171 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f2360107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 --2- 172.21.15.46:0/1267338171 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f236010d8a0 0x7f236010dc80 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 -- 172.21.15.46:0/1267338171 >> 172.21.15.46:0/1267338171 conn(0x7f2360075830 msgr2=0x7f2360077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 -- 172.21.15.46:0/1267338171 shutdown_connections 2024-03-08T23:11:27.607 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.603+0000 7f2368200640 1 -- 172.21.15.46:0/1267338171 wait complete. 2024-03-08T23:11:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.604+0000 7f2368200640 1 Processor -- start 2024-03-08T23:11:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.604+0000 7f2368200640 1 -- start start 2024-03-08T23:11:27.609 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 0x7f236019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f236010d8a0 0x7f23601a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360117d00 con 0x7f2360108250 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360117b80 con 0x7f2360107890 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2368200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2360117e80 con 0x7f236010d8a0 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2364e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 0x7f236019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.610 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46272/0 (socket says 172.21.15.46:46272) 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 -- 172.21.15.46:0/966463440 learned_addr learned my addr 172.21.15.46:0/966463440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2364e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 0x7f236019e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49514/0 (socket says 172.21.15.46:49514) 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2366200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f236010d8a0 0x7f23601a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 -- 172.21.15.46:0/966463440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f236010d8a0 msgr2=0x7f23601a2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f236010d8a0 0x7f23601a2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.611 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 -- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 msgr2=0x7f236019e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 0x7f236019e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.605+0000 7f2365800640 1 -- 172.21.15.46:0/966463440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23601a3210 con 0x7f2360107890 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f2365800640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2354009900 tx=0x7f2354004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f235e000640 1 -- 172.21.15.46:0/966463440 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354043020 con 0x7f2360107890 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23601a34a0 con 0x7f2360107890 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f235e000640 1 -- 172.21.15.46:0/966463440 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23540054e0 con 0x7f2360107890 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f235e000640 1 -- 172.21.15.46:0/966463440 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2354003d00 con 0x7f2360107890 2024-03-08T23:11:27.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.606+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23601aad90 con 0x7f2360107890 2024-03-08T23:11:27.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.609+0000 7f236aff8740 1 Processor -- start 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.609+0000 7f235e000640 1 -- 172.21.15.46:0/966463440 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2354046020 con 0x7f2360107890 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 -- start start 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb4214b310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb420a6dd0 con 0x55bb42142c70 2024-03-08T23:11:27.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb4204fc70 con 0x55bb42142350 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f236aff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb420a2e70 con 0x55bb4214b850 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f2366200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f2366200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46278/0 (socket says 172.21.15.46:46278) 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f2366200640 1 -- 172.21.15.46:0/293712728 learned_addr learned my addr 172.21.15.46:0/293712728 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f2365800640 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb4214b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.610+0000 7f2364e00640 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 -- 172.21.15.46:0/293712728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 msgr2=0x55bb4214dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 -- 172.21.15.46:0/293712728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 msgr2=0x55bb4214b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb4214b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 -- 172.21.15.46:0/293712728 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb421092b0 con 0x55bb42142350 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2366200640 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2358009f70 tx=0x7f2358010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9c70b8bceac4b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f235e000640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 0x7f23480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2352a00640 1 -- 172.21.15.46:0/293712728 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358016070 con 0x55bb42142350 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2352a00640 1 -- 172.21.15.46:0/293712728 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2358004d20 con 0x55bb42142350 2024-03-08T23:11:27.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f2352a00640 1 -- 172.21.15.46:0/293712728 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358012910 con 0x55bb42142350 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.611+0000 7f235e000640 1 -- 172.21.15.46:0/966463440 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f23540f1230 con 0x7f2360107890 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f2365800640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 0x7f23480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 -- 172.21.15.46:0/293712728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 msgr2=0x55bb42142730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2358009f70 tx=0x7f2358010800 comp rx=0 tx=0).stop 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 -- 172.21.15.46:0/293712728 shutdown_connections 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb4214b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 --2- 172.21.15.46:0/293712728 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42142730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 -- 172.21.15.46:0/293712728 >> 172.21.15.46:0/293712728 conn(0x55bb4213ca80 msgr2=0x55bb4213ce60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f2365800640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 0x7f23480a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55bb42141e20 tx=0x7f23540f4ce0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 -- 172.21.15.46:0/293712728 shutdown_connections 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.612+0000 7f236aff8740 1 -- 172.21.15.46:0/293712728 wait complete. 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.613+0000 7f236aff8740 1 -- start start 2024-03-08T23:11:27.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.613+0000 7f236aff8740 1 Processor -- start 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.613+0000 7f236aff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142350 0x55bb4214b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.613+0000 7f236aff8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f236aff8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb4214b850 0x55bb42148be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f236aff8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb4214e640 con 0x55bb42142c70 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f236aff8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb420630c0 con 0x55bb4214b850 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f236aff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb4214e7c0 con 0x55bb42142350 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f236aff8740 10 client.0 _ll_register_callbacks cb 0x55bb4209dec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:27.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49538/0 (socket says 172.21.15.46:49538) 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 -- 172.21.15.46:0/3592255370 learned_addr learned my addr 172.21.15.46:0/3592255370 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2366200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb4214b850 0x55bb42148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2366200640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142350 0x55bb4214b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 -- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142350 msgr2=0x55bb4214b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142350 0x55bb4214b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 -- 172.21.15.46:0/3592255370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb4214b850 msgr2=0x55bb42148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb4214b850 0x55bb42148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 -- 172.21.15.46:0/3592255370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb41dbb3f0 con 0x55bb42142c70 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.614+0000 7f2364e00640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f234c00bf00 tx=0x7f234c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6e8ff65da0e726b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f2347400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c004020 con 0x55bb42142c70 2024-03-08T23:11:27.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f234c0041c0 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2358004d20 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f234c024050 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f236aff8740 10 client.16212 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55bb41e857d0 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.615+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2358004d20 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f234c003920 con 0x55bb42142c70 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 10 client.16212 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:27.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 10 client.16212 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:11:27.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 20 client.16212 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:27.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb4215b970 0x55bb4215dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 4 client.16212 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:27.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb4207b4f0 con 0x55bb4215b970 2024-03-08T23:11:27.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.616+0000 7f2366200640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb4215b970 0x55bb4215dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.617+0000 7f2366200640 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb4215b970 0x55bb4215dd90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3ab4d28f80050b7f server_cookie=ad7d0c1e235b99d1 in_seq=0 out_seq=0 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.617+0000 7f2347400640 10 client.16212 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.617+0000 7f2347400640 1 -- 172.21.15.46:0/3592255370 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bb4207b4f0 con 0x55bb4215b970 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.617+0000 7f2347400640 10 client.16212 handle_command_reply: tid=0 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 client.16212 shutdown 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 msgr2=0x55bb421485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f234c00bf00 tx=0x7f234c011810 comp rx=0 tx=0).stop 2024-03-08T23:11:27.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 shutdown_connections 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb4215b970 0x55bb4215dd90 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb4214b850 0x55bb42148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142c70 0x55bb421485d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 --2- 172.21.15.46:0/3592255370 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142350 0x55bb4214b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f2347400640 0 client.16212 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 >> 172.21.15.46:0/3592255370 conn(0x55bb4213ca80 msgr2=0x55bb4213e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 shutdown_connections 2024-03-08T23:11:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 1 -- 172.21.15.46:0/3592255370 wait complete. 2024-03-08T23:11:27.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.618+0000 7f236aff8740 20 client.16212 trim_cache size 0 max 16384 2024-03-08T23:11:27.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.631+0000 7f236aff8740 1 Processor -- start 2024-03-08T23:11:27.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.631+0000 7f236aff8740 1 -- start start 2024-03-08T23:11:27.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142c70 0x55bb4214ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b7110 con 0x55bb42142350 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b4c50 con 0x55bb42142c70 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f236aff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b2cf0 con 0x55bb4214b850 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2366200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2366200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49542/0 (socket says 172.21.15.46:49542) 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2366200640 1 -- 172.21.15.46:0/4222921464 learned_addr learned my addr 172.21.15.46:0/4222921464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2364e00640 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142c70 0x55bb4214ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2366200640 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.632+0000 7f2366200640 1 -- 172.21.15.46:0/4222921464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 msgr2=0x55bb4214b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2366200640 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2366200640 1 -- 172.21.15.46:0/4222921464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142c70 msgr2=0x55bb4214ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2366200640 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142c70 0x55bb4214ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2366200640 1 -- 172.21.15.46:0/4222921464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb42148f50 con 0x55bb42142350 2024-03-08T23:11:27.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2366200640 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f23580124c0 tx=0x7f23580124f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d140c024a1abb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2352000640 1 -- 172.21.15.46:0/4222921464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358016040 con 0x55bb42142350 2024-03-08T23:11:27.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2352000640 1 -- 172.21.15.46:0/4222921464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f235802a8b0 con 0x55bb42142350 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.633+0000 7f2352000640 1 -- 172.21.15.46:0/4222921464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2358023040 con 0x55bb42142350 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 -- 172.21.15.46:0/4222921464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 msgr2=0x55bb4213b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f23580124c0 tx=0x7f23580124f0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 -- 172.21.15.46:0/4222921464 shutdown_connections 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb4214b850 0x55bb4214b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142c70 0x55bb4214ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 --2- 172.21.15.46:0/4222921464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb42142350 0x55bb4213b910 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 -- 172.21.15.46:0/4222921464 >> 172.21.15.46:0/4222921464 conn(0x55bb4213ca80 msgr2=0x55bb4213b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 -- 172.21.15.46:0/4222921464 shutdown_connections 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.634+0000 7f236aff8740 1 -- 172.21.15.46:0/4222921464 wait complete. 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 -- start start 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 Processor -- start 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42151800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb4214b850 0x55bb421b1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b32f0 con 0x55bb4214b850 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b45f0 con 0x55bb42142350 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23601b3930 con 0x55bb42142c70 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f236aff8740 10 client.0 _ll_register_callbacks cb 0x7f2358009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.635+0000 7f2364e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2366200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45398/0 (socket says 172.21.15.46:45398) 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 -- 172.21.15.46:0/2931742178 learned_addr learned my addr 172.21.15.46:0/2931742178 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2366200640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb4214b850 0x55bb421b1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 -- 172.21.15.46:0/2931742178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 msgr2=0x55bb42151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42151800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 -- 172.21.15.46:0/2931742178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb4214b850 msgr2=0x55bb421b1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb4214b850 0x55bb421b1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 -- 172.21.15.46:0/2931742178 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb421b23a0 con 0x55bb42142c70 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2364e00640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f234c003d40 tx=0x7f234c003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc6d57456b479fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.636+0000 7f2347400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:11:27.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234c00ba80 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f234c003f30 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f235802a8b0 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f234c03a050 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f236aff8740 10 client.26353 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55bb421b3500 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.637+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f235802a8b0 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f234c010820 con 0x55bb42142c70 2024-03-08T23:11:27.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 10 client.26353 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 10 client.26353 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 20 client.26353 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb421b3c00 0x55bb421b6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 4 client.26353 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55bb420b4310 con 0x55bb421b3c00 2024-03-08T23:11:27.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.638+0000 7f2366200640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb421b3c00 0x55bb421b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f2366200640 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb421b3c00 0x55bb421b6020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=884ddc2b3e2dfcc9 server_cookie=28adcf9cb37d0e1 in_seq=0 out_seq=0 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f2347400640 10 client.26353 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f2347400640 1 -- 172.21.15.46:0/2931742178 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55bb420b4310 con 0x55bb421b3c00 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f2347400640 10 client.26353 handle_command_reply: tid=0 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 client.26353 shutdown 2024-03-08T23:11:27.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 msgr2=0x55bb42151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f234c003d40 tx=0x7f234c003820 comp rx=0 tx=0).stop 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 shutdown_connections 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55bb421b3c00 0x55bb421b6020 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55bb4214b850 0x55bb421b1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55bb42142c70 0x55bb42151d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 --2- 172.21.15.46:0/2931742178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55bb42142350 0x55bb42151800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f2347400640 0 client.26353 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:11:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 >> 172.21.15.46:0/2931742178 conn(0x55bb4213ca80 msgr2=0x55bb4213ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 shutdown_connections 2024-03-08T23:11:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 1 -- 172.21.15.46:0/2931742178 wait complete. 2024-03-08T23:11:27.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.639+0000 7f236aff8740 20 client.26353 trim_cache size 0 max 16384 2024-03-08T23:11:27.650 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:11:27.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 msgr2=0x7f23480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 0x7f23480a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x55bb42141e20 tx=0x7f23540f4ce0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 msgr2=0x7f236019e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2354009900 tx=0x7f2354004430 comp rx=0 tx=0).stop 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 shutdown_connections 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f23480a2970 0x7f23480a4e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f236010d8a0 0x7f23601a2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2360108250 0x7f236019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 --2- 172.21.15.46:0/966463440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2360107890 0x7f236019e220 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 >> 172.21.15.46:0/966463440 conn(0x7f2360075830 msgr2=0x7f2360075ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 shutdown_connections 2024-03-08T23:11:27.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.640+0000 7f2368200640 1 -- 172.21.15.46:0/966463440 wait complete. 2024-03-08T23:11:27.663 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:11:27.663 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:11:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 -- 172.21.15.46:0/615245540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 msgr2=0x7f3580103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 --2- 172.21.15.46:0/615245540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580103930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f356c009890 tx=0x7f356c02bb70 comp rx=0 tx=0).stop 2024-03-08T23:11:27.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 -- 172.21.15.46:0/615245540 shutdown_connections 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 --2- 172.21.15.46:0/615245540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f3580110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 --2- 172.21.15.46:0/615245540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580103930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 --2- 172.21.15.46:0/615245540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 0x7f3580109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 -- 172.21.15.46:0/615245540 >> 172.21.15.46:0/615245540 conn(0x7f35800fef60 msgr2=0x7f3580101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 -- 172.21.15.46:0/615245540 shutdown_connections 2024-03-08T23:11:27.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.893+0000 7f3588600640 1 -- 172.21.15.46:0/615245540 wait complete. 2024-03-08T23:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 Processor -- start 2024-03-08T23:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 -- start start 2024-03-08T23:11:27.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 0x7f358019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3580113830 con 0x7f3580103e70 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801136b0 con 0x7f35801034b0 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.895+0000 7f3588600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801139b0 con 0x7f3580109570 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49562/0 (socket says 172.21.15.46:49562) 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 -- 172.21.15.46:0/4085120989 learned_addr learned my addr 172.21.15.46:0/4085120989 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585c00640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3586600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 0x7f358019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 -- 172.21.15.46:0/4085120989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 msgr2=0x7f358019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 0x7f358019e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 -- 172.21.15.46:0/4085120989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 msgr2=0x7f3580199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585200640 1 -- 172.21.15.46:0/4085120989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358019ed20 con 0x7f3580103e70 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.896+0000 7f3585c00640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f3585200640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f356c0042f0 tx=0x7f356c004320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f357e000640 1 -- 172.21.15.46:0/4085120989 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c041020 con 0x7f3580103e70 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f357e000640 1 -- 172.21.15.46:0/4085120989 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f356c02c9f0 con 0x7f3580103e70 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358019efb0 con 0x7f3580103e70 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f357e000640 1 -- 172.21.15.46:0/4085120989 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c039040 con 0x7f3580103e70 2024-03-08T23:11:27.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.897+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35801a68a0 con 0x7f3580103e70 2024-03-08T23:11:27.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 Processor -- start 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f357e000640 1 -- 172.21.15.46:0/4085120989 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f356c044020 con 0x7f3580103e70 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 -- start start 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b933310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b935c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b88edd0 con 0x55919b933850 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b837c70 con 0x55919b92a350 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.901+0000 7f358b212740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b88ae70 con 0x55919b92ac70 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46336/0 (socket says 172.21.15.46:46336) 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 -- 172.21.15.46:0/1963761870 learned_addr learned my addr 172.21.15.46:0/1963761870 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3585200640 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b935c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3585c00640 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b933310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 -- 172.21.15.46:0/1963761870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 msgr2=0x55919b933310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b933310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 -- 172.21.15.46:0/1963761870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 msgr2=0x55919b935c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b935c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.902+0000 7f3586600640 1 -- 172.21.15.46:0/1963761870 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55919b8f12b0 con 0x55919b92a350 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f3586600640 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3578009c20 tx=0x7f3578011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef9c9dd8680f3200 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f357e000640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 0x7f35680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f357e000640 1 -- 172.21.15.46:0/4085120989 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f356c0f0a20 con 0x7f3580103e70 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f3572a00640 1 -- 172.21.15.46:0/1963761870 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357801e070 con 0x55919b92a350 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f3585c00640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 0x7f35680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f3572a00640 1 -- 172.21.15.46:0/1963761870 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3578019b10 con 0x55919b92a350 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.903+0000 7f3572a00640 1 -- 172.21.15.46:0/1963761870 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357800e9a0 con 0x55919b92a350 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 -- 172.21.15.46:0/1963761870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 msgr2=0x55919b92a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3578009c20 tx=0x7f3578011810 comp rx=0 tx=0).stop 2024-03-08T23:11:27.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f3585c00640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 0x7f35680a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55919b929e20 tx=0x7f3574009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 -- 172.21.15.46:0/1963761870 shutdown_connections 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b935c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b933310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 --2- 172.21.15.46:0/1963761870 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b92a730 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 -- 172.21.15.46:0/1963761870 >> 172.21.15.46:0/1963761870 conn(0x55919b924a80 msgr2=0x55919b924e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 -- 172.21.15.46:0/1963761870 shutdown_connections 2024-03-08T23:11:27.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.904+0000 7f358b212740 1 -- 172.21.15.46:0/1963761870 wait complete. 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 -- start start 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 Processor -- start 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b9331a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b930b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b936640 con 0x55919b933850 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b84b0c0 con 0x55919b92a350 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919b9367c0 con 0x55919b92ac70 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.905+0000 7f358b212740 10 client.0 _ll_register_callbacks cb 0x55919b885ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3586600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b930b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b9331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45444/0 (socket says 172.21.15.46:45444) 2024-03-08T23:11:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b9331a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46348/0 (socket says 172.21.15.46:46348) 2024-03-08T23:11:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 -- 172.21.15.46:0/2114112433 learned_addr learned my addr 172.21.15.46:0/2114112433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 -- 172.21.15.46:0/2114112433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 msgr2=0x55919b9331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b9331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 -- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 msgr2=0x55919b930b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b930b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 -- 172.21.15.46:0/2114112433 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55919b5a33f0 con 0x55919b92ac70 2024-03-08T23:11:27.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.906+0000 7f3585200640 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f356c02c180 tx=0x7f356c034700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4b28d749a034e4f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f3567400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0bc730 con 0x55919b92ac70 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f356c0ee030 con 0x55919b92ac70 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3578019b10 con 0x55919b92ac70 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f356c0ee230 con 0x55919b92ac70 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f358b212740 10 client.26365 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:27.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55919b66d7d0 con 0x55919b92ac70 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.907+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3578019b10 con 0x55919b92ac70 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f356c03c030 con 0x55919b92ac70 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 10 client.26365 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 10 client.26365 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 20 client.26365 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b9438d0 0x55919b945cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 4 client.26365 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55919b8634f0 con 0x55919b9438d0 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.908+0000 7f3585c00640 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b9438d0 0x55919b945cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.909+0000 7f3585c00640 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b9438d0 0x55919b945cf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88dc11d9acc1cd4c server_cookie=3ed9c0601863c12a in_seq=0 out_seq=0 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.909+0000 7f3567400640 10 client.26365 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.909+0000 7f3567400640 1 -- 172.21.15.46:0/2114112433 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55919b8634f0 con 0x55919b9438d0 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.909+0000 7f3567400640 10 client.26365 handle_command_reply: tid=0 2024-03-08T23:11:27.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.909+0000 7f358b212740 1 client.26365 shutdown 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 msgr2=0x55919b9305a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f356c02c180 tx=0x7f356c034700 comp rx=0 tx=0).stop 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 shutdown_connections 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b933850 0x55919b930b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b9438d0 0x55919b945cf0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b9305a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 --2- 172.21.15.46:0/2114112433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b9331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 >> 172.21.15.46:0/2114112433 conn(0x55919b924a80 msgr2=0x55919b9267d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f3567400640 0 client.26365 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 shutdown_connections 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 1 -- 172.21.15.46:0/2114112433 wait complete. 2024-03-08T23:11:27.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.910+0000 7f358b212740 20 client.26365 trim_cache size 0 max 16384 2024-03-08T23:11:27.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 Processor -- start 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 -- start start 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b92d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801b2b80 con 0x55919b92a350 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801b06c0 con 0x55919b933850 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.923+0000 7f358b212740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801ae760 con 0x55919b92ac70 2024-03-08T23:11:27.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3585c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3585c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49592/0 (socket says 172.21.15.46:49592) 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46364/0 (socket says 172.21.15.46:46364) 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3585c00640 1 -- 172.21.15.46:0/1630822206 learned_addr learned my addr 172.21.15.46:0/1630822206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3585200640 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b92d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 -- 172.21.15.46:0/1630822206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 msgr2=0x55919b92d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b92d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 -- 172.21.15.46:0/1630822206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 msgr2=0x55919b932cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3586600640 1 -- 172.21.15.46:0/1630822206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55919b92aaa0 con 0x55919b933850 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.924+0000 7f3585c00640 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f3586600640 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3578009230 tx=0x7f357800e600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c77788f2d3a50ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f3572000640 1 -- 172.21.15.46:0/1630822206 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578028020 con 0x55919b933850 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f3572000640 1 -- 172.21.15.46:0/1630822206 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3578009890 con 0x55919b933850 2024-03-08T23:11:27.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f3572000640 1 -- 172.21.15.46:0/1630822206 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357801e040 con 0x55919b933850 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f358b212740 1 -- 172.21.15.46:0/1630822206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 msgr2=0x55919b92dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.925+0000 7f358b212740 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f3578009230 tx=0x7f357800e600 comp rx=0 tx=0).stop 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 -- 172.21.15.46:0/1630822206 shutdown_connections 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b933850 0x55919b92dd40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b92ac70 0x55919b92d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 --2- 172.21.15.46:0/1630822206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92a350 0x55919b932cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 -- 172.21.15.46:0/1630822206 >> 172.21.15.46:0/1630822206 conn(0x55919b924a80 msgr2=0x55919b92ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 -- 172.21.15.46:0/1630822206 shutdown_connections 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 -- 172.21.15.46:0/1630822206 wait complete. 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 -- start start 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.926+0000 7f358b212740 1 Processor -- start 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b93af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b933850 0x55919b9399e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801aed60 con 0x55919b92ac70 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801b0060 con 0x55919b92a350 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35801af3a0 con 0x55919b933850 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f358b212740 10 client.0 _ll_register_callbacks cb 0x7f3578009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b93af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49598/0 (socket says 172.21.15.46:49598) 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585200640 1 -- 172.21.15.46:0/1828349609 learned_addr learned my addr 172.21.15.46:0/1828349609 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3586600640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b933850 0x55919b9399e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585c00640 1 -- 172.21.15.46:0/1828349609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b933850 msgr2=0x55919b9399e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585c00640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b933850 0x55919b9399e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585c00640 1 -- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 msgr2=0x55919b9392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:27.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.927+0000 7f3585c00640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3585c00640 1 -- 172.21.15.46:0/1828349609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55919b999de0 con 0x55919b92a350 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3585200640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3585c00640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b93af50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fd0 tx=0x7f3574004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c03b087bc72671 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3567400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357400fdf0 con 0x55919b92a350 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3578009890 con 0x55919b92a350 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f357402a6e0 con 0x55919b92a350 2024-03-08T23:11:27.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.928+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f357403b050 con 0x55919b92a350 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 10 client.26372 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55919b99b050 con 0x55919b92a350 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3578009890 con 0x55919b92a350 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3574032410 con 0x55919b92a350 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 10 client.26372 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:27.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 10 client.26372 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 20 client.26372 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b99b790 0x55919b99dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 4 client.26372 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55919b89c310 con 0x55919b99b790 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.929+0000 7f3585200640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b99b790 0x55919b99dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f3585200640 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b99b790 0x55919b99dbb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3fbb494dc2a6df38 server_cookie=d905951896ec11ae in_seq=0 out_seq=0 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f3567400640 10 client.26372 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:11:27.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f3567400640 1 -- 172.21.15.46:0/1828349609 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55919b89c310 con 0x55919b99b790 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f3567400640 10 client.26372 handle_command_reply: tid=0 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f358b212740 1 client.26372 shutdown 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 msgr2=0x55919b93af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b93af50 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3574009fd0 tx=0x7f3574004ab0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 shutdown_connections 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.930+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55919b99b790 0x55919b99dbb0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55919b933850 0x55919b9399e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55919b92ac70 0x55919b9392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f3567400640 0 client.26372 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 --2- 172.21.15.46:0/1828349609 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55919b92a350 0x55919b93af50 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 >> 172.21.15.46:0/1828349609 conn(0x55919b924a80 msgr2=0x55919b923200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 shutdown_connections 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 1 -- 172.21.15.46:0/1828349609 wait complete. 2024-03-08T23:11:27.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f358b212740 20 client.26372 trim_cache size 0 max 16384 2024-03-08T23:11:27.947 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:11:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 msgr2=0x7f35680a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 0x7f35680a4eb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x55919b929e20 tx=0x7f3574009500 comp rx=0 tx=0).stop 2024-03-08T23:11:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 msgr2=0x7f358019a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.931+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f356c0042f0 tx=0x7f356c004320 comp rx=0 tx=0).stop 2024-03-08T23:11:27.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 shutdown_connections 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35680a29f0 0x7f35680a4eb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3580109570 0x7f358019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3580103e70 0x7f358019a270 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 --2- 172.21.15.46:0/4085120989 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35801034b0 0x7f3580199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 >> 172.21.15.46:0/4085120989 conn(0x7f35800fef60 msgr2=0x7f35801008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 shutdown_connections 2024-03-08T23:11:27.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:27.932+0000 7f3588600640 1 -- 172.21.15.46:0/4085120989 wait complete. 2024-03-08T23:11:27.952 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:11:27.953 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:11:28.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 -- 172.21.15.46:0/635695758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c101500 msgr2=0x7f735c1018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 --2- 172.21.15.46:0/635695758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c101500 0x7f735c1018e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f734c009900 tx=0x7f734c02bc50 comp rx=0 tx=0).stop 2024-03-08T23:11:28.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 -- 172.21.15.46:0/635695758 shutdown_connections 2024-03-08T23:11:28.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 --2- 172.21.15.46:0/635695758 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c1065d0 0x7f735c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 --2- 172.21.15.46:0/635695758 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101eb0 0x7f735c105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 --2- 172.21.15.46:0/635695758 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c101500 0x7f735c1018e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 -- 172.21.15.46:0/635695758 >> 172.21.15.46:0/635695758 conn(0x7f735c0fcd20 msgr2=0x7f735c0ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 -- 172.21.15.46:0/635695758 shutdown_connections 2024-03-08T23:11:28.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.171+0000 7f7360a00640 1 -- 172.21.15.46:0/635695758 wait complete. 2024-03-08T23:11:28.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.172+0000 7f7360a00640 1 Processor -- start 2024-03-08T23:11:28.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 -- start start 2024-03-08T23:11:28.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 0x7f735c0735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 0x7f735c073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c074070 0x7f735c1a5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1116d0 con 0x7f735c074070 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c111550 con 0x7f735c101500 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7360a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c111850 con 0x7f735c1065d0 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f735a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c074070 0x7f735c1a5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.173+0000 7f7359e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 0x7f735c0735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 0x7f735c073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 0x7f735c0735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46390/0 (socket says 172.21.15.46:46390) 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359e00640 1 -- 172.21.15.46:0/994567025 learned_addr learned my addr 172.21.15.46:0/994567025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 -- 172.21.15.46:0/994567025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 msgr2=0x7f735c0735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 0x7f735c0735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 -- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c074070 msgr2=0x7f735c1a5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c074070 0x7f735c1a5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 -- 172.21.15.46:0/994567025 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735c1a6480 con 0x7f735c1065d0 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.174+0000 7f7359400640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 0x7f735c073b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9c0 tx=0x7f734800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.175+0000 7f7352a00640 1 -- 172.21.15.46:0/994567025 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734800ccc0 con 0x7f735c1065d0 2024-03-08T23:11:28.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.175+0000 7f7352a00640 1 -- 172.21.15.46:0/994567025 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7348005020 con 0x7f735c1065d0 2024-03-08T23:11:28.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.175+0000 7f7352a00640 1 -- 172.21.15.46:0/994567025 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348004a00 con 0x7f735c1065d0 2024-03-08T23:11:28.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.175+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f735c1a6770 con 0x7f735c1065d0 2024-03-08T23:11:28.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.175+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f735c1a6c70 con 0x7f735c1065d0 2024-03-08T23:11:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 Processor -- start 2024-03-08T23:11:28.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7352a00640 1 -- 172.21.15.46:0/994567025 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f73480051d0 con 0x7f735c1065d0 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 -- start start 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314350 0x557a41314730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a4131fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a41278dd0 con 0x557a4131d850 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a41221c70 con 0x557a41314350 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.178+0000 7f7363820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a41274e70 con 0x557a41314c70 2024-03-08T23:11:28.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f735a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314350 0x557a41314730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45482/0 (socket says 172.21.15.46:45482) 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 -- 172.21.15.46:0/2167544538 learned_addr learned my addr 172.21.15.46:0/2167544538 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359400640 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a4131fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 -- 172.21.15.46:0/2167544538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314350 msgr2=0x557a41314730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314350 0x557a41314730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 -- 172.21.15.46:0/2167544538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 msgr2=0x557a4131fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a4131fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.179+0000 7f7359e00640 1 -- 172.21.15.46:0/2167544538 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a412db2b0 con 0x557a41314c70 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7352a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 0x7f73400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7359e00640 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f734c005ea0 tx=0x7f734c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5371d821e0833c05 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7352a00640 1 -- 172.21.15.46:0/994567025 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f73480cf0d0 con 0x7f735c1065d0 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7347000640 1 -- 172.21.15.46:0/2167544538 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c042070 con 0x557a41314c70 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7347000640 1 -- 172.21.15.46:0/2167544538 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734c004240 con 0x557a41314c70 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f7347000640 1 -- 172.21.15.46:0/2167544538 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c004ee0 con 0x557a41314c70 2024-03-08T23:11:28.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.180+0000 7f735a800640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 0x7f73400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- 172.21.15.46:0/2167544538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 msgr2=0x557a4131d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f734c005ea0 tx=0x7f734c009560 comp rx=0 tx=0).stop 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f735a800640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 0x7f73400a4eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7354005fd0 tx=0x7f7354009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- 172.21.15.46:0/2167544538 shutdown_connections 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a4131fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314c70 0x557a4131d310 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 --2- 172.21.15.46:0/2167544538 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314350 0x557a41314730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- 172.21.15.46:0/2167544538 >> 172.21.15.46:0/2167544538 conn(0x557a4130ea80 msgr2=0x557a4130ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- 172.21.15.46:0/2167544538 shutdown_connections 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- 172.21.15.46:0/2167544538 wait complete. 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.181+0000 7f7363820740 1 -- start start 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 Processor -- start 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a4131aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a413205e0 con 0x557a41314350 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a412350c0 con 0x557a41314c70 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a41320760 con 0x557a4131d850 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7363820740 10 client.0 _ll_register_callbacks cb 0x557a4126fec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.182+0000 7f7359e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49650/0 (socket says 172.21.15.46:49650) 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 -- 172.21.15.46:0/3165267438 learned_addr learned my addr 172.21.15.46:0/3165267438 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359400640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f735a800640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a4131aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 -- 172.21.15.46:0/3165267438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 msgr2=0x557a4131aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a4131aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 -- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 msgr2=0x557a4131a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 -- 172.21.15.46:0/3165267438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a40f8d3f0 con 0x557a41314350 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f7359e00640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f734c036040 tx=0x7f734c02ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61ce358ecb30ab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.183+0000 7f733b400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:11:28.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c04d070 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f734c004ee0 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f734c004240 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f734c042050 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f7363820740 10 client.16248 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557a410577d0 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.184+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734c004240 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f734c052870 con 0x557a41314350 2024-03-08T23:11:28.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 10 client.16248 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 10 client.16248 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 20 client.16248 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a4132d6d0 0x557a4132fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 4 client.16248 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557a4124d4f0 con 0x557a4132d6d0 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.185+0000 7f7359400640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a4132d6d0 0x557a4132fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.186+0000 7f7359400640 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a4132d6d0 0x557a4132fac0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6e5d3ed99e527a9 server_cookie=d12a31308d61708f in_seq=0 out_seq=0 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.186+0000 7f733b400640 10 client.16248 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.186+0000 7f733b400640 1 -- 172.21.15.46:0/3165267438 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557a4124d4f0 con 0x557a4132d6d0 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.186+0000 7f733b400640 10 client.16248 handle_command_reply: tid=0 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 client.16248 shutdown 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 msgr2=0x557a4131d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f734c036040 tx=0x7f734c02ce90 comp rx=0 tx=0).stop 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 shutdown_connections 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a4132d6d0 0x557a4132fac0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a4131aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 --2- 172.21.15.46:0/3165267438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a4131d1c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f733b400640 0 client.16248 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 >> 172.21.15.46:0/3165267438 conn(0x557a4130ea80 msgr2=0x557a413107d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 shutdown_connections 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 1 -- 172.21.15.46:0/3165267438 wait complete. 2024-03-08T23:11:28.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.187+0000 7f7363820740 20 client.16248 trim_cache size 0 max 16384 2024-03-08T23:11:28.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 Processor -- start 2024-03-08T23:11:28.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 -- start start 2024-03-08T23:11:28.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a41317810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1b2bb0 con 0x557a4131d850 2024-03-08T23:11:28.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1b06f0 con 0x557a41314c70 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.201+0000 7f7363820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1ae790 con 0x557a41314350 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f735a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a41317810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45508/0 (socket says 172.21.15.46:45508) 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359e00640 1 -- 172.21.15.46:0/2269077164 learned_addr learned my addr 172.21.15.46:0/2269077164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 -- 172.21.15.46:0/2269077164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 msgr2=0x557a4131cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 -- 172.21.15.46:0/2269077164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 msgr2=0x557a41317810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a41317810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 -- 172.21.15.46:0/2269077164 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a4131ae20 con 0x557a41314c70 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359e00640 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.202+0000 7f7359400640 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131d1c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f73480cc430 tx=0x7f73480bb040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c26ea230d0fd0b72 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7346600640 1 -- 172.21.15.46:0/2269077164 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734801c040 con 0x557a41314c70 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7346600640 1 -- 172.21.15.46:0/2269077164 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734809ba70 con 0x557a41314c70 2024-03-08T23:11:28.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7346600640 1 -- 172.21.15.46:0/2269077164 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348099670 con 0x557a41314c70 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 -- 172.21.15.46:0/2269077164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 msgr2=0x557a4131d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131d1c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f73480cc430 tx=0x7f73480bb040 comp rx=0 tx=0).stop 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 -- 172.21.15.46:0/2269077164 shutdown_connections 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a4131d850 0x557a41317810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a4131d1c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 --2- 172.21.15.46:0/2269077164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a41314350 0x557a4131cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 -- 172.21.15.46:0/2269077164 >> 172.21.15.46:0/2269077164 conn(0x557a4130ea80 msgr2=0x557a41334050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 -- 172.21.15.46:0/2269077164 shutdown_connections 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.203+0000 7f7363820740 1 -- 172.21.15.46:0/2269077164 wait complete. 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.204+0000 7f7363820740 1 -- start start 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.204+0000 7f7363820740 1 Processor -- start 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a41323fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a413837e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1aed90 con 0x557a41314350 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1b0090 con 0x557a41314c70 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f735c1af3d0 con 0x557a4131d850 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7363820740 10 client.0 _ll_register_callbacks cb 0x7f7348005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a41323fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49666/0 (socket says 172.21.15.46:49666) 2024-03-08T23:11:28.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359e00640 1 -- 172.21.15.46:0/2315849820 learned_addr learned my addr 172.21.15.46:0/2315849820 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f735a800640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a413837e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 -- 172.21.15.46:0/2315849820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 msgr2=0x557a413837e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a413837e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 -- 172.21.15.46:0/2315849820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 msgr2=0x557a41323a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359400640 1 -- 172.21.15.46:0/2315849820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a41383d20 con 0x557a41314c70 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f7359e00640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:28.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.205+0000 7f735a800640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a413837e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7359400640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a41323fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f73480cc430 tx=0x7f734809af30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a45290a8bfe6bff3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7348084060 con 0x557a41314c70 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f734809f1b0 con 0x557a41314c70 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f734809ba70 con 0x557a41314c70 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f734801c030 con 0x557a41314c70 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 10 client.26390 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:28.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x557a41385020 con 0x557a41314c70 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f734809ba70 con 0x557a41314c70 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f73480cc6e0 con 0x557a41314c70 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 10 client.26390 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 10 client.26390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.206+0000 7f7363820740 20 client.26390 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a41385670 0x557a41387a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7363820740 4 client.26390 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:11:28.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557a41286310 con 0x557a41385670 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7359e00640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a41385670 0x557a41387a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7359e00640 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a41385670 0x557a41387a90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24fb6c52d91c3094 server_cookie=411df20625825120 in_seq=0 out_seq=0 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f733b400640 10 client.26390 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f733b400640 1 -- 172.21.15.46:0/2315849820 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557a41286310 con 0x557a41385670 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f733b400640 10 client.26390 handle_command_reply: tid=0 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.207+0000 7f7363820740 1 client.26390 shutdown 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 msgr2=0x557a41323fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a41323fc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f73480cc430 tx=0x7f734809af30 comp rx=0 tx=0).stop 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 shutdown_connections 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557a41385670 0x557a41387a90 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557a4131d850 0x557a413837e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f733b400640 0 client.26390 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557a41314c70 0x557a41323fc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 --2- 172.21.15.46:0/2315849820 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557a41314350 0x557a41323a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 >> 172.21.15.46:0/2315849820 conn(0x557a4130ea80 msgr2=0x557a4130d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 shutdown_connections 2024-03-08T23:11:28.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 1 -- 172.21.15.46:0/2315849820 wait complete. 2024-03-08T23:11:28.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.208+0000 7f7363820740 20 client.26390 trim_cache size 0 max 16384 2024-03-08T23:11:28.220 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:11:28.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 msgr2=0x7f73400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 0x7f73400a4eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7354005fd0 tx=0x7f7354009500 comp rx=0 tx=0).stop 2024-03-08T23:11:28.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 msgr2=0x7f735c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:28.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 0x7f735c073b30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f734800e9c0 tx=0x7f734800ee80 comp rx=0 tx=0).stop 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 shutdown_connections 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f73400a29f0 0x7f73400a4eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f735c074070 0x7f735c1a5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f735c1065d0 0x7f735c073b30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 --2- 172.21.15.46:0/994567025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f735c101500 0x7f735c0735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 >> 172.21.15.46:0/994567025 conn(0x7f735c0fcd20 msgr2=0x7f735c0ff020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 shutdown_connections 2024-03-08T23:11:28.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:28.209+0000 7f7360a00640 1 -- 172.21.15.46:0/994567025 wait complete. 2024-03-08T23:11:28.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:28 smithi046 ceph-mon[31859]: pgmap v113: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:28 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2038876749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:11:28.365 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:28 smithi192 ceph-mon[36700]: pgmap v113: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:28.365 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:28 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2038876749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:11:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:28 smithi062 ceph-mon[35761]: pgmap v113: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:11:28.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:28 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2038876749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:11:29.230 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:11:29.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:11:29.231 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:11:29.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 -- 172.21.15.46:0/1333252900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec103e70 msgr2=0x7f80ec110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 --2- 172.21.15.46:0/1333252900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec103e70 0x7f80ec110330 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009c10 tx=0x7f80e402bef0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 -- 172.21.15.46:0/1333252900 shutdown_connections 2024-03-08T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 --2- 172.21.15.46:0/1333252900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec103e70 0x7f80ec110330 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 --2- 172.21.15.46:0/1333252900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec1034b0 0x7f80ec103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.465+0000 7f80f2000640 1 --2- 172.21.15.46:0/1333252900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.466+0000 7f80f2000640 1 -- 172.21.15.46:0/1333252900 >> 172.21.15.46:0/1333252900 conn(0x7f80ec0fef60 msgr2=0x7f80ec101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.466+0000 7f80f2000640 1 -- 172.21.15.46:0/1333252900 shutdown_connections 2024-03-08T23:11:29.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.466+0000 7f80f2000640 1 -- 172.21.15.46:0/1333252900 wait complete. 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 Processor -- start 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 -- start start 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec1034b0 0x7f80ec19fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec103e70 0x7f80ec19ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1138a0 con 0x7f80ec103e70 2024-03-08T23:11:29.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec113720 con 0x7f80ec1034b0 2024-03-08T23:11:29.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80f2000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec113a20 con 0x7f80ec109570 2024-03-08T23:11:29.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.467+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45524/0 (socket says 172.21.15.46:45524) 2024-03-08T23:11:29.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 -- 172.21.15.46:0/2730556640 learned_addr learned my addr 172.21.15.46:0/2730556640 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80eb400640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec1034b0 0x7f80ec19fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80e3e00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec103e70 0x7f80ec19ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 -- 172.21.15.46:0/2730556640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec1034b0 msgr2=0x7f80ec19fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec1034b0 0x7f80ec19fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 -- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec103e70 msgr2=0x7f80ec19ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec103e70 0x7f80ec19ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.468+0000 7f80ebe00640 1 -- 172.21.15.46:0/2730556640 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec19a3e0 con 0x7f80ec109570 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80ebe00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f80e4004af0 tx=0x7f80e40336b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80e8c00640 1 -- 172.21.15.46:0/2730556640 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e4043020 con 0x7f80ec109570 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80e8c00640 1 -- 172.21.15.46:0/2730556640 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e4003e50 con 0x7f80ec109570 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80e8c00640 1 -- 172.21.15.46:0/2730556640 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e403e450 con 0x7f80ec109570 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec19a670 con 0x7f80ec109570 2024-03-08T23:11:29.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.469+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec1a69e0 con 0x7f80ec109570 2024-03-08T23:11:29.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.472+0000 7f80f4da7740 1 Processor -- start 2024-03-08T23:11:29.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80e8c00640 1 -- 172.21.15.46:0/2730556640 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f80e4046020 con 0x7f80ec109570 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 -- start start 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x55770777a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 0x5577077835a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707785ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577076dea50 con 0x557707783ae0 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557707687b00 con 0x55770777af00 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.473+0000 7f80f4da7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577076daaf0 con 0x55770777a5e0 2024-03-08T23:11:29.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x55770777a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 0x5577077835a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x55770777a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45536/0 (socket says 172.21.15.46:45536) 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e3e00640 1 -- 172.21.15.46:0/3572769896 learned_addr learned my addr 172.21.15.46:0/3572769896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e3e00640 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707785ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 -- 172.21.15.46:0/3572769896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 msgr2=0x55770777a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x55770777a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 -- 172.21.15.46:0/3572769896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 msgr2=0x557707785ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707785ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 -- 172.21.15.46:0/3572769896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577076a51e0 con 0x55770777af00 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e3e00640 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707785ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80e8c00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 0x7f80d00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.474+0000 7f80eb400640 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 0x5577077835a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009c10 tx=0x7f80dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5a0e1cf54124292 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80e8c00640 1 -- 172.21.15.46:0/2730556640 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f80e40f15b0 con 0x7f80ec109570 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80e3e00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 0x7f80d00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80dbe00640 1 -- 172.21.15.46:0/3572769896 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc004230 con 0x55770777af00 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80dbe00640 1 -- 172.21.15.46:0/3572769896 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0043d0 con 0x55770777af00 2024-03-08T23:11:29.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80dbe00640 1 -- 172.21.15.46:0/3572769896 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0197c0 con 0x55770777af00 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80f4da7740 1 -- 172.21.15.46:0/3572769896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 msgr2=0x5577077835a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 0x5577077835a0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f80dc009c10 tx=0x7f80dc011810 comp rx=0 tx=0).stop 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.475+0000 7f80e3e00640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 0x7f80d00a4eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x557707775ea0 tx=0x7f80d4013040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 -- 172.21.15.46:0/3572769896 shutdown_connections 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707785ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777af00 0x5577077835a0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3572769896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x55770777a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 -- 172.21.15.46:0/3572769896 >> 172.21.15.46:0/3572769896 conn(0x557707774d10 msgr2=0x5577077750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 -- 172.21.15.46:0/3572769896 shutdown_connections 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 -- 172.21.15.46:0/3572769896 wait complete. 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 -- start start 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.476+0000 7f80f4da7740 1 Processor -- start 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x557707780ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x557707781010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577077868d0 con 0x557707783ae0 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55770769af60 con 0x55770777a5e0 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557707786a50 con 0x55770777af00 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80f4da7740 10 client.0 _ll_register_callbacks cb 0x557707686b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49702/0 (socket says 172.21.15.46:49702) 2024-03-08T23:11:29.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80ebe00640 1 -- 172.21.15.46:0/3950816156 learned_addr learned my addr 172.21.15.46:0/3950816156 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80e3e00640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x557707781010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.477+0000 7f80eb400640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x557707780ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 -- 172.21.15.46:0/3950816156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 msgr2=0x557707781010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x557707781010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 -- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 msgr2=0x557707780ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x557707780ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 -- 172.21.15.46:0/3950816156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577074bb7e0 con 0x557707783ae0 2024-03-08T23:11:29.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f80e40ba330 tx=0x7f80e40bd5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec79a8989cabee6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:11:29.489 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40c1ab0 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f80e40f11d0 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.478+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80dc0043d0 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80e40c6100 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 10 client.16266 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55770751bf70 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0043d0 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f80e4004b40 con 0x557707783ae0 2024-03-08T23:11:29.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 10 client.16266 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 20 client.16266 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557707793e10 0x557707796250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 4 client.16266 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.479+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5577076ebf90 con 0x557707793e10 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.480+0000 7f80eb400640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557707793e10 0x557707796250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.480+0000 7f80eb400640 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557707793e10 0x557707796250 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72c6292c145b28b1 server_cookie=c5eb6f06afe1c6e2 in_seq=0 out_seq=0 2024-03-08T23:11:29.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.480+0000 7f80d8c00640 10 client.16266 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80d8c00640 1 -- 172.21.15.46:0/3950816156 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5577076ebf90 con 0x557707793e10 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80d8c00640 10 client.16266 handle_command_reply: tid=0 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 client.16266 shutdown 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 msgr2=0x557707781550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f80e40ba330 tx=0x7f80e40bd5b0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 shutdown_connections 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557707793e10 0x557707796250 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x557707781550 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x557707781010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3950816156 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x557707780ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80d8c00640 0 client.16266 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 >> 172.21.15.46:0/3950816156 conn(0x557707774d10 msgr2=0x557707776a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 shutdown_connections 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.481+0000 7f80f4da7740 1 -- 172.21.15.46:0/3950816156 wait complete. 2024-03-08T23:11:29.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.482+0000 7f80f4da7740 20 client.16266 trim_cache size 0 max 16384 2024-03-08T23:11:29.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.498+0000 7f80f4da7740 1 Processor -- start 2024-03-08T23:11:29.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 -- start start 2024-03-08T23:11:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x557707783200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 0x55770777e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1b2a10 con 0x55770777af00 2024-03-08T23:11:29.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1b0550 con 0x557707783ae0 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80f4da7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1ae5f0 con 0x55770777a5e0 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80e3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80e3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49708/0 (socket says 172.21.15.46:49708) 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80e3e00640 1 -- 172.21.15.46:0/378094809 learned_addr learned my addr 172.21.15.46:0/378094809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80eb400640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x557707783200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.499+0000 7f80ebe00640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 0x55770777e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 -- 172.21.15.46:0/378094809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 msgr2=0x557707783200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x557707783200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 -- 172.21.15.46:0/378094809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 msgr2=0x55770777e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 0x55770777e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 -- 172.21.15.46:0/378094809 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577073b3170 con 0x55770777af00 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80ebe00640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 0x55770777e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80eb400640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x557707783200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.500+0000 7f80e3e00640 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f80d4004190 tx=0x7f80d4005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9f2b88c749d10a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80db400640 1 -- 172.21.15.46:0/378094809 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4003c40 con 0x55770777af00 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80db400640 1 -- 172.21.15.46:0/378094809 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80d4003de0 con 0x55770777af00 2024-03-08T23:11:29.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80db400640 1 -- 172.21.15.46:0/378094809 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d400f580 con 0x55770777af00 2024-03-08T23:11:29.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 -- 172.21.15.46:0/378094809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 msgr2=0x55770777dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f80d4004190 tx=0x7f80d4005850 comp rx=0 tx=0).stop 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 -- 172.21.15.46:0/378094809 shutdown_connections 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557707783ae0 0x55770777e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55770777af00 0x55770777dd30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 --2- 172.21.15.46:0/378094809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777a5e0 0x557707783200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 -- 172.21.15.46:0/378094809 >> 172.21.15.46:0/378094809 conn(0x557707774d10 msgr2=0x55770779abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 -- 172.21.15.46:0/378094809 shutdown_connections 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.501+0000 7f80f4da7740 1 -- 172.21.15.46:0/378094809 wait complete. 2024-03-08T23:11:29.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 -- start start 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 Processor -- start 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x5577077ebf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x5577077ec4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1aebf0 con 0x557707783ae0 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1afef0 con 0x55770777a5e0 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec1af230 con 0x55770777af00 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.502+0000 7f80f4da7740 10 client.0 _ll_register_callbacks cb 0x7f80dc0097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:29.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80eb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x5577077ebf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49720/0 (socket says 172.21.15.46:49720) 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 -- 172.21.15.46:0/3595335062 learned_addr learned my addr 172.21.15.46:0/3595335062 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80e3e00640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x5577077ec4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 -- 172.21.15.46:0/3595335062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 msgr2=0x5577077ec4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x5577077ec4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 -- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 msgr2=0x5577077ebf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x5577077ebf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80ebe00640 1 -- 172.21.15.46:0/3595335062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5577076aa050 con 0x557707783ae0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.503+0000 7f80eb400640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x5577077ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80ebe00640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f80e402c430 tx=0x7f80e4004370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85917eaf0fe48023 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e40a0030 con 0x557707783ae0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f80e40a80f0 con 0x557707783ae0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80d4003de0 con 0x557707783ae0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80e40bd730 con 0x557707783ae0 2024-03-08T23:11:29.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80f4da7740 10 client.16272 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5577076aa050 con 0x557707783ae0 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80d4003de0 con 0x557707783ae0 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f80e402cb10 con 0x557707783ae0 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.504+0000 7f80f4da7740 10 client.16272 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80f4da7740 10 client.16272 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80f4da7740 20 client.16272 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5577077f4a70 0x5577077f6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80f4da7740 4 client.16272 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5577076e62e0 con 0x5577077f4a70 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80eb400640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5577077f4a70 0x5577077f6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80eb400640 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5577077f4a70 0x5577077f6e90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af8702e4d384c66f server_cookie=79f1efe28db2044d in_seq=0 out_seq=0 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.505+0000 7f80d8c00640 10 client.16272 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.506+0000 7f80d8c00640 1 -- 172.21.15.46:0/3595335062 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5577076e62e0 con 0x5577077f4a70 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.506+0000 7f80d8c00640 10 client.16272 handle_command_reply: tid=0 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.506+0000 7f80f4da7740 1 client.16272 shutdown 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 msgr2=0x55770778a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f80e402c430 tx=0x7f80e4004370 comp rx=0 tx=0).stop 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 shutdown_connections 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5577077f4a70 0x5577077f6e90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557707783ae0 0x55770778a180 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55770777af00 0x5577077ec4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80d8c00640 0 client.16272 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 --2- 172.21.15.46:0/3595335062 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55770777a5e0 0x5577077ebf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 >> 172.21.15.46:0/3595335062 conn(0x557707774d10 msgr2=0x557707780a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 shutdown_connections 2024-03-08T23:11:29.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 1 -- 172.21.15.46:0/3595335062 wait complete. 2024-03-08T23:11:29.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.507+0000 7f80f4da7740 20 client.16272 trim_cache size 0 max 16384 2024-03-08T23:11:29.517 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:11:29.517 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:11:29.517 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "7b1394f0-d716-48e2-a0e2-ebc8801b4f1f", 2024-03-08T23:11:29.518 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:11:29.518 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:11:29.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 msgr2=0x7f80d00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 0x7f80d00a4eb0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x557707775ea0 tx=0x7f80d4013040 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 msgr2=0x7f80ec199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f80e4004af0 tx=0x7f80e40336b0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 shutdown_connections 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f80d00a29f0 0x7f80d00a4eb0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f80ec109570 0x7f80ec199b20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f80ec103e70 0x7f80ec19ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 --2- 172.21.15.46:0/2730556640 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f80ec1034b0 0x7f80ec19fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 >> 172.21.15.46:0/2730556640 conn(0x7f80ec0fef60 msgr2=0x7f80ec100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 shutdown_connections 2024-03-08T23:11:29.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.508+0000 7f80f2000640 1 -- 172.21.15.46:0/2730556640 wait complete. 2024-03-08T23:11:29.537 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:11:29.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:11:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.815+0000 7fab2e800640 1 -- 172.21.15.46:0/2296467928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 msgr2=0x7fab280ff840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.815+0000 7fab2e800640 1 --2- 172.21.15.46:0/2296467928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab280ff840 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fab10009900 tx=0x7fab1002bc30 comp rx=0 tx=0).stop 2024-03-08T23:11:29.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 -- 172.21.15.46:0/2296467928 shutdown_connections 2024-03-08T23:11:29.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 --2- 172.21.15.46:0/2296467928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab280ffd80 0x7fab2810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 --2- 172.21.15.46:0/2296467928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab280ff840 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 --2- 172.21.15.46:0/2296467928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab281073d0 0x7fab281077b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 -- 172.21.15.46:0/2296467928 >> 172.21.15.46:0/2296467928 conn(0x7fab28076340 msgr2=0x7fab280fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 -- 172.21.15.46:0/2296467928 shutdown_connections 2024-03-08T23:11:29.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.816+0000 7fab2e800640 1 -- 172.21.15.46:0/2296467928 wait complete. 2024-03-08T23:11:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.817+0000 7fab2e800640 1 Processor -- start 2024-03-08T23:11:29.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.817+0000 7fab2e800640 1 -- start start 2024-03-08T23:11:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 0x7fab2819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 0x7fab2819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab28111650 con 0x7fab280ffd80 2024-03-08T23:11:29.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281114d0 con 0x7fab280ff3c0 2024-03-08T23:11:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab2e800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281117d0 con 0x7fab281073d0 2024-03-08T23:11:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46488/0 (socket says 172.21.15.46:46488) 2024-03-08T23:11:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab27400640 1 -- 172.21.15.46:0/720141091 learned_addr learned my addr 172.21.15.46:0/720141091 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab27e00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 0x7fab2819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.818+0000 7fab1ea00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 0x7fab2819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 -- 172.21.15.46:0/720141091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 msgr2=0x7fab2819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 0x7fab2819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 -- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 msgr2=0x7fab2819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 0x7fab2819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 -- 172.21.15.46:0/720141091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2819ed40 con 0x7fab280ff3c0 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27e00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 0x7fab2819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab1ea00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 0x7fab2819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.819+0000 7fab27400640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9f0 tx=0x7fab1800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.820+0000 7fab24c00640 1 -- 172.21.15.46:0/720141091 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1800cd80 con 0x7fab280ff3c0 2024-03-08T23:11:29.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.820+0000 7fab24c00640 1 -- 172.21.15.46:0/720141091 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab18010910 con 0x7fab280ff3c0 2024-03-08T23:11:29.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.820+0000 7fab24c00640 1 -- 172.21.15.46:0/720141091 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab180188f0 con 0x7fab280ff3c0 2024-03-08T23:11:29.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.820+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2819f030 con 0x7fab280ff3c0 2024-03-08T23:11:29.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.820+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab281a68c0 con 0x7fab280ff3c0 2024-03-08T23:11:29.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.824+0000 7fab24c00640 1 -- 172.21.15.46:0/720141091 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fab18010ac0 con 0x7fab280ff3c0 2024-03-08T23:11:29.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.825+0000 7fab31400740 1 Processor -- start 2024-03-08T23:11:29.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab24c00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 0x7fab0c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 -- start start 2024-03-08T23:11:29.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab24c00640 1 -- 172.21.15.46:0/720141091 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fab180cf520 con 0x7fab280ff3c0 2024-03-08T23:11:29.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ef540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629440a8410 con 0x5629440e6ea0 2024-03-08T23:11:29.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56294404ad50 con 0x5629440efa80 2024-03-08T23:11:29.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab31400740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562943ff3b80 con 0x5629440e6580 2024-03-08T23:11:29.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab27e00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 0x7fab0c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.826+0000 7fab27400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab1ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ef540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45588/0 (socket says 172.21.15.46:45588) 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 -- 172.21.15.46:0/3465810490 learned_addr learned my addr 172.21.15.46:0/3465810490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ef540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49736/0 (socket says 172.21.15.46:49736) 2024-03-08T23:11:29.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27e00640 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 -- 172.21.15.46:0/3465810490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 msgr2=0x5629440f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 -- 172.21.15.46:0/3465810490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 msgr2=0x5629440ef540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ef540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 -- 172.21.15.46:0/3465810490 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562943e297d0 con 0x5629440e6580 2024-03-08T23:11:29.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.827+0000 7fab27400640 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab18097c80 tx=0x7fab1809bfe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25120b7cf318f275 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.828+0000 7fab1cc00640 1 -- 172.21.15.46:0/3465810490 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab180a3fc0 con 0x5629440e6580 2024-03-08T23:11:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.828+0000 7fab27e00640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 0x7fab0c0a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fab200099e0 tx=0x7fab2000ff60 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.828+0000 7fab1cc00640 1 -- 172.21.15.46:0/3465810490 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab180860e0 con 0x5629440e6580 2024-03-08T23:11:29.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.828+0000 7fab1cc00640 1 -- 172.21.15.46:0/3465810490 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1809cb50 con 0x5629440e6580 2024-03-08T23:11:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 -- 172.21.15.46:0/3465810490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 msgr2=0x5629440e6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab18097c80 tx=0x7fab1809bfe0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 -- 172.21.15.46:0/3465810490 shutdown_connections 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ef540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 --2- 172.21.15.46:0/3465810490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x5629440e6960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 -- 172.21.15.46:0/3465810490 >> 172.21.15.46:0/3465810490 conn(0x5629440e0cb0 msgr2=0x5629440e1090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 -- 172.21.15.46:0/3465810490 shutdown_connections 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.829+0000 7fab31400740 1 -- 172.21.15.46:0/3465810490 wait complete. 2024-03-08T23:11:29.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.830+0000 7fab31400740 1 -- start start 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.830+0000 7fab31400740 1 Processor -- start 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.830+0000 7fab31400740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ed080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440efa80 0x5629440ed5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562944006ee0 con 0x5629440e6ea0 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562944046df0 con 0x5629440e6580 2024-03-08T23:11:29.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629440f2870 con 0x5629440efa80 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab31400740 10 client.0 _ll_register_callbacks cb 0x56294400d8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:46514/0 (socket says 172.21.15.46:46514) 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 -- 172.21.15.46:0/3547036209 learned_addr learned my addr 172.21.15.46:0/3547036209 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ed080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27e00640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440efa80 0x5629440ed5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 -- 172.21.15.46:0/3547036209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440efa80 msgr2=0x5629440ed5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440efa80 0x5629440ed5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 -- 172.21.15.46:0/3547036209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 msgr2=0x5629440ed080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ed080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.831+0000 7fab27400640 1 -- 172.21.15.46:0/3547036209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562943d5f3f0 con 0x5629440e6580 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.832+0000 7fab27400640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fab1809b520 tx=0x7fab1809faf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bff9b3ed45435d43 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.832+0000 7fab15600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.833+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab180a3fc0 con 0x5629440e6580 2024-03-08T23:11:29.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.833+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fab1809cb50 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.833+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab180860e0 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.833+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab1809cd90 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 10 client.26420 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562943d1f0f0 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab180860e0 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fab18097670 con 0x5629440e6580 2024-03-08T23:11:29.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 10 client.26420 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 10 client.26420 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 20 client.26420 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629440ffbf0 0x562944101fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 4 client.26420 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562944058290 con 0x5629440ffbf0 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.834+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629440ffbf0 0x562944101fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.845 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.836+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629440ffbf0 0x562944101fe0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5765b5344f58c16 server_cookie=2419a5990f27d18e in_seq=0 out_seq=0 2024-03-08T23:11:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.836+0000 7fab15600640 10 client.26420 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab15600640 1 -- 172.21.15.46:0/3547036209 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562944058290 con 0x5629440ffbf0 2024-03-08T23:11:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab15600640 10 client.26420 handle_command_reply: tid=0 2024-03-08T23:11:29.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 client.26420 shutdown 2024-03-08T23:11:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 msgr2=0x5629440ecb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fab1809b520 tx=0x7fab1809faf0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 shutdown_connections 2024-03-08T23:11:29.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629440ffbf0 0x562944101fe0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440efa80 0x5629440ed5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6ea0 0x5629440ed080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 --2- 172.21.15.46:0/3547036209 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6580 0x5629440ecb40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab15600640 0 client.26420 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.837+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 >> 172.21.15.46:0/3547036209 conn(0x5629440e0cb0 msgr2=0x5629440e2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.838+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 shutdown_connections 2024-03-08T23:11:29.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.838+0000 7fab31400740 1 -- 172.21.15.46:0/3547036209 wait complete. 2024-03-08T23:11:29.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.838+0000 7fab31400740 20 client.26420 trim_cache size 0 max 16384 2024-03-08T23:11:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 Processor -- start 2024-03-08T23:11:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 -- start start 2024-03-08T23:11:29.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6ea0 0x5629440ed3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440fb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281b2e30 con 0x5629440e6580 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281b0970 con 0x5629440efa80 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.874+0000 7fab31400740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281aea10 con 0x5629440e6ea0 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49754/0 (socket says 172.21.15.46:49754) 2024-03-08T23:11:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 -- 172.21.15.46:0/3112152776 learned_addr learned my addr 172.21.15.46:0/3112152776 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27e00640 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 -- 172.21.15.46:0/3112152776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6ea0 msgr2=0x5629440ed3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6ea0 0x5629440ed3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 -- 172.21.15.46:0/3112152776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 msgr2=0x5629440fb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440fb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.875+0000 7fab27400640 1 -- 172.21.15.46:0/3112152776 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629440f21e0 con 0x5629440e6580 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.876+0000 7fab27400640 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9c0 tx=0x7fab18097bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6ecfb5a2559e2f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.876+0000 7fab17e00640 1 -- 172.21.15.46:0/3112152776 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1809b180 con 0x5629440e6580 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.876+0000 7fab17e00640 1 -- 172.21.15.46:0/3112152776 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab18005620 con 0x5629440e6580 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.876+0000 7fab17e00640 1 -- 172.21.15.46:0/3112152776 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab180860b0 con 0x5629440e6580 2024-03-08T23:11:29.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 -- 172.21.15.46:0/3112152776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 msgr2=0x5629440ef270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9c0 tx=0x7fab18097bf0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 -- 172.21.15.46:0/3112152776 shutdown_connections 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440efa80 0x5629440fb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6ea0 0x5629440ed3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 --2- 172.21.15.46:0/3112152776 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440e6580 0x5629440ef270 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.877+0000 7fab31400740 1 -- 172.21.15.46:0/3112152776 >> 172.21.15.46:0/3112152776 conn(0x5629440e0cb0 msgr2=0x5629440fab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.878+0000 7fab31400740 1 -- 172.21.15.46:0/3112152776 shutdown_connections 2024-03-08T23:11:29.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.878+0000 7fab31400740 1 -- 172.21.15.46:0/3112152776 wait complete. 2024-03-08T23:11:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.878+0000 7fab31400740 1 -- start start 2024-03-08T23:11:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.878+0000 7fab31400740 1 Processor -- start 2024-03-08T23:11:29.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6ea0 0x5629441580a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 0x562944152460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281af010 con 0x5629440efa80 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281b0310 con 0x5629440e6ea0 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab281af650 con 0x5629440e6580 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab31400740 10 client.0 _ll_register_callbacks cb 0x7fab180973c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:11:29.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.879+0000 7fab27400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45626/0 (socket says 172.21.15.46:45626) 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 -- 172.21.15.46:0/3641202933 learned_addr learned my addr 172.21.15.46:0/3641202933 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27e00640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 0x562944152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6ea0 0x5629441580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 -- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6ea0 msgr2=0x5629441580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6ea0 0x5629441580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 -- 172.21.15.46:0/3641202933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 msgr2=0x562944152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:11:29.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 0x562944152460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 -- 172.21.15.46:0/3641202933 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562944152ba0 con 0x5629440e6580 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27e00640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 0x562944152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab27400640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab1809b7b0 tx=0x7fab18097ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de585c9f7cf5a4c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.880+0000 7fab15600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1809b180 con 0x5629440e6580 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fab18097780 con 0x5629440e6580 2024-03-08T23:11:29.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab18005620 con 0x5629440e6580 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab180ab050 con 0x5629440e6580 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab31400740 10 client.26422 fetch_fsmap learned FSMap version 27 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x562944152ba0 con 0x5629440e6580 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab18005620 con 0x5629440e6580 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.881+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fab18018a90 con 0x5629440e6580 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 10 client.26422 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:11:29.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 10 client.26422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 20 client.26422 populate_metadata read hostname 'smithi046' 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56294415f670 0x562944161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 4 client.26422 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5629440525e0 con 0x56294415f670 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.882+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56294415f670 0x562944161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab1ea00640 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56294415f670 0x562944161a90 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf110c809456c388 server_cookie=915de1238f5a9f6a in_seq=0 out_seq=0 2024-03-08T23:11:29.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab15600640 10 client.26422 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab15600640 1 -- 172.21.15.46:0/3641202933 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5629440525e0 con 0x56294415f670 2024-03-08T23:11:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab15600640 10 client.26422 handle_command_reply: tid=0 2024-03-08T23:11:29.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab31400740 1 client.26422 shutdown 2024-03-08T23:11:29.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 msgr2=0x562944157b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.883+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fab1809b7b0 tx=0x7fab18097ac0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 shutdown_connections 2024-03-08T23:11:29.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56294415f670 0x562944161a90 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629440efa80 0x562944152460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629440e6ea0 0x5629441580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 --2- 172.21.15.46:0/3641202933 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629440e6580 0x562944157b60 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 >> 172.21.15.46:0/3641202933 conn(0x5629440e0cb0 msgr2=0x5629440dff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab15600640 0 client.26422 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.884+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 shutdown_connections 2024-03-08T23:11:29.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.885+0000 7fab31400740 1 -- 172.21.15.46:0/3641202933 wait complete. 2024-03-08T23:11:29.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.885+0000 7fab31400740 20 client.26422 trim_cache size 0 max 16384 2024-03-08T23:11:29.894 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:11:29.894 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:11:29.894 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: "7b1394f0-d716-48e2-a0e2-ebc8801b4f1f": { 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "7b1394f0-d716-48e2-a0e2-ebc8801b4f1f", 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:11:29.895 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:11:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 msgr2=0x7fab0c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 0x7fab0c0a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fab200099e0 tx=0x7fab2000ff60 comp rx=0 tx=0).stop 2024-03-08T23:11:29.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 msgr2=0x7fab28199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fab1800e9f0 tx=0x7fab1800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 shutdown_connections 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fab0c0a29f0 0x7fab0c0a4eb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fab281073d0 0x7fab2819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fab280ffd80 0x7fab2819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 --2- 172.21.15.46:0/720141091 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fab280ff3c0 0x7fab28199d50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.886+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 >> 172.21.15.46:0/720141091 conn(0x7fab28076340 msgr2=0x7fab280fd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:11:29.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.887+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 shutdown_connections 2024-03-08T23:11:29.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:11:29.887+0000 7fab2e800640 1 -- 172.21.15.46:0/720141091 wait complete. 2024-03-08T23:11:29.910 INFO:tasks.cephfs.filesystem:scrub status for tag:7b1394f0-d716-48e2-a0e2-ebc8801b4f1f - {'path': '/', 'tag': '7b1394f0-d716-48e2-a0e2-ebc8801b4f1f', 'options': 'recursive,force'} 2024-03-08T23:11:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: pgmap v114: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-08T23:11:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:11:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:11:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:11:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:11:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: pgmap v114: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:11:30.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:11:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: pgmap v114: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2024-03-08T23:11:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:11:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:11:30.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:11:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:11:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:30.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:31.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:31 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:31.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:31 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:31.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:31 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:11:32.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:32 smithi062 ceph-mon[35761]: pgmap v115: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-08T23:11:32.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:32 smithi046 ceph-mon[31859]: pgmap v115: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-08T23:11:32.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:32 smithi192 ceph-mon[36700]: pgmap v115: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.5 KiB/s wr, 5 op/s 2024-03-08T23:11:34.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:34 smithi062 ceph-mon[35761]: pgmap v116: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-03-08T23:11:34.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:34 smithi046 ceph-mon[31859]: pgmap v116: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-03-08T23:11:34.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:34 smithi192 ceph-mon[36700]: pgmap v116: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.6 KiB/s wr, 9 op/s 2024-03-08T23:11:36.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:36 smithi046 ceph-mon[31859]: pgmap v117: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:36.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:36 smithi192 ceph-mon[36700]: pgmap v117: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:36.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:36 smithi062 ceph-mon[35761]: pgmap v117: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:38.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:38 smithi046 ceph-mon[31859]: pgmap v118: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:38 smithi192 ceph-mon[36700]: pgmap v118: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:38.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:38 smithi062 ceph-mon[35761]: pgmap v118: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.6 KiB/s wr, 6 op/s 2024-03-08T23:11:40.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:40 smithi046 ceph-mon[31859]: pgmap v119: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-03-08T23:11:40.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:40 smithi192 ceph-mon[36700]: pgmap v119: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-03-08T23:11:40.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:40 smithi062 ceph-mon[35761]: pgmap v119: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 9 op/s 2024-03-08T23:11:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:42 smithi046 ceph-mon[31859]: pgmap v120: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-03-08T23:11:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:42 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:42.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:42 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:42.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:42 smithi192 ceph-mon[36700]: pgmap v120: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-03-08T23:11:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:42 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:42.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:42 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:42.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:42 smithi062 ceph-mon[35761]: pgmap v120: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 4.2 KiB/s wr, 6 op/s 2024-03-08T23:11:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:42 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:42 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:43.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:43 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:11:43.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:43 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:11:43.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:43 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:11:44.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:44 smithi192 ceph-mon[36700]: pgmap v121: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-03-08T23:11:44.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:44 smithi062 ceph-mon[35761]: pgmap v121: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-03-08T23:11:44.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:44 smithi046 ceph-mon[31859]: pgmap v121: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 7.8 KiB/s wr, 9 op/s 2024-03-08T23:11:46.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:46 smithi046 ceph-mon[31859]: pgmap v122: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:46.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:46 smithi192 ceph-mon[36700]: pgmap v122: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:46.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:46 smithi062 ceph-mon[35761]: pgmap v122: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:48.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:48 smithi192 ceph-mon[36700]: pgmap v123: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:48.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:48 smithi062 ceph-mon[35761]: pgmap v123: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:48 smithi046 ceph-mon[31859]: pgmap v123: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2024-03-08T23:11:50.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:50 smithi192 ceph-mon[36700]: pgmap v124: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-08T23:11:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:50 smithi062 ceph-mon[35761]: pgmap v124: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-08T23:11:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:50 smithi046 ceph-mon[31859]: pgmap v124: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.9 KiB/s wr, 8 op/s 2024-03-08T23:11:52.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:52 smithi192 ceph-mon[36700]: pgmap v125: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 5 op/s 2024-03-08T23:11:52.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:52 smithi062 ceph-mon[35761]: pgmap v125: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 5 op/s 2024-03-08T23:11:52.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:52 smithi046 ceph-mon[31859]: pgmap v125: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.7 KiB/s wr, 5 op/s 2024-03-08T23:11:54.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:54 smithi192 ceph-mon[36700]: pgmap v126: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-08T23:11:54.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:54 smithi062 ceph-mon[35761]: pgmap v126: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-08T23:11:54.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:54 smithi046 ceph-mon[31859]: pgmap v126: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2024-03-08T23:11:56.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:56 smithi192 ceph-mon[36700]: pgmap v127: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:56.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:56 smithi062 ceph-mon[35761]: pgmap v127: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:56.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:56 smithi046 ceph-mon[31859]: pgmap v127: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:58.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:11:58 smithi192 ceph-mon[36700]: pgmap v128: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:58.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:11:58 smithi062 ceph-mon[35761]: pgmap v128: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:58.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:11:58 smithi046 ceph-mon[31859]: pgmap v128: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2024-03-08T23:11:59.911 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:11:59.912 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:12:00.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 -- 172.21.15.46:0/3090621981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 msgr2=0x7f4760114760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 --2- 172.21.15.46:0/3090621981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f4760114760 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4758009900 tx=0x7f475802bc30 comp rx=0 tx=0).stop 2024-03-08T23:12:00.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 -- 172.21.15.46:0/3090621981 shutdown_connections 2024-03-08T23:12:00.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 --2- 172.21.15.46:0/3090621981 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f4760114760 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 --2- 172.21.15.46:0/3090621981 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 0x7f4760107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 --2- 172.21.15.46:0/3090621981 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f476010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 -- 172.21.15.46:0/3090621981 >> 172.21.15.46:0/3090621981 conn(0x7f4760075830 msgr2=0x7f4760077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 -- 172.21.15.46:0/3090621981 shutdown_connections 2024-03-08T23:12:00.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.149+0000 7f4764c00640 1 -- 172.21.15.46:0/3090621981 wait complete. 2024-03-08T23:12:00.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.150+0000 7f4764c00640 1 Processor -- start 2024-03-08T23:12:00.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.150+0000 7f4764c00640 1 -- start start 2024-03-08T23:12:00.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 0x7f476019e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f476019e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760117d00 con 0x7f4760107890 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760117b80 con 0x7f4760108250 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f4764c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760117e80 con 0x7f476010d8a0 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f475e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f475e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44880/0 (socket says 172.21.15.46:44880) 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f475e800640 1 -- 172.21.15.46:0/3156244546 learned_addr learned my addr 172.21.15.46:0/3156244546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f475de00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 0x7f476019e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.151+0000 7f475d400640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f476019e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 -- 172.21.15.46:0/3156244546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 msgr2=0x7f476019e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f476019e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 -- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 msgr2=0x7f476019e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 0x7f476019e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 -- 172.21.15.46:0/3156244546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47601a30e0 con 0x7f476010d8a0 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f475e800640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f475802efd0 tx=0x7f4758004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f4756a00640 1 -- 172.21.15.46:0/3156244546 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4758041070 con 0x7f476010d8a0 2024-03-08T23:12:00.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47601a3370 con 0x7f476010d8a0 2024-03-08T23:12:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f4756a00640 1 -- 172.21.15.46:0/3156244546 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47580054e0 con 0x7f476010d8a0 2024-03-08T23:12:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f4756a00640 1 -- 172.21.15.46:0/3156244546 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4758003cc0 con 0x7f476010d8a0 2024-03-08T23:12:00.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.152+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47601aac60 con 0x7f476010d8a0 2024-03-08T23:12:00.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4767834740 1 Processor -- start 2024-03-08T23:12:00.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4756a00640 1 -- 172.21.15.46:0/3156244546 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4758044070 con 0x7f476010d8a0 2024-03-08T23:12:00.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4767834740 1 -- start start 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4767834740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4767834740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.156+0000 7f4767834740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f43081fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f4767834740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f4307d6410 con 0x55f43081da80 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f4767834740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f430778d50 con 0x55f430814580 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f4767834740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f430721b80 con 0x55f430814ea0 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37504/0 (socket says 172.21.15.46:37504) 2024-03-08T23:12:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44896/0 (socket says 172.21.15.46:44896) 2024-03-08T23:12:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475d400640 1 -- 172.21.15.46:0/495840843 learned_addr learned my addr 172.21.15.46:0/495840843 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475d400640 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f43081fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 -- 172.21.15.46:0/495840843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 msgr2=0x55f430814960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 -- 172.21.15.46:0/495840843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 msgr2=0x55f43081fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f43081fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475de00640 1 -- 172.21.15.46:0/495840843 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f4305577d0 con 0x55f430814ea0 2024-03-08T23:12:00.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.157+0000 7f475d400640 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f475de00640 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4750009fa0 tx=0x7f47500118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf3ed3fb6ea5ca8f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f4756a00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 0x7f47440a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f4756a00640 1 -- 172.21.15.46:0/3156244546 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f47580f0e10 con 0x7f476010d8a0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f474f000640 1 -- 172.21.15.46:0/495840843 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750003c30 con 0x55f430814ea0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f475d400640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 0x7f47440a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f474f000640 1 -- 172.21.15.46:0/495840843 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4750003dd0 con 0x55f430814ea0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.158+0000 7f474f000640 1 -- 172.21.15.46:0/495840843 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475000fa50 con 0x55f430814ea0 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 -- 172.21.15.46:0/495840843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 msgr2=0x55f43081d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f475d400640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 0x7f47440a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55f43080fe40 tx=0x7f4748005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4750009fa0 tx=0x7f47500118d0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 -- 172.21.15.46:0/495840843 shutdown_connections 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f43081fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43081d540 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 --2- 172.21.15.46:0/495840843 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430814960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 -- 172.21.15.46:0/495840843 >> 172.21.15.46:0/495840843 conn(0x55f43080ecb0 msgr2=0x55f43080f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 -- 172.21.15.46:0/495840843 shutdown_connections 2024-03-08T23:12:00.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.159+0000 7f4767834740 1 -- 172.21.15.46:0/495840843 wait complete. 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 -- start start 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 Processor -- start 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43081aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f43081da80 0x55f43081b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f430734ee0 con 0x55f430814ea0 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f430774df0 con 0x55f430814580 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.160+0000 7f4767834740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f430820870 con 0x55f43081da80 2024-03-08T23:12:00.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f4767834740 10 client.0 _ll_register_callbacks cb 0x55f43073b8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33188/0 (socket says 172.21.15.46:33188) 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475d400640 1 -- 172.21.15.46:0/1485964432 learned_addr learned my addr 172.21.15.46:0/1485964432 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475e800640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f43081da80 0x55f43081b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43081aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 -- 172.21.15.46:0/1485964432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f43081da80 msgr2=0x55f43081b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f43081da80 0x55f43081b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 -- 172.21.15.46:0/1485964432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 msgr2=0x55f43081b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475de00640 1 -- 172.21.15.46:0/1485964432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f43048d3f0 con 0x55f430814580 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.161+0000 7f475d400640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f475de00640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43081aad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f475000a010 tx=0x7f4750009990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7a2eca4b45f595 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4743400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750028020 con 0x55f430814580 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f475000f9e0 con 0x55f430814580 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4750003dd0 con 0x55f430814580 2024-03-08T23:12:00.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4750033050 con 0x55f430814580 2024-03-08T23:12:00.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4767834740 10 client.26438 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:00.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.162+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f43044d0f0 con 0x55f430814580 2024-03-08T23:12:00.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4750003dd0 con 0x55f430814580 2024-03-08T23:12:00.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4750005040 con 0x55f430814580 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 10 client.26438 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 10 client.26438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 20 client.26438 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f43082dca0 0x55f430830090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 4 client.26438 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.163+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f430786290 con 0x55f43082dca0 2024-03-08T23:12:00.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.164+0000 7f475d400640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f43082dca0 0x55f430830090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.164+0000 7f475d400640 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f43082dca0 0x55f430830090 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb942e57ac9158b server_cookie=4c920f6ee01f983a in_seq=0 out_seq=0 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.164+0000 7f4743400640 10 client.26438 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4743400640 1 -- 172.21.15.46:0/1485964432 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f430786290 con 0x55f43082dca0 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4743400640 10 client.26438 handle_command_reply: tid=0 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 client.26438 shutdown 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 msgr2=0x55f43081aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43081aad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f475000a010 tx=0x7f4750009990 comp rx=0 tx=0).stop 2024-03-08T23:12:00.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 shutdown_connections 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f43082dca0 0x55f430830090 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f43081da80 0x55f43081b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f430814ea0 0x55f43081b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 --2- 172.21.15.46:0/1485964432 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43081aad0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 >> 172.21.15.46:0/1485964432 conn(0x55f43080ecb0 msgr2=0x55f43080df10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4743400640 0 client.26438 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 shutdown_connections 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 1 -- 172.21.15.46:0/1485964432 wait complete. 2024-03-08T23:12:00.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.165+0000 7f4767834740 20 client.26438 trim_cache size 0 max 16384 2024-03-08T23:12:00.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 Processor -- start 2024-03-08T23:12:00.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 -- start start 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f430824ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f430815dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b70a0 con 0x55f43081da80 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b4be0 con 0x55f430814580 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.182+0000 7f4767834740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b2c80 con 0x55f430814ea0 2024-03-08T23:12:00.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f430815dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37518/0 (socket says 172.21.15.46:37518) 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 -- 172.21.15.46:0/2324660835 learned_addr learned my addr 172.21.15.46:0/2324660835 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475d400640 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f430824ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 -- 172.21.15.46:0/2324660835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 msgr2=0x55f430824ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f430824ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 -- 172.21.15.46:0/2324660835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 msgr2=0x55f430815dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f430815dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 -- 172.21.15.46:0/2324660835 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f430814cd0 con 0x55f430814580 2024-03-08T23:12:00.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.183+0000 7f475de00640 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4750005410 tx=0x7f4750005bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ba44a26af6fd2b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f474e600640 1 -- 172.21.15.46:0/2324660835 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475003c070 con 0x55f430814580 2024-03-08T23:12:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f474e600640 1 -- 172.21.15.46:0/2324660835 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f475000f540 con 0x55f430814580 2024-03-08T23:12:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f474e600640 1 -- 172.21.15.46:0/2324660835 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750028070 con 0x55f430814580 2024-03-08T23:12:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 -- 172.21.15.46:0/2324660835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 msgr2=0x55f430824ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f4750005410 tx=0x7f4750005bd0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 -- 172.21.15.46:0/2324660835 shutdown_connections 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f430815dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f430824ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 --2- 172.21.15.46:0/2324660835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f430824ab0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 -- 172.21.15.46:0/2324660835 >> 172.21.15.46:0/2324660835 conn(0x55f43080ecb0 msgr2=0x55f4308152a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 -- 172.21.15.46:0/2324660835 shutdown_connections 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.184+0000 7f4767834740 1 -- 172.21.15.46:0/2324660835 wait complete. 2024-03-08T23:12:00.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 -- start start 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 Processor -- start 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43087cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f4308243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.185+0000 7f4767834740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b3280 con 0x55f43081da80 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f4767834740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b4580 con 0x55f430814580 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f4767834740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47601b38c0 con 0x55f430814ea0 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f4767834740 10 client.0 _ll_register_callbacks cb 0x7f4750009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:00.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37530/0 (socket says 172.21.15.46:37530) 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f4308243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 -- 172.21.15.46:0/3555254034 learned_addr learned my addr 172.21.15.46:0/3555254034 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475d400640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43087cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 -- 172.21.15.46:0/3555254034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 msgr2=0x55f43087cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43087cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 -- 172.21.15.46:0/3555254034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 msgr2=0x55f4308243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f4308243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475de00640 1 -- 172.21.15.46:0/3555254034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f430824b20 con 0x55f430814580 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.186+0000 7f475e800640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f4308243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f475de00640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4750011910 tx=0x7f4750005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97fad85b4b3bc463 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475003c040 con 0x55f430814580 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4750004fd0 con 0x55f430814580 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f475000f540 con 0x55f430814580 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4750028040 con 0x55f430814580 2024-03-08T23:12:00.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4767834740 10 client.26444 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55f430884270 con 0x55f430814580 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f475000f540 con 0x55f430814580 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.187+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4750037950 con 0x55f430814580 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 10 client.26444 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 10 client.26444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 20 client.26444 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f4308848e0 0x55f430886d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 4 client.26444 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:12:00.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f4307805e0 con 0x55f4308848e0 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f475d400640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f4308848e0 0x55f430886d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f475d400640 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f4308848e0 0x55f430886d00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fe154bda5813248 server_cookie=edd16e8eea20e86a in_seq=0 out_seq=0 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.188+0000 7f4743400640 10 client.26444 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4743400640 1 -- 172.21.15.46:0/3555254034 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f4307805e0 con 0x55f4308848e0 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4743400640 10 client.26444 handle_command_reply: tid=0 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 client.26444 shutdown 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 msgr2=0x55f43087ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4750011910 tx=0x7f4750005f10 comp rx=0 tx=0).stop 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 shutdown_connections 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f4308848e0 0x55f430886d00 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f43081da80 0x55f4308243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f430814ea0 0x55f43087cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4743400640 0 client.26444 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 --2- 172.21.15.46:0/3555254034 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f430814580 0x55f43087ca40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 >> 172.21.15.46:0/3555254034 conn(0x55f43080ecb0 msgr2=0x55f430815dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 shutdown_connections 2024-03-08T23:12:00.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 1 -- 172.21.15.46:0/3555254034 wait complete. 2024-03-08T23:12:00.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.189+0000 7f4767834740 20 client.26444 trim_cache size 0 max 16384 2024-03-08T23:12:00.201 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:00.201 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:12:00.201 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:12:00.201 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:00.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 msgr2=0x7f47440a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 0x7f47440a4eb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x55f43080fe40 tx=0x7f4748005f90 comp rx=0 tx=0).stop 2024-03-08T23:12:00.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 msgr2=0x7f47601a29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f475802efd0 tx=0x7f4758004270 comp rx=0 tx=0).stop 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 shutdown_connections 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f47440a29f0 0x7f47440a4eb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f476010d8a0 0x7f47601a29b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4760108250 0x7f476019e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 --2- 172.21.15.46:0/3156244546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4760107890 0x7f476019e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 >> 172.21.15.46:0/3156244546 conn(0x7f4760075830 msgr2=0x7f4760075df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 shutdown_connections 2024-03-08T23:12:00.203 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.190+0000 7f4764c00640 1 -- 172.21.15.46:0/3156244546 wait complete. 2024-03-08T23:12:00.214 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:12:00.214 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 -- 172.21.15.46:0/1295380534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 msgr2=0x7f04481018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 --2- 172.21.15.46:0/1295380534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f04481018d0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0438009900 tx=0x7f043802bc40 comp rx=0 tx=0).stop 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 -- 172.21.15.46:0/1295380534 shutdown_connections 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 --2- 172.21.15.46:0/1295380534 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 0x7f044810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 --2- 172.21.15.46:0/1295380534 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0448101ea0 0x7f0448105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 --2- 172.21.15.46:0/1295380534 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f04481018d0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 -- 172.21.15.46:0/1295380534 >> 172.21.15.46:0/1295380534 conn(0x7f04480fcd90 msgr2=0x7f04480ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 -- 172.21.15.46:0/1295380534 shutdown_connections 2024-03-08T23:12:00.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.434+0000 7f044f000640 1 -- 172.21.15.46:0/1295380534 wait complete. 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.435+0000 7f044f000640 1 Processor -- start 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.435+0000 7f044f000640 1 -- start start 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.435+0000 7f044f000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.435+0000 7f044f000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 0x7f044819a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f044f000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f044819e690 0x7f044819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f044f000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448111650 con 0x7f044819e690 2024-03-08T23:12:00.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f044f000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04481114d0 con 0x7f04481014f0 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f044f000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04481117d0 con 0x7f04481065c0 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37536/0 (socket says 172.21.15.46:37536) 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 -- 172.21.15.46:0/910886030 learned_addr learned my addr 172.21.15.46:0/910886030 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f043fe00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 0x7f044819a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f044d000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f044819e690 0x7f044819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 -- 172.21.15.46:0/910886030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 msgr2=0x7f044819a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 0x7f044819a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 -- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f044819e690 msgr2=0x7f044819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f044819e690 0x7f044819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.436+0000 7f0447e00640 1 -- 172.21.15.46:0/910886030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044819f180 con 0x7f04481014f0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f0447e00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0438006010 tx=0x7f043802cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438041070 con 0x7f04481014f0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0438004450 con 0x7f04481014f0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044819f410 con 0x7f04481014f0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438003e30 con 0x7f04481014f0 2024-03-08T23:12:00.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.437+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04481a6bd0 con 0x7f04481014f0 2024-03-08T23:12:00.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.438+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0448068530 con 0x7f04481014f0 2024-03-08T23:12:00.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.441+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0438044070 con 0x7f04481014f0 2024-03-08T23:12:00.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.442+0000 7f0445600640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 0x7f042c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.442+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f04380f1100 con 0x7f04481014f0 2024-03-08T23:12:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.442+0000 7f043fe00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 0x7f042c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:00.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.443+0000 7f043fe00640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 0x7f042c0a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04481008d0 tx=0x7f0430005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:00.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.449+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f04380bd560 con 0x7f04481014f0 2024-03-08T23:12:00.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:00 smithi192 ceph-mon[36700]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-08T23:12:00.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:00 smithi062 ceph-mon[35761]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-08T23:12:00.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.703+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0448101f70 con 0x7f04481014f0 2024-03-08T23:12:00.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.704+0000 7f0445600640 1 -- 172.21.15.46:0/910886030 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11340 (secure 0 0 0) 0x7f04380c1600 con 0x7f04481014f0 2024-03-08T23:12:00.708 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:12:00.708 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:10:43.831330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:12:00.709 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 msgr2=0x7f042c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 0x7f042c0a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f04481008d0 tx=0x7f0430005f90 comp rx=0 tx=0).stop 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 msgr2=0x7f0448199dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0438006010 tx=0x7f043802cb40 comp rx=0 tx=0).stop 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 shutdown_connections 2024-03-08T23:12:00.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f042c0a2970 0x7f042c0a4e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f044819e690 0x7f044819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04481065c0 0x7f044819a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 --2- 172.21.15.46:0/910886030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04481014f0 0x7f0448199dd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 >> 172.21.15.46:0/910886030 conn(0x7f04480fcd90 msgr2=0x7f04480ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 shutdown_connections 2024-03-08T23:12:00.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.708+0000 7f044f000640 1 -- 172.21.15.46:0/910886030 wait complete. 2024-03-08T23:12:00.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:00 smithi046 ceph-mon[31859]: pgmap v129: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-03-08T23:12:00.741 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:00.741 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:12:00.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.994+0000 7f79b1e00640 1 -- 172.21.15.46:0/1027615107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac0ffad0 msgr2=0x7f79ac0ffeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:00.998 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.994+0000 7f79b1e00640 1 --2- 172.21.15.46:0/1027615107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac0ffad0 0x7f79ac0ffeb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f799c009ca0 tx=0x7f799c02c100 comp rx=0 tx=0).stop 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.994+0000 7f79b1e00640 1 -- 172.21.15.46:0/1027615107 shutdown_connections 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 --2- 172.21.15.46:0/1027615107 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac10e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 --2- 172.21.15.46:0/1027615107 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac100480 0x7f79ac105fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 --2- 172.21.15.46:0/1027615107 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac0ffad0 0x7f79ac0ffeb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 -- 172.21.15.46:0/1027615107 >> 172.21.15.46:0/1027615107 conn(0x7f79ac076820 msgr2=0x7f79ac0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 -- 172.21.15.46:0/1027615107 shutdown_connections 2024-03-08T23:12:00.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.995+0000 7f79b1e00640 1 -- 172.21.15.46:0/1027615107 wait complete. 2024-03-08T23:12:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 Processor -- start 2024-03-08T23:12:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 -- start start 2024-03-08T23:12:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0ffad0 0x7f79ac100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac1014c0 0x7f79ac1a5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac111720 con 0x7f79ac1014c0 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1115a0 con 0x7f79ac0ffad0 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.996+0000 7f79b1e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1118a0 con 0x7f79ac1066e0 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44956/0 (socket says 172.21.15.46:44956) 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79ab400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac1014c0 0x7f79ac1a5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 -- 172.21.15.46:0/3234127352 learned_addr learned my addr 172.21.15.46:0/3234127352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79aaa00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0ffad0 0x7f79ac100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 -- 172.21.15.46:0/3234127352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0ffad0 msgr2=0x7f79ac100a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0ffad0 0x7f79ac100a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 -- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac1014c0 msgr2=0x7f79ac1a5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac1014c0 0x7f79ac1a5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 -- 172.21.15.46:0/3234127352 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ac1a6530 con 0x7f79ac1066e0 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.997+0000 7f79a2000640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f799400e9f0 tx=0x7f799400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.998+0000 7f79b0a00640 1 -- 172.21.15.46:0/3234127352 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799400ccc0 con 0x7f79ac1066e0 2024-03-08T23:12:01.003 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.998+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79ac1a6820 con 0x7f79ac1066e0 2024-03-08T23:12:01.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.998+0000 7f79b0a00640 1 -- 172.21.15.46:0/3234127352 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7994010910 con 0x7f79ac1066e0 2024-03-08T23:12:01.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.998+0000 7f79b0a00640 1 -- 172.21.15.46:0/3234127352 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994018920 con 0x7f79ac1066e0 2024-03-08T23:12:01.004 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:00.998+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79ac1a6dc0 con 0x7f79ac1066e0 2024-03-08T23:12:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.001+0000 7f79b4a75740 1 Processor -- start 2024-03-08T23:12:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.001+0000 7f79b4a75740 1 -- start start 2024-03-08T23:12:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.001+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.001+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79b4a75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fbaa8dd0 con 0x55e1fbb4d850 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79b4a75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fba51c70 con 0x55e1fbb44c70 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79b4a75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fbaa4e70 con 0x55e1fbb44350 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:44972/0 (socket says 172.21.15.46:44972) 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 -- 172.21.15.46:0/141422705 learned_addr learned my addr 172.21.15.46:0/141422705 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79aaa00640 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 -- 172.21.15.46:0/141422705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 msgr2=0x55e1fbb4d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 -- 172.21.15.46:0/141422705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 msgr2=0x55e1fbb4fc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:12:01.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4fc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.002+0000 7f79ab400640 1 -- 172.21.15.46:0/141422705 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1fbb0b2b0 con 0x55e1fbb44350 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.003+0000 7f79ab400640 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c20 tx=0x7f79a4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97f387c20adfeb7c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.003+0000 7f79b0a00640 1 -- 172.21.15.46:0/3234127352 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7994010ac0 con 0x7f79ac1066e0 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.003+0000 7f799be00640 1 -- 172.21.15.46:0/141422705 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a401e070 con 0x55e1fbb44350 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.003+0000 7f799be00640 1 -- 172.21.15.46:0/141422705 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a4019b10 con 0x55e1fbb44350 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.003+0000 7f799be00640 1 -- 172.21.15.46:0/141422705 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a400e9b0 con 0x55e1fbb44350 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- 172.21.15.46:0/141422705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 msgr2=0x55e1fbb44730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c20 tx=0x7f79a4011810 comp rx=0 tx=0).stop 2024-03-08T23:12:01.009 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- 172.21.15.46:0/141422705 shutdown_connections 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 --2- 172.21.15.46:0/141422705 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb44730 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- 172.21.15.46:0/141422705 >> 172.21.15.46:0/141422705 conn(0x55e1fbb3ea80 msgr2=0x55e1fbb3ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- 172.21.15.46:0/141422705 shutdown_connections 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- 172.21.15.46:0/141422705 wait complete. 2024-03-08T23:12:01.010 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b0a00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 0x7f79900a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.004+0000 7f79b4a75740 1 -- start start 2024-03-08T23:12:01.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b0a00640 1 -- 172.21.15.46:0/3234127352 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f79940cf430 con 0x7f79ac1066e0 2024-03-08T23:12:01.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 Processor -- start 2024-03-08T23:12:01.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79ab400640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 0x7f79900a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb4d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fbb50640 con 0x55e1fbb4d850 2024-03-08T23:12:01.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fba650c0 con 0x55e1fbb44c70 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1fbb507c0 con 0x55e1fbb44350 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.005+0000 7f79b4a75740 10 client.0 _ll_register_callbacks cb 0x55e1fba9fec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37560/0 (socket says 172.21.15.46:37560) 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 -- 172.21.15.46:0/1668101386 learned_addr learned my addr 172.21.15.46:0/1668101386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb4d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79ab400640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 -- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 msgr2=0x55e1fbb4d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb4d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 -- 172.21.15.46:0/1668101386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 msgr2=0x55e1fbb4ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.006+0000 7f79a2000640 1 -- 172.21.15.46:0/1668101386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1fb7bd3f0 con 0x55e1fbb44c70 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f79a2000640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f79940cce20 tx=0x7f799409b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfcfc2ff62da8202 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f79ab400640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 0x7f79900a4d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c20 tx=0x7f79a400e530 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f7998c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79940a3ed0 con 0x55e1fbb44c70 2024-03-08T23:12:01.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f7994097e50 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a4019b10 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f799409ac30 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f79b4a75740 10 client.26462 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.007+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e1fb8877d0 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a4019b10 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f7994018ac0 con 0x55e1fbb44c70 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 10 client.26462 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 10 client.26462 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 20 client.26462 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbb5d940 0x55e1fbb5fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 4 client.26462 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.008+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1fba7d4f0 con 0x55e1fbb5d940 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.009+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbb5d940 0x55e1fbb5fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.009+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbb5d940 0x55e1fbb5fd30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e70ffb6ff84cbd86 server_cookie=43afaf23f769586f in_seq=0 out_seq=0 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.009+0000 7f7998c00640 10 client.26462 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:01.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.010+0000 7f7998c00640 1 -- 172.21.15.46:0/1668101386 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e1fba7d4f0 con 0x55e1fbb5d940 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.010+0000 7f7998c00640 10 client.26462 handle_command_reply: tid=0 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.010+0000 7f79b4a75740 1 client.26462 shutdown 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.010+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 msgr2=0x55e1fbb4a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.010+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f79940cce20 tx=0x7f799409b410 comp rx=0 tx=0).stop 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 shutdown_connections 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbb5d940 0x55e1fbb5fd30 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb4d850 0x55e1fbb4ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb4a640 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f7998c00640 0 client.26462 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1668101386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb4d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 >> 172.21.15.46:0/1668101386 conn(0x55e1fbb3ea80 msgr2=0x55e1fbb407d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 shutdown_connections 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 1 -- 172.21.15.46:0/1668101386 wait complete. 2024-03-08T23:12:01.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.011+0000 7f79b4a75740 20 client.26462 trim_cache size 0 max 16384 2024-03-08T23:12:01.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.037+0000 7f79b4a75740 1 Processor -- start 2024-03-08T23:12:01.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.037+0000 7f79b4a75740 1 -- start start 2024-03-08T23:12:01.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb3d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 0x55e1fbb54a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1b2d20 con 0x55e1fbb44350 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1b0860 con 0x55e1fbb44c70 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79b4a75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1ae900 con 0x55e1fbb4d850 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79aaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79aaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33258/0 (socket says 172.21.15.46:33258) 2024-03-08T23:12:01.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79ab400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 0x55e1fbb54a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79aaa00640 1 -- 172.21.15.46:0/3102007656 learned_addr learned my addr 172.21.15.46:0/3102007656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79a2000640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb3d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79ab400640 1 -- 172.21.15.46:0/3102007656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 msgr2=0x55e1fbb3d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79ab400640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb3d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79ab400640 1 -- 172.21.15.46:0/3102007656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 msgr2=0x55e1fbb497a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.038+0000 7f79ab400640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f79ab400640 1 -- 172.21.15.46:0/3102007656 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1fbb4b0d0 con 0x55e1fbb4d850 2024-03-08T23:12:01.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f79aaa00640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f79a2000640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb3d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f79ab400640 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 0x55e1fbb54a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79a400e4d0 tx=0x7f79a402a070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b5a9ad7c56da826 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f799b400640 1 -- 172.21.15.46:0/3102007656 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a401e050 con 0x55e1fbb4d850 2024-03-08T23:12:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f799b400640 1 -- 172.21.15.46:0/3102007656 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a401f040 con 0x55e1fbb4d850 2024-03-08T23:12:01.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.039+0000 7f799b400640 1 -- 172.21.15.46:0/3102007656 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4004e30 con 0x55e1fbb4d850 2024-03-08T23:12:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 -- 172.21.15.46:0/3102007656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 msgr2=0x55e1fbb54a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 0x55e1fbb54a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f79a400e4d0 tx=0x7f79a402a070 comp rx=0 tx=0).stop 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 -- 172.21.15.46:0/3102007656 shutdown_connections 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb4d850 0x55e1fbb54a00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb44c70 0x55e1fbb3d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 --2- 172.21.15.46:0/3102007656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44350 0x55e1fbb497a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.040+0000 7f79b4a75740 1 -- 172.21.15.46:0/3102007656 >> 172.21.15.46:0/3102007656 conn(0x55e1fbb3ea80 msgr2=0x55e1fbb3d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.041+0000 7f79b4a75740 1 -- 172.21.15.46:0/3102007656 shutdown_connections 2024-03-08T23:12:01.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.041+0000 7f79b4a75740 1 -- 172.21.15.46:0/3102007656 wait complete. 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.041+0000 7f79b4a75740 1 -- start start 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.041+0000 7f79b4a75740 1 Processor -- start 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 0x55e1fbb539e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1aef00 con 0x55e1fbb44c70 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1b0200 con 0x55e1fbb4d850 2024-03-08T23:12:01.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79ac1af540 con 0x55e1fbb44350 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79b4a75740 10 client.0 _ll_register_callbacks cb 0x7f79a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79a2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 0x55e1fbb539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79ab400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79ab400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37580/0 (socket says 172.21.15.46:37580) 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79ab400640 1 -- 172.21.15.46:0/1546653888 learned_addr learned my addr 172.21.15.46:0/1546653888 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.042+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 -- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 msgr2=0x55e1fbb534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 -- 172.21.15.46:0/1546653888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 msgr2=0x55e1fbb53f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 -- 172.21.15.46:0/1546653888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1fbbb3ee0 con 0x55e1fbb44c70 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79ab400640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79a2000640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 0x55e1fbb539e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f79940cc8d0 tx=0x7f799400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359dc5db3bdb3e3f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f7998c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7994097530 con 0x55e1fbb44c70 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f79940976d0 con 0x55e1fbb44c70 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79a401f040 con 0x55e1fbb44c70 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7994010430 con 0x55e1fbb44c70 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79b4a75740 10 client.16323 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.043+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e1fbbb51c0 con 0x55e1fbb44c70 2024-03-08T23:12:01.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79a401f040 con 0x55e1fbb44c70 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f79940105d0 con 0x55e1fbb44c70 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 10 client.16323 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 10 client.16323 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 20 client.16323 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbbb5700 0x55e1fbbb7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 4 client.16323 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e1fbab6310 con 0x55e1fbbb5700 2024-03-08T23:12:01.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.044+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbbb5700 0x55e1fbbb7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79aaa00640 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbbb5700 0x55e1fbbb7b20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6ed79cd95f985d50 server_cookie=50e7e3238c442c59 in_seq=0 out_seq=0 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f7998c00640 10 client.16323 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f7998c00640 1 -- 172.21.15.46:0/1546653888 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e1fbab6310 con 0x55e1fbbb5700 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f7998c00640 10 client.16323 handle_command_reply: tid=0 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 client.16323 shutdown 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 msgr2=0x55e1fbb539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 0x55e1fbb539e0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f79940cc8d0 tx=0x7f799400cf60 comp rx=0 tx=0).stop 2024-03-08T23:12:01.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 shutdown_connections 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55e1fbbb5700 0x55e1fbbb7b20 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e1fbb4d850 0x55e1fbb53f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e1fbb44c70 0x55e1fbb539e0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 --2- 172.21.15.46:0/1546653888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e1fbb44350 0x55e1fbb534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f7998c00640 0 client.16323 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.045+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 >> 172.21.15.46:0/1546653888 conn(0x55e1fbb3ea80 msgr2=0x55e1fbb40740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 shutdown_connections 2024-03-08T23:12:01.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b4a75740 1 -- 172.21.15.46:0/1546653888 wait complete. 2024-03-08T23:12:01.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b4a75740 20 client.16323 trim_cache size 0 max 16384 2024-03-08T23:12:01.058 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 msgr2=0x7f79900a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 0x7f79900a4d70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f79a4009c20 tx=0x7f79a400e530 comp rx=0 tx=0).stop 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 msgr2=0x7f79ac100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.046+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f799400e9f0 tx=0x7f799400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 shutdown_connections 2024-03-08T23:12:01.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f79900a28b0 0x7f79900a4d70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f79ac1014c0 0x7f79ac1a5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f79ac1066e0 0x7f79ac100f80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 --2- 172.21.15.46:0/3234127352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f79ac0ffad0 0x7f79ac100a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 >> 172.21.15.46:0/3234127352 conn(0x7f79ac076820 msgr2=0x7f79ac0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 shutdown_connections 2024-03-08T23:12:01.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.047+0000 7f79b1e00640 1 -- 172.21.15.46:0/3234127352 wait complete. 2024-03-08T23:12:01.069 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:01.070 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:12:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 -- 172.21.15.46:0/2847289503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 msgr2=0x7f640c114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 --2- 172.21.15.46:0/2847289503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c114520 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6404009cd0 tx=0x7f640402c0f0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 -- 172.21.15.46:0/2847289503 shutdown_connections 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 --2- 172.21.15.46:0/2847289503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c114520 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 --2- 172.21.15.46:0/2847289503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 --2- 172.21.15.46:0/2847289503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.260+0000 7f6412a00640 1 -- 172.21.15.46:0/2847289503 >> 172.21.15.46:0/2847289503 conn(0x7f640c0fef60 msgr2=0x7f640c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.261+0000 7f6412a00640 1 -- 172.21.15.46:0/2847289503 shutdown_connections 2024-03-08T23:12:01.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.261+0000 7f6412a00640 1 -- 172.21.15.46:0/2847289503 wait complete. 2024-03-08T23:12:01.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.261+0000 7f6412a00640 1 Processor -- start 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 -- start start 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c19e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c1a2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c117c90 con 0x7f640c076140 2024-03-08T23:12:01.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c117b10 con 0x7f640c112130 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6412a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c117e10 con 0x7f640c075820 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6403e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6403e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33288/0 (socket says 172.21.15.46:33288) 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f6403e00640 1 -- 172.21.15.46:0/16350454 learned_addr learned my addr 172.21.15.46:0/16350454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.262+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c19e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f6410a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c1a2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 -- 172.21.15.46:0/16350454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 msgr2=0x7f640c1a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c1a2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 -- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 msgr2=0x7f640c19e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 -- 172.21.15.46:0/16350454 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c1a31a0 con 0x7f640c075820 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f6403e00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f6410a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c1a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.263+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c19e1b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00e9f0 tx=0x7f63fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.264+0000 7f6409600640 1 -- 172.21.15.46:0/16350454 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc00ccc0 con 0x7f640c075820 2024-03-08T23:12:01.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.264+0000 7f6409600640 1 -- 172.21.15.46:0/16350454 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f63fc005020 con 0x7f640c075820 2024-03-08T23:12:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.264+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f640c1a3490 con 0x7f640c075820 2024-03-08T23:12:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.264+0000 7f6409600640 1 -- 172.21.15.46:0/16350454 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc0049e0 con 0x7f640c075820 2024-03-08T23:12:01.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.264+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640c1aad20 con 0x7f640c075820 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.267+0000 7f6409600640 1 -- 172.21.15.46:0/16350454 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f63fc005560 con 0x7f640c075820 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.267+0000 7f64157d1740 1 Processor -- start 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.267+0000 7f64157d1740 1 -- start start 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 0x55ff70b52310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 0x55ff70b54c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70aaddd0 con 0x55ff70b49350 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70a56c70 con 0x55ff70b52850 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f64157d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70aa9e70 con 0x55ff70b49c70 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6410a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f640be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 0x55ff70b52310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6410a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33296/0 (socket says 172.21.15.46:33296) 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6410a00640 1 -- 172.21.15.46:0/3572728901 learned_addr learned my addr 172.21.15.46:0/3572728901 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6403e00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 0x55ff70b54c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6410a00640 1 -- 172.21.15.46:0/3572728901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 msgr2=0x55ff70b52310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.268+0000 7f6410a00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 0x55ff70b52310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6410a00640 1 -- 172.21.15.46:0/3572728901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 msgr2=0x55ff70b54c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6410a00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 0x55ff70b54c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6410a00640 1 -- 172.21.15.46:0/3572728901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff70b102b0 con 0x55ff70b49350 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f640be00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 0x55ff70b52310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6403e00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 0x55ff70b54c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6409600640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 0x7f63f00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6410a00640 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f640c06a7a0 tx=0x7f6404005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bc5aa138d2e85b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6409600640 1 -- 172.21.15.46:0/16350454 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f63fc0cf170 con 0x7f640c075820 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 0x7f63f00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6401200640 1 -- 172.21.15.46:0/3572728901 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404043070 con 0x55ff70b49350 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6401200640 1 -- 172.21.15.46:0/3572728901 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f640402cca0 con 0x55ff70b49350 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.269+0000 7f6401200640 1 -- 172.21.15.46:0/3572728901 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404039040 con 0x55ff70b49350 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 -- 172.21.15.46:0/3572728901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 msgr2=0x55ff70b49730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f640c06a7a0 tx=0x7f6404005fa0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f640be00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 0x7f63f00a4eb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55ff70b48e20 tx=0x7f63fc097040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 -- 172.21.15.46:0/3572728901 shutdown_connections 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b52850 0x55ff70b54c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49c70 0x55ff70b52310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 --2- 172.21.15.46:0/3572728901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b49730 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 -- 172.21.15.46:0/3572728901 >> 172.21.15.46:0/3572728901 conn(0x55ff70b43a80 msgr2=0x55ff70b43e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 -- 172.21.15.46:0/3572728901 shutdown_connections 2024-03-08T23:12:01.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.270+0000 7f64157d1740 1 -- 172.21.15.46:0/3572728901 wait complete. 2024-03-08T23:12:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.271+0000 7f64157d1740 1 -- start start 2024-03-08T23:12:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.271+0000 7f64157d1740 1 Processor -- start 2024-03-08T23:12:01.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b52200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b52850 0x55ff70b4fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70b555e0 con 0x55ff70b49350 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70a6a0c0 con 0x55ff70b49c70 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff70b55760 con 0x55ff70b52850 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f64157d1740 10 client.0 _ll_register_callbacks cb 0x55ff70aa4ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37624/0 (socket says 172.21.15.46:37624) 2024-03-08T23:12:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6410a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b52200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 -- 172.21.15.46:0/2169138267 learned_addr learned my addr 172.21.15.46:0/2169138267 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 -- 172.21.15.46:0/2169138267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b52850 msgr2=0x55ff70b4fb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:12:01.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b52850 0x55ff70b4fb30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 -- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 msgr2=0x55ff70b52200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b52200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6403e00640 1 -- 172.21.15.46:0/2169138267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff707c23f0 con 0x55ff70b49c70 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.272+0000 7f6410a00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b52200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.273+0000 7f6403e00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f63f4012d20 tx=0x7f63f400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88b2f11b951d1c36 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.273+0000 7f63f9600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.273+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f40120d0 con 0x55ff70b49c70 2024-03-08T23:12:01.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.273+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f63f4004030 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.273+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f640402cca0 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.276+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63f4012270 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.276+0000 7f64157d1740 10 client.26480 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.276+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ff7088c7d0 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f640402cca0 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f63f401c580 con 0x55ff70b49c70 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 10 client.26480 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 10 client.26480 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:12:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 20 client.26480 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70b62960 0x55ff70b64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 4 client.26480 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.277+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff70a824f0 con 0x55ff70b62960 2024-03-08T23:12:01.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.278+0000 7f6410a00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70b62960 0x55ff70b64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.278+0000 7f6410a00640 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70b62960 0x55ff70b64db0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=22a51cc0817177fb server_cookie=c1fd52dd34a4625b in_seq=0 out_seq=0 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.278+0000 7f63f9600640 10 client.26480 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.279+0000 7f63f9600640 1 -- 172.21.15.46:0/2169138267 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ff70a824f0 con 0x55ff70b62960 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.279+0000 7f63f9600640 10 client.26480 handle_command_reply: tid=0 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.279+0000 7f64157d1740 1 client.26480 shutdown 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 msgr2=0x55ff70b4f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f63f4012d20 tx=0x7f63f400f790 comp rx=0 tx=0).stop 2024-03-08T23:12:01.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 shutdown_connections 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70b62960 0x55ff70b64db0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b52850 0x55ff70b4fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b4f590 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f63f9600640 0 client.26480 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 --2- 172.21.15.46:0/2169138267 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b49350 0x55ff70b52200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 >> 172.21.15.46:0/2169138267 conn(0x55ff70b43a80 msgr2=0x55ff70b457d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 shutdown_connections 2024-03-08T23:12:01.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 1 -- 172.21.15.46:0/2169138267 wait complete. 2024-03-08T23:12:01.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.280+0000 7f64157d1740 20 client.26480 trim_cache size 0 max 16384 2024-03-08T23:12:01.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.306+0000 7f64157d1740 1 Processor -- start 2024-03-08T23:12:01.310 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 -- start start 2024-03-08T23:12:01.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b52200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b4c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b6ff0 con 0x55ff70b52850 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b4b30 con 0x55ff70b49c70 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f64157d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b2bd0 con 0x55ff70b49350 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f6410a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b52200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f6410a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45054/0 (socket says 172.21.15.46:45054) 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.307+0000 7f6410a00640 1 -- 172.21.15.46:0/3839767729 learned_addr learned my addr 172.21.15.46:0/3839767729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b4c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 -- 172.21.15.46:0/3839767729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 msgr2=0x55ff70b52200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b52200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 -- 172.21.15.46:0/3839767729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 msgr2=0x55ff70b4c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b4c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 -- 172.21.15.46:0/3839767729 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff70b49aa0 con 0x55ff70b49350 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.308+0000 7f6410a00640 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6404005e30 tx=0x7f6404005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c66c19b0ac80fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.309+0000 7f63fbe00640 1 -- 172.21.15.46:0/3839767729 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404039040 con 0x55ff70b49350 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.309+0000 7f63fbe00640 1 -- 172.21.15.46:0/3839767729 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f640402ce50 con 0x55ff70b49350 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.309+0000 7f63fbe00640 1 -- 172.21.15.46:0/3839767729 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6404043040 con 0x55ff70b49350 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.309+0000 7f64157d1740 1 -- 172.21.15.46:0/3839767729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 msgr2=0x55ff70b51cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.309+0000 7f64157d1740 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f6404005e30 tx=0x7f6404005d80 comp rx=0 tx=0).stop 2024-03-08T23:12:01.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 -- 172.21.15.46:0/3839767729 shutdown_connections 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b4c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b52200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 --2- 172.21.15.46:0/3839767729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b51cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 -- 172.21.15.46:0/3839767729 >> 172.21.15.46:0/3839767729 conn(0x55ff70b43a80 msgr2=0x55ff70b4de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 -- 172.21.15.46:0/3839767729 shutdown_connections 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 -- 172.21.15.46:0/3839767729 wait complete. 2024-03-08T23:12:01.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 -- start start 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.310+0000 7f64157d1740 1 Processor -- start 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b58660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b31d0 con 0x55ff70b52850 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b44d0 con 0x55ff70b49c70 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f640c1b3810 con 0x55ff70b49350 2024-03-08T23:12:01.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f64157d1740 10 client.0 _ll_register_callbacks cb 0x7f63f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f6403e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37642/0 (socket says 172.21.15.46:37642) 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f6410a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b58660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f6403e00640 1 -- 172.21.15.46:0/3631515541 learned_addr learned my addr 172.21.15.46:0/3631515541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.311+0000 7f6410a00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 -- 172.21.15.46:0/3631515541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 msgr2=0x55ff70b58660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b58660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 -- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 msgr2=0x55ff70b590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 -- 172.21.15.46:0/3631515541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ff70bb9140 con 0x55ff70b49c70 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f6403e00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55ff70b50b60 tx=0x7f63f401dd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8862bc48f2e61f8b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f63f9600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f402d020 con 0x55ff70b49c70 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f640402ce50 con 0x55ff70b49c70 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.312+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f63f40052d0 con 0x55ff70b49c70 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63f40120d0 con 0x55ff70b49c70 2024-03-08T23:12:01.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 10 client.26486 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55ff70bba440 con 0x55ff70b49c70 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f640402ce50 con 0x55ff70b49c70 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f63f4003c90 con 0x55ff70b49c70 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 10 client.26486 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 10 client.26486 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 20 client.26486 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70bbac20 0x55ff70bbd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 4 client.26486 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:01.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.313+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ff70abb310 con 0x55ff70bbac20 2024-03-08T23:12:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.314+0000 7f6410a00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70bbac20 0x55ff70bbd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.314+0000 7f6410a00640 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70bbac20 0x55ff70bbd040 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36660226ec974f5f server_cookie=358d75dcde34e31d in_seq=0 out_seq=0 2024-03-08T23:12:01.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.314+0000 7f63f9600640 10 client.26486 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f63f9600640 1 -- 172.21.15.46:0/3631515541 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ff70abb310 con 0x55ff70bbac20 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f63f9600640 10 client.26486 handle_command_reply: tid=0 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 client.26486 shutdown 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 msgr2=0x55ff70b58ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55ff70b50b60 tx=0x7f63f401dd50 comp rx=0 tx=0).stop 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 shutdown_connections 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ff70b52850 0x55ff70b590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ff70b49c70 0x55ff70b58ba0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ff70bbac20 0x55ff70bbd040 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 --2- 172.21.15.46:0/3631515541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ff70b49350 0x55ff70b58660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 >> 172.21.15.46:0/3631515541 conn(0x55ff70b43a80 msgr2=0x55ff70b42200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f63f9600640 0 client.26486 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 shutdown_connections 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 1 -- 172.21.15.46:0/3631515541 wait complete. 2024-03-08T23:12:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.315+0000 7f64157d1740 20 client.26486 trim_cache size 0 max 16384 2024-03-08T23:12:01.327 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:01.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 msgr2=0x7f63f00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 0x7f63f00a4eb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x55ff70b48e20 tx=0x7f63fc097040 comp rx=0 tx=0).stop 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 msgr2=0x7f640c19e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c19e1b0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63fc00e9f0 tx=0x7f63fc00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 shutdown_connections 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f63f00a29f0 0x7f63f00a4eb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f640c112130 0x7f640c1a2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f640c076140 0x7f640c19e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 --2- 172.21.15.46:0/16350454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f640c075820 0x7f640c19e1b0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 >> 172.21.15.46:0/16350454 conn(0x7f640c0fef60 msgr2=0x7f640c113850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 shutdown_connections 2024-03-08T23:12:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.316+0000 7f6412a00640 1 -- 172.21.15.46:0/16350454 wait complete. 2024-03-08T23:12:01.335 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:01 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:12:01.336 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:01 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/910886030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:01.337 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:01.338 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:12:01.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 -- 172.21.15.46:0/3221320088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9641034b0 msgr2=0x7fc964103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 --2- 172.21.15.46:0/3221320088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9641034b0 0x7fc964103930 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc94c009900 tx=0x7fc94c02bc40 comp rx=0 tx=0).stop 2024-03-08T23:12:01.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 -- 172.21.15.46:0/3221320088 shutdown_connections 2024-03-08T23:12:01.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 --2- 172.21.15.46:0/3221320088 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964103e70 0x7fc964110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 --2- 172.21.15.46:0/3221320088 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc9641034b0 0x7fc964103930 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 --2- 172.21.15.46:0/3221320088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc964109570 0x7fc964109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 -- 172.21.15.46:0/3221320088 >> 172.21.15.46:0/3221320088 conn(0x7fc9640fef60 msgr2=0x7fc964101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 -- 172.21.15.46:0/3221320088 shutdown_connections 2024-03-08T23:12:01.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.545+0000 7fc968e00640 1 -- 172.21.15.46:0/3221320088 wait complete. 2024-03-08T23:12:01.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.546+0000 7fc968e00640 1 Processor -- start 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 -- start start 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9641034b0 0x7fc96419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 0x7fc96419ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 0x7fc964199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc964113880 con 0x7fc964103e70 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc964113700 con 0x7fc964109570 2024-03-08T23:12:01.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc968e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc964113a00 con 0x7fc9641034b0 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc95be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 0x7fc96419ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc95be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 0x7fc96419ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33360/0 (socket says 172.21.15.46:33360) 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc95be00640 1 -- 172.21.15.46:0/3548701614 learned_addr learned my addr 172.21.15.46:0/3548701614 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc962800640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 0x7fc964199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.547+0000 7fc961e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9641034b0 0x7fc96419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 -- 172.21.15.46:0/3548701614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9641034b0 msgr2=0x7fc96419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9641034b0 0x7fc96419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 -- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 msgr2=0x7fc96419ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 0x7fc96419ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 -- 172.21.15.46:0/3548701614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96419a3e0 con 0x7fc964109570 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc962800640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 0x7fc964199b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009fd0 tx=0x7fc95c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.548+0000 7fc95aa00640 1 -- 172.21.15.46:0/3548701614 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c00ce20 con 0x7fc964109570 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.549+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96419a6d0 con 0x7fc964109570 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.549+0000 7fc95aa00640 1 -- 172.21.15.46:0/3548701614 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc95c005020 con 0x7fc964109570 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.549+0000 7fc95aa00640 1 -- 172.21.15.46:0/3548701614 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c010720 con 0x7fc964109570 2024-03-08T23:12:01.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.549+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9641a69e0 con 0x7fc964109570 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.552+0000 7fc95aa00640 1 -- 172.21.15.46:0/3548701614 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc95c005540 con 0x7fc964109570 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.552+0000 7fc96b9f9740 1 Processor -- start 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.552+0000 7fc96b9f9740 1 -- start start 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588118ffdd0 con 0x55881199bc70 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588118a8c70 con 0x55881199b350 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588118fbe70 con 0x5588119a4850 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc95be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc95be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37678/0 (socket says 172.21.15.46:37678) 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc95be00640 1 -- 172.21.15.46:0/2636162889 learned_addr learned my addr 172.21.15.46:0/2636162889 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33376/0 (socket says 172.21.15.46:33376) 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.553+0000 7fc95be00640 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 -- 172.21.15.46:0/2636162889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 msgr2=0x5588119a6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119a6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 -- 172.21.15.46:0/2636162889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 msgr2=0x55881199b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 -- 172.21.15.46:0/2636162889 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588119622b0 con 0x55881199bc70 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc95be00640 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc95aa00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 0x7fc9480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc961e00640 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc95400bbf0 tx=0x7fc9540107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e5f594f725fc39 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc95aa00640 1 -- 172.21.15.46:0/3548701614 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc95c0cf910 con 0x7fc964109570 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc95be00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 0x7fc9480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc953400640 1 -- 172.21.15.46:0/2636162889 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954004030 con 0x55881199bc70 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc953400640 1 -- 172.21.15.46:0/2636162889 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9540041d0 con 0x55881199bc70 2024-03-08T23:12:01.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.554+0000 7fc953400640 1 -- 172.21.15.46:0/2636162889 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9540116a0 con 0x55881199bc70 2024-03-08T23:12:01.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 -- 172.21.15.46:0/2636162889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 msgr2=0x5588119a4310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc95400bbf0 tx=0x7fc9540107a0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc95be00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 0x7fc9480a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x558811996c10 tx=0x7fc94c005b50 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 -- 172.21.15.46:0/2636162889 shutdown_connections 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119a6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a4310 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/2636162889 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x55881199b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 -- 172.21.15.46:0/2636162889 >> 172.21.15.46:0/2636162889 conn(0x558811995a80 msgr2=0x558811995e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.555+0000 7fc96b9f9740 1 -- 172.21.15.46:0/2636162889 shutdown_connections 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.556+0000 7fc96b9f9740 1 -- 172.21.15.46:0/2636162889 wait complete. 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.556+0000 7fc96b9f9740 1 -- start start 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.556+0000 7fc96b9f9740 1 Processor -- start 2024-03-08T23:12:01.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 0x5588119a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 0x5588119a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588119a7640 con 0x55881199b350 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588118bc0c0 con 0x5588119a4850 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588119a77c0 con 0x55881199bc70 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc96b9f9740 10 client.0 _ll_register_callbacks cb 0x5588118f6ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 0x5588119a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33390/0 (socket says 172.21.15.46:33390) 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc961e00640 1 -- 172.21.15.46:0/3366830630 learned_addr learned my addr 172.21.15.46:0/3366830630 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc95be00640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 0x5588119a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 -- 172.21.15.46:0/3366830630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 msgr2=0x5588119a14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 0x5588119a14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 -- 172.21.15.46:0/3366830630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 msgr2=0x5588119a4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc962800640 1 -- 172.21.15.46:0/3366830630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588116143f0 con 0x5588119a4850 2024-03-08T23:12:01.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc95be00640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 0x5588119a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.557+0000 7fc961e00640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc962800640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 0x5588119a1a10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0ccca0 tx=0x7fc95c09bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc9fbd6b84293d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc947e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c0a3ec0 con 0x5588119a4850 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc95c0a4060 con 0x5588119a4850 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9540041d0 con 0x5588119a4850 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc95c09b000 con 0x5588119a4850 2024-03-08T23:12:01.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc96b9f9740 10 client.26498 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5588116de7d0 con 0x5588119a4850 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.558+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9540041d0 con 0x5588119a4850 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc95c09ac70 con 0x5588119a4850 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 10 client.26498 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 10 client.26498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 20 client.26498 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588119b48a0 0x5588119b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 4 client.26498 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588118d44f0 con 0x5588119b48a0 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.559+0000 7fc961e00640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588119b48a0 0x5588119b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc961e00640 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588119b48a0 0x5588119b6cf0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76ad2d596c45aa59 server_cookie=fcac575a3a8864fa in_seq=0 out_seq=0 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc947e00640 10 client.26498 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc947e00640 1 -- 172.21.15.46:0/3366830630 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588118d44f0 con 0x5588119b48a0 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc947e00640 10 client.26498 handle_command_reply: tid=0 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc96b9f9740 1 client.26498 shutdown 2024-03-08T23:12:01.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 msgr2=0x5588119a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.560+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 0x5588119a1a10 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc95c0ccca0 tx=0x7fc95c09bbb0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 shutdown_connections 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5588119b48a0 0x5588119b6cf0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5588119a4850 0x5588119a1a10 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55881199bc70 0x5588119a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/3366830630 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 >> 172.21.15.46:0/3366830630 conn(0x558811995a80 msgr2=0x5588119977d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc947e00640 0 client.26498 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:01.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 shutdown_connections 2024-03-08T23:12:01.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 1 -- 172.21.15.46:0/3366830630 wait complete. 2024-03-08T23:12:01.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.561+0000 7fc96b9f9740 20 client.26498 trim_cache size 0 max 16384 2024-03-08T23:12:01.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:01 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:12:01.575 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:01 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/910886030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:01.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.575+0000 7fc96b9f9740 1 Processor -- start 2024-03-08T23:12:01.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.575+0000 7fc96b9f9740 1 -- start start 2024-03-08T23:12:01.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 0x5588119a4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x55881199e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641b2bd0 con 0x55881199b350 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641b0710 con 0x55881199bc70 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641ae7b0 con 0x5588119a4850 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc961e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33392/0 (socket says 172.21.15.46:33392) 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc961e00640 1 -- 172.21.15.46:0/4058728717 learned_addr learned my addr 172.21.15.46:0/4058728717 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.576+0000 7fc95be00640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 0x5588119a4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc962800640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x55881199e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 -- 172.21.15.46:0/4058728717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 msgr2=0x55881199e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x55881199e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 -- 172.21.15.46:0/4058728717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 msgr2=0x5588119a4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 0x5588119a4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 -- 172.21.15.46:0/4058728717 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588119952c0 con 0x55881199b350 2024-03-08T23:12:01.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc95be00640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 0x5588119a4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:01.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc961e00640 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc95400bb10 tx=0x7fc95400b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1004bb5b1112ed server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.577+0000 7fc952a00640 1 -- 172.21.15.46:0/4058728717 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9540035f0 con 0x55881199b350 2024-03-08T23:12:01.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc952a00640 1 -- 172.21.15.46:0/4058728717 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc954005040 con 0x55881199b350 2024-03-08T23:12:01.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc952a00640 1 -- 172.21.15.46:0/4058728717 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954028380 con 0x55881199b350 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 -- 172.21.15.46:0/4058728717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 msgr2=0x5588119a3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc95400bb10 tx=0x7fc95400b820 comp rx=0 tx=0).stop 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 -- 172.21.15.46:0/4058728717 shutdown_connections 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x55881199e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199bc70 0x5588119a4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/4058728717 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199b350 0x5588119a3c00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.578+0000 7fc96b9f9740 1 -- 172.21.15.46:0/4058728717 >> 172.21.15.46:0/4058728717 conn(0x558811995a80 msgr2=0x55881199fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.579+0000 7fc96b9f9740 1 -- 172.21.15.46:0/4058728717 shutdown_connections 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.579+0000 7fc96b9f9740 1 -- 172.21.15.46:0/4058728717 wait complete. 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.579+0000 7fc96b9f9740 1 -- start start 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.579+0000 7fc96b9f9740 1 Processor -- start 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119aa470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641aedb0 con 0x55881199bc70 2024-03-08T23:12:01.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641b00b0 con 0x55881199b350 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9641af3f0 con 0x5588119a4850 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc96b9f9740 10 client.0 _ll_register_callbacks cb 0x7fc95400b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.580+0000 7fc95be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37694/0 (socket says 172.21.15.46:37694) 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 -- 172.21.15.46:0/498554055 learned_addr learned my addr 172.21.15.46:0/498554055 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc962800640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119aa470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 -- 172.21.15.46:0/498554055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 msgr2=0x5588119aa470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119aa470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 -- 172.21.15.46:0/498554055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 msgr2=0x5588119a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 -- 172.21.15.46:0/498554055 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588119aa9b0 con 0x55881199b350 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc961e00640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc954003da0 tx=0x7fc954004fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=19a169ed01de89c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc947e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.581+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954036060 con 0x55881199b350 2024-03-08T23:12:01.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc954005040 con 0x55881199b350 2024-03-08T23:12:01.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc9540035f0 con 0x55881199b350 2024-03-08T23:12:01.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc954005600 con 0x55881199b350 2024-03-08T23:12:01.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 10 client.26504 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:01.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x558811a0bb60 con 0x55881199b350 2024-03-08T23:12:01.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc954005040 con 0x55881199b350 2024-03-08T23:12:01.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc954028ce0 con 0x55881199b350 2024-03-08T23:12:01.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 10 client.26504 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 10 client.26504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 20 client.26504 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x558811a0c140 0x558811a0e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 4 client.26504 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.582+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55881190d310 con 0x558811a0c140 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc95be00640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x558811a0c140 0x558811a0e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc95be00640 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x558811a0c140 0x558811a0e560 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38b796d0d73f48e4 server_cookie=6d56657ed0eb0c1c in_seq=0 out_seq=0 2024-03-08T23:12:01.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc947e00640 10 client.26504 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc947e00640 1 -- 172.21.15.46:0/498554055 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55881190d310 con 0x558811a0c140 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc947e00640 10 client.26504 handle_command_reply: tid=0 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.583+0000 7fc96b9f9740 1 client.26504 shutdown 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 msgr2=0x5588119abb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc954003da0 tx=0x7fc954004fd0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 shutdown_connections 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x558811a0c140 0x558811a0e560 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5588119a4850 0x5588119aa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55881199bc70 0x5588119a9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 --2- 172.21.15.46:0/498554055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55881199b350 0x5588119abb40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 >> 172.21.15.46:0/498554055 conn(0x558811995a80 msgr2=0x558811996eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc947e00640 0 client.26504 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 shutdown_connections 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 1 -- 172.21.15.46:0/498554055 wait complete. 2024-03-08T23:12:01.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.584+0000 7fc96b9f9740 20 client.26504 trim_cache size 0 max 16384 2024-03-08T23:12:01.596 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:01.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 msgr2=0x7fc9480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 0x7fc9480a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x558811996c10 tx=0x7fc94c005b50 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 msgr2=0x7fc964199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 0x7fc964199b20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fc95c009fd0 tx=0x7fc95c00ef10 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 shutdown_connections 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc9480a2970 0x7fc9480a4e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc964109570 0x7fc964199b20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc964103e70 0x7fc96419ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 --2- 172.21.15.46:0/3548701614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc9641034b0 0x7fc96419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:01.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 >> 172.21.15.46:0/3548701614 conn(0x7fc9640fef60 msgr2=0x7fc9641006f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:01.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 shutdown_connections 2024-03-08T23:12:01.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:01.585+0000 7fc968e00640 1 -- 172.21.15.46:0/3548701614 wait complete. 2024-03-08T23:12:01.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:01 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:12:01.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:01 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/910886030' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:02.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:02 smithi192 ceph-mon[36700]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-08T23:12:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:02 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:12:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:02 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:12:02.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:02 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:12:02.608 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:12:02.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:12:02.608 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:12:02.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:02 smithi062 ceph-mon[35761]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-08T23:12:02.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:02 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:12:02.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:02 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:12:02.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:02 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:12:02.700 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:02 smithi046 ceph-mon[31859]: pgmap v130: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-08T23:12:02.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:02 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:12:02.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:02 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:12:02.701 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:02 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:12:02.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 -- 172.21.15.46:0/2269588670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 msgr2=0x7f29dc103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 --2- 172.21.15.46:0/2269588670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc103930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f29c8009930 tx=0x7f29c802bc30 comp rx=0 tx=0).stop 2024-03-08T23:12:02.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 -- 172.21.15.46:0/2269588670 shutdown_connections 2024-03-08T23:12:02.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 --2- 172.21.15.46:0/2269588670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc103e70 0x7f29dc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 --2- 172.21.15.46:0/2269588670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc103930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 --2- 172.21.15.46:0/2269588670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc109570 0x7f29dc109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 -- 172.21.15.46:0/2269588670 >> 172.21.15.46:0/2269588670 conn(0x7f29dc0fef40 msgr2=0x7f29dc101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 -- 172.21.15.46:0/2269588670 shutdown_connections 2024-03-08T23:12:02.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.848+0000 7f29e2400640 1 -- 172.21.15.46:0/2269588670 wait complete. 2024-03-08T23:12:02.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.849+0000 7f29e2400640 1 Processor -- start 2024-03-08T23:12:02.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 -- start start 2024-03-08T23:12:02.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 0x7f29dc19ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc109570 0x7f29dc199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc113830 con 0x7f29dc109570 2024-03-08T23:12:02.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1136b0 con 0x7f29dc103e70 2024-03-08T23:12:02.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29e2400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1139b0 con 0x7f29dc1034b0 2024-03-08T23:12:02.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.850+0000 7f29db400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:45144/0 (socket says 172.21.15.46:45144) 2024-03-08T23:12:02.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 -- 172.21.15.46:0/1970143963 learned_addr learned my addr 172.21.15.46:0/1970143963 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:02.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29dbe00640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc109570 0x7f29dc199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29daa00640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 0x7f29dc19ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 -- 172.21.15.46:0/1970143963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 msgr2=0x7f29dc19ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 0x7f29dc19ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 -- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc109570 msgr2=0x7f29dc199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc109570 0x7f29dc199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 -- 172.21.15.46:0/1970143963 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29dc19a3d0 con 0x7f29dc1034b0 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29daa00640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 0x7f29dc19ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.851+0000 7f29db400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00e9f0 tx=0x7f29cc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.852+0000 7f29d3e00640 1 -- 172.21.15.46:0/1970143963 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc00cd80 con 0x7f29dc1034b0 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.852+0000 7f29d3e00640 1 -- 172.21.15.46:0/1970143963 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29cc010910 con 0x7f29dc1034b0 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.852+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29dc19a6c0 con 0x7f29dc1034b0 2024-03-08T23:12:02.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.852+0000 7f29d3e00640 1 -- 172.21.15.46:0/1970143963 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29cc018990 con 0x7f29dc1034b0 2024-03-08T23:12:02.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.852+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29dc1a69d0 con 0x7f29dc1034b0 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 Processor -- start 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 -- start start 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29d3e00640 1 -- 172.21.15.46:0/1970143963 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f29cc010ac0 con 0x7f29dc1034b0 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34262d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e342638ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e342591a50 con 0x55e34262df00 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e34253ab00 con 0x55e34262d5e0 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.855+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e34258daf0 con 0x55e342636ae0 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29db400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:33430/0 (socket says 172.21.15.46:33430) 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29daa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e342638ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29db400640 1 -- 172.21.15.46:0/3824699867 learned_addr learned my addr 172.21.15.46:0/3824699867 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34262d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 -- 172.21.15.46:0/3824699867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 msgr2=0x55e342638ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e342638ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 -- 172.21.15.46:0/3824699867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 msgr2=0x55e3426365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29dbe00640 1 -- 172.21.15.46:0/3824699867 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3425581e0 con 0x55e34262d5e0 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29daa00640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e342638ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:02.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.856+0000 7f29db400640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29dbe00640 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34262d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009f00 tx=0x7f29d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2885f67443ffcae server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29d3e00640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 0x7f29c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29d0c00640 1 -- 172.21.15.46:0/3824699867 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4004d20 con 0x55e34262d5e0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29d0c00640 1 -- 172.21.15.46:0/3824699867 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29d4004030 con 0x55e34262d5e0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29d0c00640 1 -- 172.21.15.46:0/3824699867 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d4005280 con 0x55e34262d5e0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29d3e00640 1 -- 172.21.15.46:0/1970143963 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f29cc0cf560 con 0x7f29dc1034b0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.857+0000 7f29db400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 0x7f29c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3824699867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 msgr2=0x55e34262d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34262d9c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f29d4009f00 tx=0x7f29d4011810 comp rx=0 tx=0).stop 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3824699867 shutdown_connections 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e342638ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e3426365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3824699867 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34262d9c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3824699867 >> 172.21.15.46:0/3824699867 conn(0x55e342627d10 msgr2=0x55e3426280f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3824699867 shutdown_connections 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29db400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 0x7f29c00a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00e9c0 tx=0x7f29cc0d2f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3824699867 wait complete. 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.858+0000 7f29e4ffe740 1 -- start start 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 Processor -- start 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342633ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342634530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3426398d0 con 0x55e342636ae0 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e34254df60 con 0x55e34262df00 2024-03-08T23:12:02.874 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e342639a50 con 0x55e34262d5e0 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.859+0000 7f29e4ffe740 10 client.0 _ll_register_callbacks cb 0x55e342539b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43264/0 (socket says 172.21.15.46:43264) 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 -- 172.21.15.46:0/454926018 learned_addr learned my addr 172.21.15.46:0/454926018 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342633ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29dbe00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342634530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 -- 172.21.15.46:0/454926018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 msgr2=0x55e342633ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342633ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.875 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 -- 172.21.15.46:0/454926018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 msgr2=0x55e342634530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342634530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 -- 172.21.15.46:0/454926018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e34236e7e0 con 0x55e34262df00 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29dbe00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342634530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29daa00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f29c8004ee0 tx=0x7f29c802c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1e4f2a8d45e07a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.860+0000 7f29c5600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8004050 con 0x55e34262df00 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f29c80041f0 con 0x55e34262df00 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29d4004030 con 0x55e34262df00 2024-03-08T23:12:02.876 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29c80043f0 con 0x55e34262df00 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29e4ffe740 10 client.26516 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e3423cef70 con 0x55e34262df00 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.861+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29d4004030 con 0x55e34262df00 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f29c8003ab0 con 0x55e34262df00 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 10 client.26516 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 10 client.26516 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 20 client.26516 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:02.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e342646e30 0x55e342649220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 4 client.26516 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e34259ef90 con 0x55e342646e30 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.862+0000 7f29dbe00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e342646e30 0x55e342649220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.863+0000 7f29dbe00640 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e342646e30 0x55e342649220 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fe432a91de3d446 server_cookie=2c9f3e1062cb1357 in_seq=0 out_seq=0 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.863+0000 7f29c5600640 10 client.26516 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.863+0000 7f29c5600640 1 -- 172.21.15.46:0/454926018 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e34259ef90 con 0x55e342646e30 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.863+0000 7f29c5600640 10 client.26516 handle_command_reply: tid=0 2024-03-08T23:12:02.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 client.26516 shutdown 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 msgr2=0x55e342633ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f29c8004ee0 tx=0x7f29c802c880 comp rx=0 tx=0).stop 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 shutdown_connections 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e342646e30 0x55e342649220 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342634530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342633ff0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/454926018 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342633ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 >> 172.21.15.46:0/454926018 conn(0x55e342627d10 msgr2=0x55e342629a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29c5600640 0 client.26516 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 shutdown_connections 2024-03-08T23:12:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 1 -- 172.21.15.46:0/454926018 wait complete. 2024-03-08T23:12:02.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.864+0000 7f29e4ffe740 20 client.26516 trim_cache size 0 max 16384 2024-03-08T23:12:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 Processor -- start 2024-03-08T23:12:02.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 -- start start 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342636120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342631270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1b2c30 con 0x55e342636ae0 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1b0770 con 0x55e34262df00 2024-03-08T23:12:02.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1ae810 con 0x55e34262d5e0 2024-03-08T23:12:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.881+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342631270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43270/0 (socket says 172.21.15.46:43270) 2024-03-08T23:12:02.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 -- 172.21.15.46:0/3972286974 learned_addr learned my addr 172.21.15.46:0/3972286974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29dbe00640 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342636120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 -- 172.21.15.46:0/3972286974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 msgr2=0x55e342636120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342636120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 -- 172.21.15.46:0/3972286974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 msgr2=0x55e342631270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342631270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 -- 172.21.15.46:0/3972286974 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e342266170 con 0x55e34262df00 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29daa00640 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f29c8003f00 tx=0x7f29c8001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=978f167f07e1e4b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.882+0000 7f29c7e00640 1 -- 172.21.15.46:0/3972286974 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8009780 con 0x55e34262df00 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29c7e00640 1 -- 172.21.15.46:0/3972286974 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29c8042070 con 0x55e34262df00 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29c7e00640 1 -- 172.21.15.46:0/3972286974 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c8004210 con 0x55e34262df00 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3972286974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 msgr2=0x55e342630d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f29c8003f00 tx=0x7f29c8001e70 comp rx=0 tx=0).stop 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3972286974 shutdown_connections 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e342636ae0 0x55e342631270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262df00 0x55e342630d30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/3972286974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e34262d5e0 0x55e342636120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3972286974 >> 172.21.15.46:0/3972286974 conn(0x55e342627d10 msgr2=0x55e342632380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3972286974 shutdown_connections 2024-03-08T23:12:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.883+0000 7f29e4ffe740 1 -- 172.21.15.46:0/3972286974 wait complete. 2024-03-08T23:12:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 -- start start 2024-03-08T23:12:02.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 Processor -- start 2024-03-08T23:12:02.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e34263cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e34263d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1aee10 con 0x55e34262df00 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1b0110 con 0x55e34262d5e0 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.884+0000 7f29e4ffe740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29dc1af450 con 0x55e342636ae0 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29e4ffe740 10 client.0 _ll_register_callbacks cb 0x7f29d4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43280/0 (socket says 172.21.15.46:43280) 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 -- 172.21.15.46:0/1324899929 learned_addr learned my addr 172.21.15.46:0/1324899929 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29daa00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e34263cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e34263d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 -- 172.21.15.46:0/1324899929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 msgr2=0x55e34263d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e34263d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 -- 172.21.15.46:0/1324899929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 msgr2=0x55e34263cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e34263cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 -- 172.21.15.46:0/1324899929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e34255d050 con 0x55e34262d5e0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29daa00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e34263cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29dbe00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f29d4011d10 tx=0x7f29d4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99dcd030f85faa88 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.885+0000 7f29c5600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29d401e040 con 0x55e34262d5e0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f29c8042070 con 0x55e34262d5e0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f29d400fa90 con 0x55e34262d5e0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29d400fc90 con 0x55e34262d5e0 2024-03-08T23:12:02.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29e4ffe740 10 client.26522 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e34269e770 con 0x55e34262d5e0 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29c8042070 con 0x55e34262d5e0 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.886+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f29d4024040 con 0x55e34262d5e0 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 10 client.26522 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 10 client.26522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 20 client.26522 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e34269ed80 0x55e3426a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 4 client.26522 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:12:02.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e3425992e0 con 0x55e34269ed80 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29daa00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e34269ed80 0x55e3426a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29daa00640 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e34269ed80 0x55e3426a11a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f3321a0cefdcf41 server_cookie=60e9554c6a1a379e in_seq=0 out_seq=0 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.887+0000 7f29c5600640 10 client.26522 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29c5600640 1 -- 172.21.15.46:0/1324899929 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e3425992e0 con 0x55e34269ed80 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29c5600640 10 client.26522 handle_command_reply: tid=0 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 client.26522 shutdown 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 msgr2=0x55e34263ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f29d4011d10 tx=0x7f29d4011810 comp rx=0 tx=0).stop 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 shutdown_connections 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e34269ed80 0x55e3426a11a0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e342636ae0 0x55e34263d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e34262df00 0x55e34263cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 --2- 172.21.15.46:0/1324899929 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e34262d5e0 0x55e34263ca30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29c5600640 0 client.26522 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 >> 172.21.15.46:0/1324899929 conn(0x55e342627d10 msgr2=0x55e3426339d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 shutdown_connections 2024-03-08T23:12:02.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 1 -- 172.21.15.46:0/1324899929 wait complete. 2024-03-08T23:12:02.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.889+0000 7f29e4ffe740 20 client.26522 trim_cache size 0 max 16384 2024-03-08T23:12:02.900 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:02.900 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:12:02.900 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "52666932-8050-47fb-aad2-98a09a0484a9", 2024-03-08T23:12:02.901 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:12:02.901 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:02.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 msgr2=0x7f29c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 0x7f29c00a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00e9c0 tx=0x7f29cc0d2f90 comp rx=0 tx=0).stop 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 msgr2=0x7f29dc19fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00e9f0 tx=0x7f29cc00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 shutdown_connections 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f29c00a2970 0x7f29c00a4e30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f29dc109570 0x7f29dc199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f29dc103e70 0x7f29dc19ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 --2- 172.21.15.46:0/1970143963 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29dc1034b0 0x7f29dc19fa50 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:02.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 >> 172.21.15.46:0/1970143963 conn(0x7f29dc0fef40 msgr2=0x7f29dc100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:02.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 shutdown_connections 2024-03-08T23:12:02.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:02.890+0000 7f29e2400640 1 -- 172.21.15.46:0/1970143963 wait complete. 2024-03-08T23:12:02.922 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:12:02.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 -- 172.21.15.46:0/2140367806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65401034b0 msgr2=0x7f6540103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 --2- 172.21.15.46:0/2140367806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65401034b0 0x7f6540103930 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f652c009930 tx=0x7f652c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 -- 172.21.15.46:0/2140367806 shutdown_connections 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 --2- 172.21.15.46:0/2140367806 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540103e70 0x7f6540110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 --2- 172.21.15.46:0/2140367806 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f65401034b0 0x7f6540103930 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 --2- 172.21.15.46:0/2140367806 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6540109570 0x7f6540109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 -- 172.21.15.46:0/2140367806 >> 172.21.15.46:0/2140367806 conn(0x7f65400fef60 msgr2=0x7f6540101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 -- 172.21.15.46:0/2140367806 shutdown_connections 2024-03-08T23:12:03.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.201+0000 7f6544c00640 1 -- 172.21.15.46:0/2140367806 wait complete. 2024-03-08T23:12:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.203+0000 7f6544c00640 1 Processor -- start 2024-03-08T23:12:03.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.203+0000 7f6544c00640 1 -- start start 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.203+0000 7f6544c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.203+0000 7f6544c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6540103e70 0x7f654019a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.203+0000 7f6544c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 0x7f654019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f6544c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540113880 con 0x7f65401034b0 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f6544c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540113700 con 0x7f6540109570 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f6544c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6540113a00 con 0x7f6540103e70 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 0x7f654019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50628/0 (socket says 172.21.15.46:50628) 2024-03-08T23:12:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653de00640 1 -- 172.21.15.46:0/3010107228 learned_addr learned my addr 172.21.15.46:0/3010107228 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:03.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653d400640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6540103e70 0x7f654019a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653de00640 1 -- 172.21.15.46:0/3010107228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6540103e70 msgr2=0x7f654019a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.204+0000 7f653de00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6540103e70 0x7f654019a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.205+0000 7f653de00640 1 -- 172.21.15.46:0/3010107228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 msgr2=0x7f654019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.205+0000 7f653de00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 0x7f654019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.205+0000 7f653de00640 1 -- 172.21.15.46:0/3010107228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654019ee30 con 0x7f65401034b0 2024-03-08T23:12:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.205+0000 7f653e800640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 0x7f654019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:03.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.205+0000 7f653de00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9f0 tx=0x7f653000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.206+0000 7f6536a00640 1 -- 172.21.15.46:0/3010107228 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653000cd80 con 0x7f65401034b0 2024-03-08T23:12:03.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.206+0000 7f6536a00640 1 -- 172.21.15.46:0/3010107228 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6530010910 con 0x7f65401034b0 2024-03-08T23:12:03.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.206+0000 7f6536a00640 1 -- 172.21.15.46:0/3010107228 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530018990 con 0x7f65401034b0 2024-03-08T23:12:03.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.206+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654019f120 con 0x7f65401034b0 2024-03-08T23:12:03.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.206+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65401a69b0 con 0x7f65401034b0 2024-03-08T23:12:03.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.210+0000 7f65477f3740 1 Processor -- start 2024-03-08T23:12:03.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.210+0000 7f65477f3740 1 -- start start 2024-03-08T23:12:03.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.210+0000 7f6536a00640 1 -- 172.21.15.46:0/3010107228 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6530010ac0 con 0x7f65401034b0 2024-03-08T23:12:03.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.210+0000 7f65477f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f65477f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b1892d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f65477f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b1895680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f65477f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b1843410 con 0x55e3b1893290 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f65477f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b17e5d50 con 0x55e3b1881390 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f65477f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b178eb80 con 0x55e3b1881d40 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43310/0 (socket says 172.21.15.46:43310) 2024-03-08T23:12:03.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653e800640 1 -- 172.21.15.46:0/3323086130 learned_addr learned my addr 172.21.15.46:0/3323086130 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653e800640 1 -- 172.21.15.46:0/3323086130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 msgr2=0x55e3b1892d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653d400640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b1895680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653de00640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b1892d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.211+0000 7f653e800640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b1892d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653e800640 1 -- 172.21.15.46:0/3323086130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 msgr2=0x55e3b1895680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653e800640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b1895680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653e800640 1 -- 172.21.15.46:0/3323086130 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3b15c47d0 con 0x55e3b1881390 2024-03-08T23:12:03.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653de00640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b1892d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653d400640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b1895680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f653e800640 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6538011810 tx=0x7f6538011cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b436e4aa2ee3dd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f652b000640 1 -- 172.21.15.46:0/3323086130 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538011310 con 0x55e3b1881390 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.212+0000 7f652b000640 1 -- 172.21.15.46:0/3323086130 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6538003e50 con 0x55e3b1881390 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.213+0000 7f6536a00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 0x7f65240a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.213+0000 7f652b000640 1 -- 172.21.15.46:0/3323086130 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6538004620 con 0x55e3b1881390 2024-03-08T23:12:03.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.213+0000 7f6536a00640 1 -- 172.21.15.46:0/3010107228 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f65300cf4e0 con 0x7f65401034b0 2024-03-08T23:12:03.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.213+0000 7f653de00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 0x7f65240a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 -- 172.21.15.46:0/3323086130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 msgr2=0x55e3b1881770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6538011810 tx=0x7f6538011cd0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 -- 172.21.15.46:0/3323086130 shutdown_connections 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b1895680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b1892d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 --2- 172.21.15.46:0/3323086130 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1881770 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 -- 172.21.15.46:0/3323086130 >> 172.21.15.46:0/3323086130 conn(0x55e3b187bcb0 msgr2=0x55e3b187c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 -- 172.21.15.46:0/3323086130 shutdown_connections 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f65477f3740 1 -- 172.21.15.46:0/3323086130 wait complete. 2024-03-08T23:12:03.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.214+0000 7f653de00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 0x7f65240a4f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55e3b187cf40 tx=0x7f6530000bc0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.215+0000 7f65477f3740 1 -- start start 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.215+0000 7f65477f3740 1 Processor -- start 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 0x55e3b1890240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 0x55e3b1890780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b1890cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b17a1ee0 con 0x55e3b1881390 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b17e1df0 con 0x55e3b1881d40 2024-03-08T23:12:03.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3b1896050 con 0x55e3b1893290 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f65477f3740 10 client.0 _ll_register_callbacks cb 0x55e3b17a88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f653e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 0x55e3b1890240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f653e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 0x55e3b1890240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50644/0 (socket says 172.21.15.46:50644) 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f653e800640 1 -- 172.21.15.46:0/3497901969 learned_addr learned my addr 172.21.15.46:0/3497901969 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f653d400640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 0x55e3b1890780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.216+0000 7f653e800640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b1890cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 -- 172.21.15.46:0/3497901969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 msgr2=0x55e3b1890cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b1890cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 -- 172.21.15.46:0/3497901969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 msgr2=0x55e3b1890240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 0x55e3b1890240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 -- 172.21.15.46:0/3497901969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3b14fa3f0 con 0x55e3b1881d40 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653e800640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b1890cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f653d400640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 0x55e3b1890780 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f652c036040 tx=0x7f652c0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b55d56ad340892 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.217+0000 7f651f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.218+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c005d50 con 0x55e3b1881d40 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.218+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f652c02cc90 con 0x55e3b1881d40 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.218+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6538003e50 con 0x55e3b1881d40 2024-03-08T23:12:03.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.218+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f652c005020 con 0x55e3b1881d40 2024-03-08T23:12:03.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.219+0000 7f65477f3740 10 client.26534 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.219+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e3b14ba0f0 con 0x55e3b1881d40 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.219+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6538003e50 con 0x55e3b1881d40 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f652c0035f0 con 0x55e3b1881d40 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 10 client.26534 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 10 client.26534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 20 client.26534 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18a3390 0x55e3b18a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 4 client.26534 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3b17f3290 con 0x55e3b18a3390 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.220+0000 7f653e800640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18a3390 0x55e3b18a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.222+0000 7f653e800640 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18a3390 0x55e3b18a57c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a7e5c8d11e4ada6 server_cookie=9f719df6baaf7d7e in_seq=0 out_seq=0 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.222+0000 7f651f400640 10 client.26534 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.222+0000 7f651f400640 1 -- 172.21.15.46:0/3497901969 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3b17f3290 con 0x55e3b18a3390 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.222+0000 7f651f400640 10 client.26534 handle_command_reply: tid=0 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.222+0000 7f65477f3740 1 client.26534 shutdown 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 msgr2=0x55e3b1890780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 0x55e3b1890780 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f652c036040 tx=0x7f652c0095e0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 shutdown_connections 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18a3390 0x55e3b18a57c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b1890cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881d40 0x55e3b1890780 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 --2- 172.21.15.46:0/3497901969 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881390 0x55e3b1890240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 >> 172.21.15.46:0/3497901969 conn(0x55e3b187bcb0 msgr2=0x55e3b187da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f651f400640 0 client.26534 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 shutdown_connections 2024-03-08T23:12:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.223+0000 7f65477f3740 1 -- 172.21.15.46:0/3497901969 wait complete. 2024-03-08T23:12:03.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.224+0000 7f65477f3740 20 client.26534 trim_cache size 0 max 16384 2024-03-08T23:12:03.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.248+0000 7f65477f3740 1 Processor -- start 2024-03-08T23:12:03.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.248+0000 7f65477f3740 1 -- start start 2024-03-08T23:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.248+0000 7f65477f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.248+0000 7f65477f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 0x55e3b1890ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f65477f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b189a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f65477f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401ae940 con 0x55e3b1881d40 2024-03-08T23:12:03.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f65477f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401ac480 con 0x55e3b1881390 2024-03-08T23:12:03.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f65477f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401aa520 con 0x55e3b1893290 2024-03-08T23:12:03.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 0x55e3b1890ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43328/0 (socket says 172.21.15.46:43328) 2024-03-08T23:12:03.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653e800640 1 -- 172.21.15.46:0/1534187278 learned_addr learned my addr 172.21.15.46:0/1534187278 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:03.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653e800640 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b189a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653d400640 1 -- 172.21.15.46:0/1534187278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 msgr2=0x55e3b189a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.249+0000 7f653d400640 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b189a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f653d400640 1 -- 172.21.15.46:0/1534187278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 msgr2=0x55e3b1892970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f653d400640 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f653d400640 1 -- 172.21.15.46:0/1534187278 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3b1881aa0 con 0x55e3b1881d40 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f653e800640 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f653d400640 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 0x55e3b1890ae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f652c003a10 tx=0x7f652c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=419d39586bcbe556 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.250+0000 7f652a600640 1 -- 172.21.15.46:0/1534187278 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c056020 con 0x55e3b1881d40 2024-03-08T23:12:03.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.251+0000 7f652a600640 1 -- 172.21.15.46:0/1534187278 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f652c0035f0 con 0x55e3b1881d40 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.251+0000 7f652a600640 1 -- 172.21.15.46:0/1534187278 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c02cd10 con 0x55e3b1881d40 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- 172.21.15.46:0/1534187278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 msgr2=0x55e3b1890ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 0x55e3b1890ae0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f652c003a10 tx=0x7f652c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- 172.21.15.46:0/1534187278 shutdown_connections 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1893290 0x55e3b189a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1881d40 0x55e3b1890ae0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 --2- 172.21.15.46:0/1534187278 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b1892970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- 172.21.15.46:0/1534187278 >> 172.21.15.46:0/1534187278 conn(0x55e3b187bcb0 msgr2=0x55e3b18aa120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- 172.21.15.46:0/1534187278 shutdown_connections 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- 172.21.15.46:0/1534187278 wait complete. 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.252+0000 7f65477f3740 1 -- start start 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 Processor -- start 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b18f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b18f3270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b18f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401aab20 con 0x55e3b1893290 2024-03-08T23:12:03.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401abe20 con 0x55e3b1881390 2024-03-08T23:12:03.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65401ab160 con 0x55e3b1881d40 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.253+0000 7f65477f3740 10 client.0 _ll_register_callbacks cb 0x7f6538009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.254+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b18f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.254+0000 7f653e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b18f2d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43330/0 (socket says 172.21.15.46:43330) 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.254+0000 7f653e800640 1 -- 172.21.15.46:0/4127971468 learned_addr learned my addr 172.21.15.46:0/4127971468 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.254+0000 7f653d400640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b18f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.254+0000 7f653e800640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b18f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 -- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 msgr2=0x55e3b18f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b18f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 -- 172.21.15.46:0/4127971468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 msgr2=0x55e3b18f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b18f1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 -- 172.21.15.46:0/4127971468 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3b18f2110 con 0x55e3b1881d40 2024-03-08T23:12:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653e800640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b18f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:03.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f653d400640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b18f3270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f652c003ab0 tx=0x7f652c003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57101c39e91b36ab server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:03.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.255+0000 7f651f400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:12:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.256+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652c056070 con 0x55e3b1881d40 2024-03-08T23:12:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.256+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f652c0046c0 con 0x55e3b1881d40 2024-03-08T23:12:03.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.256+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f652c0035f0 con 0x55e3b1881d40 2024-03-08T23:12:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.256+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f652c0037a0 con 0x55e3b1881d40 2024-03-08T23:12:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 10 client.26527 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e3b18fa6c0 con 0x55e3b1881d40 2024-03-08T23:12:03.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f652c0035f0 con 0x55e3b1881d40 2024-03-08T23:12:03.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f652c052960 con 0x55e3b1881d40 2024-03-08T23:12:03.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 10 client.26527 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:03.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 10 client.26527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:03.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 20 client.26527 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:03.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18fae10 0x55e3b18fd230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:03.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 4 client.26527 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:12:03.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.257+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e3b17ed5e0 con 0x55e3b18fae10 2024-03-08T23:12:03.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.258+0000 7f653e800640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18fae10 0x55e3b18fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:03.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.258+0000 7f651f400640 10 client.26527 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:03.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.259+0000 7f653e800640 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18fae10 0x55e3b18fd230 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b128677ee2af62b server_cookie=93feb98da0976ec0 in_seq=0 out_seq=0 2024-03-08T23:12:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.259+0000 7f651f400640 1 -- 172.21.15.46:0/4127971468 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e3b17ed5e0 con 0x55e3b18fae10 2024-03-08T23:12:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.259+0000 7f651f400640 10 client.26527 handle_command_reply: tid=0 2024-03-08T23:12:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.259+0000 7f65477f3740 1 client.26527 shutdown 2024-03-08T23:12:03.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 msgr2=0x55e3b18f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b18f3270 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f652c003ab0 tx=0x7f652c003c20 comp rx=0 tx=0).stop 2024-03-08T23:12:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 shutdown_connections 2024-03-08T23:12:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3b18fae10 0x55e3b18fd230 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3b1893290 0x55e3b18f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f651f400640 0 client.26527 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:03.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3b1881d40 0x55e3b18f3270 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 --2- 172.21.15.46:0/4127971468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3b1881390 0x55e3b18f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 >> 172.21.15.46:0/4127971468 conn(0x55e3b187bcb0 msgr2=0x55e3b188b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 shutdown_connections 2024-03-08T23:12:03.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 1 -- 172.21.15.46:0/4127971468 wait complete. 2024-03-08T23:12:03.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.260+0000 7f65477f3740 20 client.26527 trim_cache size 0 max 16384 2024-03-08T23:12:03.279 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:03.279 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:12:03.280 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:12:03.280 INFO:teuthology.orchestra.run.smithi046.stdout: "52666932-8050-47fb-aad2-98a09a0484a9": { 2024-03-08T23:12:03.280 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:12:03.280 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "52666932-8050-47fb-aad2-98a09a0484a9", 2024-03-08T23:12:03.281 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:12:03.281 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:12:03.281 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:12:03.281 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:03.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 msgr2=0x7f65240a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 0x7f65240a4f30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55e3b187cf40 tx=0x7f6530000bc0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 msgr2=0x7f6540199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:03.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f653000e9f0 tx=0x7f653000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 shutdown_connections 2024-03-08T23:12:03.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f65240a2a70 0x7f65240a4f30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6540109570 0x7f654019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6540103e70 0x7f654019a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 --2- 172.21.15.46:0/3010107228 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f65401034b0 0x7f6540199e40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:03.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.261+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 >> 172.21.15.46:0/3010107228 conn(0x7f65400fef60 msgr2=0x7f6540100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:03.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.262+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 shutdown_connections 2024-03-08T23:12:03.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:03.262+0000 7f6544c00640 1 -- 172.21.15.46:0/3010107228 wait complete. 2024-03-08T23:12:03.290 INFO:tasks.cephfs.filesystem:scrub status for tag:52666932-8050-47fb-aad2-98a09a0484a9 - {'path': '/', 'tag': '52666932-8050-47fb-aad2-98a09a0484a9', 'options': 'recursive,force'} 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: pgmap v131: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:04.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:04 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: pgmap v131: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:04 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: pgmap v131: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 25 KiB/s wr, 22 op/s 2024-03-08T23:12:04.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:04.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:04 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:06.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:06 smithi192 ceph-mon[36700]: pgmap v132: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:06.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:06 smithi062 ceph-mon[35761]: pgmap v132: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:06.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:06 smithi046 ceph-mon[31859]: pgmap v132: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:08.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:08 smithi192 ceph-mon[36700]: pgmap v133: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:08.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:08 smithi062 ceph-mon[35761]: pgmap v133: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:08.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:08 smithi046 ceph-mon[31859]: pgmap v133: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:10.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:10 smithi192 ceph-mon[36700]: pgmap v134: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:10 smithi062 ceph-mon[35761]: pgmap v134: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:10 smithi046 ceph-mon[31859]: pgmap v134: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:12.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:12 smithi192 ceph-mon[36700]: pgmap v135: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:12 smithi062 ceph-mon[35761]: pgmap v135: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:12 smithi046 ceph-mon[31859]: pgmap v135: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:12:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:14 smithi062 ceph-mon[35761]: pgmap v136: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:14.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:14 smithi046 ceph-mon[31859]: pgmap v136: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:14.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:14 smithi192 ceph-mon[36700]: pgmap v136: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:12:15.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:15 smithi062 ceph-mon[35761]: pgmap v137: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:15.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:15 smithi046 ceph-mon[31859]: pgmap v137: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:15.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:15 smithi192 ceph-mon[36700]: pgmap v137: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:17 smithi062 ceph-mon[35761]: pgmap v138: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:17 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:17.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:17 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:17 smithi046 ceph-mon[31859]: pgmap v138: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:17 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:17.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:17 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:17 smithi192 ceph-mon[36700]: pgmap v138: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:12:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:17 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:18.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:17 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:18.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:18 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:19.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:18 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:19.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:18 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:19.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:19 smithi046 ceph-mon[31859]: pgmap v139: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:12:20.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:19 smithi192 ceph-mon[36700]: pgmap v139: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:12:20.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:19 smithi062 ceph-mon[35761]: pgmap v139: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:12:21.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:21 smithi046 ceph-mon[31859]: pgmap v140: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:12:22.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:21 smithi192 ceph-mon[36700]: pgmap v140: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:12:22.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:21 smithi062 ceph-mon[35761]: pgmap v140: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:12:23.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:23 smithi046 ceph-mon[31859]: pgmap v141: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:24.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:23 smithi192 ceph-mon[36700]: pgmap v141: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:24.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:23 smithi062 ceph-mon[35761]: pgmap v141: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:25.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:25 smithi046 ceph-mon[31859]: pgmap v142: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:26.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:25 smithi192 ceph-mon[36700]: pgmap v142: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:26.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:25 smithi062 ceph-mon[35761]: pgmap v142: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:27.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:27 smithi046 ceph-mon[31859]: pgmap v143: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:28.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:27 smithi192 ceph-mon[36700]: pgmap v143: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:28.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:27 smithi062 ceph-mon[35761]: pgmap v143: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:29.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:29 smithi046 ceph-mon[31859]: pgmap v144: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:30.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:29 smithi192 ceph-mon[36700]: pgmap v144: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:30.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:29 smithi062 ceph-mon[35761]: pgmap v144: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:31.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:31 smithi046 ceph-mon[31859]: pgmap v145: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:32.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:31 smithi192 ceph-mon[36700]: pgmap v145: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:32.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:31 smithi062 ceph-mon[35761]: pgmap v145: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:33.291 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:12:33.292 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:12:33.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 -- 172.21.15.46:0/855781303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc0ffa60 msgr2=0x7feffc0ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 --2- 172.21.15.46:0/855781303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc0ffa60 0x7feffc0ffe40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fefec009900 tx=0x7fefec02bcc0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 -- 172.21.15.46:0/855781303 shutdown_connections 2024-03-08T23:12:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 --2- 172.21.15.46:0/855781303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 0x7feffc1142e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 --2- 172.21.15.46:0/855781303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc100380 0x7feffc1119b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 --2- 172.21.15.46:0/855781303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc0ffa60 0x7feffc0ffe40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 -- 172.21.15.46:0/855781303 >> 172.21.15.46:0/855781303 conn(0x7feffc075330 msgr2=0x7feffc075740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 -- 172.21.15.46:0/855781303 shutdown_connections 2024-03-08T23:12:33.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.522+0000 7feffb200640 1 -- 172.21.15.46:0/855781303 wait complete. 2024-03-08T23:12:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 Processor -- start 2024-03-08T23:12:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 -- start start 2024-03-08T23:12:33.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0ffa60 0x7feffc19dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 0x7feffc1a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc117b90 con 0x7feffc100380 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc117a10 con 0x7feffc111ef0 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.523+0000 7feffb200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc117d10 con 0x7feffc0ffa60 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50124/0 (socket says 172.21.15.46:50124) 2024-03-08T23:12:33.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 -- 172.21.15.46:0/1440080443 learned_addr learned my addr 172.21.15.46:0/1440080443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feffa800640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 0x7feffc1a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff9e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0ffa60 0x7feffc19dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 -- 172.21.15.46:0/1440080443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0ffa60 msgr2=0x7feffc19dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0ffa60 0x7feffc19dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 -- 172.21.15.46:0/1440080443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 msgr2=0x7feffc1a28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 0x7feffc1a28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.524+0000 7feff3e00640 1 -- 172.21.15.46:0/1440080443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc1a2fe0 con 0x7feffc100380 2024-03-08T23:12:33.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feff3e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fefe400e9f0 tx=0x7fefe400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feff2a00640 1 -- 172.21.15.46:0/1440080443 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe400cd80 con 0x7feffc100380 2024-03-08T23:12:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feff2a00640 1 -- 172.21.15.46:0/1440080443 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefe4005500 con 0x7feffc100380 2024-03-08T23:12:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc1a32d0 con 0x7feffc100380 2024-03-08T23:12:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feff2a00640 1 -- 172.21.15.46:0/1440080443 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4010690 con 0x7feffc100380 2024-03-08T23:12:33.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.525+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc1aab60 con 0x7feffc100380 2024-03-08T23:12:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.528+0000 7feff2a00640 1 -- 172.21.15.46:0/1440080443 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fefe40046c0 con 0x7feffc100380 2024-03-08T23:12:33.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.528+0000 7ff003dcf740 1 Processor -- start 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 -- start start 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 0x565361806750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565361806c90 0x565361809080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5653617bf410 con 0x5653617fe0b0 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x565361761d50 con 0x565361806c90 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.529+0000 7ff003dcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56536170ab80 con 0x5653617fd790 2024-03-08T23:12:33.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feff9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 0x565361806750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52998/0 (socket says 172.21.15.46:52998) 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 -- 172.21.15.46:0/1775051148 learned_addr learned my addr 172.21.15.46:0/1775051148 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feff3e00640 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565361806c90 0x565361809080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 -- 172.21.15.46:0/1775051148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565361806c90 msgr2=0x565361809080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565361806c90 0x565361809080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 -- 172.21.15.46:0/1775051148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 msgr2=0x565361806750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 0x565361806750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 -- 172.21.15.46:0/1775051148 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653615407d0 con 0x5653617fd790 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feff9e00640 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 0x565361806750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feff2a00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 0x7fefe00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feffa800640 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feff4009f90 tx=0x7feff40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=defa15246d1f3f42 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.530+0000 7feff2a00640 1 -- 172.21.15.46:0/1440080443 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fefe40cf610 con 0x7feffc100380 2024-03-08T23:12:33.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.531+0000 7feff9e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 0x7fefe00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.531+0000 7fefeb400640 1 -- 172.21.15.46:0/1775051148 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4004d20 con 0x5653617fd790 2024-03-08T23:12:33.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.531+0000 7fefeb400640 1 -- 172.21.15.46:0/1775051148 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feff4004050 con 0x5653617fd790 2024-03-08T23:12:33.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.531+0000 7fefeb400640 1 -- 172.21.15.46:0/1775051148 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff4005220 con 0x5653617fd790 2024-03-08T23:12:33.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- 172.21.15.46:0/1775051148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 msgr2=0x5653617fdb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7feff9e00640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 0x7fefe00a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fefec005ab0 tx=0x7fefec02e040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7feff4009f90 tx=0x7feff40118d0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- 172.21.15.46:0/1775051148 shutdown_connections 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x565361806c90 0x565361809080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fe0b0 0x565361806750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 --2- 172.21.15.46:0/1775051148 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fd790 0x5653617fdb70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- 172.21.15.46:0/1775051148 >> 172.21.15.46:0/1775051148 conn(0x5653617f7ec0 msgr2=0x5653617f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- 172.21.15.46:0/1775051148 shutdown_connections 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- 172.21.15.46:0/1775051148 wait complete. 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.532+0000 7ff003dcf740 1 -- start start 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 Processor -- start 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361803ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x565361804760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56536171dee0 con 0x5653617fd790 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56536175ddf0 con 0x5653617fe0b0 2024-03-08T23:12:33.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x565361809a80 con 0x565361806c90 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7ff003dcf740 10 client.0 _ll_register_callbacks cb 0x5653617248d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7feff9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361803ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7feff3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7feffa800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x565361804760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7feff3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:50580/0 (socket says 172.21.15.46:50580) 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.533+0000 7feff3e00640 1 -- 172.21.15.46:0/40220447 learned_addr learned my addr 172.21.15.46:0/40220447 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 -- 172.21.15.46:0/40220447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 msgr2=0x565361804760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x565361804760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 -- 172.21.15.46:0/40220447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 msgr2=0x565361803ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361803ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 -- 172.21.15.46:0/40220447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653614763f0 con 0x5653617fe0b0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7feff3e00640 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c6a0 tx=0x7fefe4005730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e5953223b5d72ba server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7fefdfe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe409b210 con 0x5653617fe0b0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fefe40cf230 con 0x5653617fe0b0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feff4004050 con 0x5653617fe0b0 2024-03-08T23:12:33.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefe409bfb0 con 0x5653617fe0b0 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.534+0000 7ff003dcf740 10 client.26552 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5653614360f0 con 0x5653617fe0b0 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feff4004050 con 0x5653617fe0b0 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fefe40ccc20 con 0x5653617fe0b0 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 10 client.26552 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 10 client.26552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 20 client.26552 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:33.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x565361816e60 0x565361819250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 4 client.26552 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56536176f290 con 0x565361816e60 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.535+0000 7feff9e00640 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x565361816e60 0x565361819250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.536+0000 7feff9e00640 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x565361816e60 0x565361819250 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88281a7561241dac server_cookie=e13211c5b7977d4 in_seq=0 out_seq=0 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.536+0000 7fefdfe00640 10 client.26552 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.537+0000 7fefdfe00640 1 -- 172.21.15.46:0/40220447 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56536176f290 con 0x565361816e60 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.537+0000 7fefdfe00640 10 client.26552 handle_command_reply: tid=0 2024-03-08T23:12:33.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.537+0000 7ff003dcf740 1 client.26552 shutdown 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 msgr2=0x565361804220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c6a0 tx=0x7fefe4005730 comp rx=0 tx=0).stop 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 shutdown_connections 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x565361816e60 0x565361819250 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x565361804760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x565361804220 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 --2- 172.21.15.46:0/40220447 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361803ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 >> 172.21.15.46:0/40220447 conn(0x5653617f7ec0 msgr2=0x5653617f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7fefdfe00640 0 client.26552 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:33.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 shutdown_connections 2024-03-08T23:12:33.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 1 -- 172.21.15.46:0/40220447 wait complete. 2024-03-08T23:12:33.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.538+0000 7ff003dcf740 20 client.26552 trim_cache size 0 max 16384 2024-03-08T23:12:33.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.554+0000 7ff003dcf740 1 Processor -- start 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 -- start start 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x56536180dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x56536180e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b6d90 con 0x5653617fd790 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b48d0 con 0x5653617fe0b0 2024-03-08T23:12:33.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7ff003dcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b2970 con 0x565361806c90 2024-03-08T23:12:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7feff9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7feffa800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x56536180e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7feff9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50148/0 (socket says 172.21.15.46:50148) 2024-03-08T23:12:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7feff9e00640 1 -- 172.21.15.46:0/2129002565 learned_addr learned my addr 172.21.15.46:0/2129002565 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.555+0000 7feff3e00640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x56536180dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 -- 172.21.15.46:0/2129002565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 msgr2=0x56536180e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x56536180e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 -- 172.21.15.46:0/2129002565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 msgr2=0x565361806410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 -- 172.21.15.46:0/2129002565 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5653617f3790 con 0x5653617fe0b0 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feffa800640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x56536180e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:33.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff9e00640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7feff3e00640 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x56536180dcc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fefe4099b60 tx=0x7fefe400e940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a1d13c98db3983 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7fefeaa00640 1 -- 172.21.15.46:0/2129002565 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe409b210 con 0x5653617fe0b0 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7fefeaa00640 1 -- 172.21.15.46:0/2129002565 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefe40a4050 con 0x5653617fe0b0 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.556+0000 7fefeaa00640 1 -- 172.21.15.46:0/2129002565 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4086070 con 0x5653617fe0b0 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 -- 172.21.15.46:0/2129002565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 msgr2=0x56536180dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x56536180dcc0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fefe4099b60 tx=0x7fefe400e940 comp rx=0 tx=0).stop 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 -- 172.21.15.46:0/2129002565 shutdown_connections 2024-03-08T23:12:33.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x565361806c90 0x56536180e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fe0b0 0x56536180dcc0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 --2- 172.21.15.46:0/2129002565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5653617fd790 0x565361806410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 -- 172.21.15.46:0/2129002565 >> 172.21.15.46:0/2129002565 conn(0x5653617f7ec0 msgr2=0x5653618025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 -- 172.21.15.46:0/2129002565 shutdown_connections 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.557+0000 7ff003dcf740 1 -- 172.21.15.46:0/2129002565 wait complete. 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 -- start start 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 Processor -- start 2024-03-08T23:12:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fd790 0x565361866e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 0x5653618612e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b2f70 con 0x565361806c90 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b4270 con 0x5653617fd790 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1b35b0 con 0x5653617fe0b0 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.558+0000 7ff003dcf740 10 client.0 _ll_register_callbacks cb 0x7feff4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53032/0 (socket says 172.21.15.46:53032) 2024-03-08T23:12:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 -- 172.21.15.46:0/3226385150 learned_addr learned my addr 172.21.15.46:0/3226385150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff9e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fd790 0x565361866e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feffa800640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 0x5653618612e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 -- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fd790 msgr2=0x565361866e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fd790 0x565361866e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 -- 172.21.15.46:0/3226385150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 msgr2=0x5653618612e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 0x5653618612e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 -- 172.21.15.46:0/3226385150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5653618619d0 con 0x5653617fe0b0 2024-03-08T23:12:33.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feffa800640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 0x5653618612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7feff3e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefe4097d10 tx=0x7fefe409cef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2977686796f1d240 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.559+0000 7fefdfe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe409b210 con 0x5653617fe0b0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fefe409ad00 con 0x5653617fe0b0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefe40a4050 con 0x5653617fe0b0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefe40cce30 con 0x5653617fe0b0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7ff003dcf740 10 client.26545 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56536186e650 con 0x5653617fe0b0 2024-03-08T23:12:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefe40a4050 con 0x5653617fe0b0 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.560+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fefe40a6080 con 0x5653617fe0b0 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 10 client.26545 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 10 client.26545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 20 client.26545 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56536186eaa0 0x565361870ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 4 client.26545 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:12:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5653617695e0 con 0x56536186eaa0 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7feff9e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56536186eaa0 0x565361870ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7feff9e00640 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56536186eaa0 0x565361870ec0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2b3c4d605b42289 server_cookie=5ed418c2f1fc593c in_seq=0 out_seq=0 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.561+0000 7fefdfe00640 10 client.26545 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7fefdfe00640 1 -- 172.21.15.46:0/3226385150 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5653617695e0 con 0x56536186eaa0 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7fefdfe00640 10 client.26545 handle_command_reply: tid=0 2024-03-08T23:12:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 client.26545 shutdown 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 msgr2=0x5653618673c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fefe4097d10 tx=0x7fefe409cef0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 shutdown_connections 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56536186eaa0 0x565361870ec0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x565361806c90 0x5653618612e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5653617fe0b0 0x5653618673c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 --2- 172.21.15.46:0/3226385150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5653617fd790 0x565361866e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7fefdfe00640 0 client.26545 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 >> 172.21.15.46:0/3226385150 conn(0x5653617f7ec0 msgr2=0x565361803ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 shutdown_connections 2024-03-08T23:12:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 1 -- 172.21.15.46:0/3226385150 wait complete. 2024-03-08T23:12:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.562+0000 7ff003dcf740 20 client.26545 trim_cache size 0 max 16384 2024-03-08T23:12:33.574 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:33.574 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:12:33.575 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:12:33.575 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:33.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 msgr2=0x7fefe00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 0x7fefe00a4e30 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fefec005ab0 tx=0x7fefec02e040 comp rx=0 tx=0).stop 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 msgr2=0x7feffc19e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fefe400e9f0 tx=0x7fefe400edf0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 shutdown_connections 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fefe00a2970 0x7fefe00a4e30 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7feffc111ef0 0x7feffc1a28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7feffc100380 0x7feffc19e530 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 --2- 172.21.15.46:0/1440080443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feffc0ffa60 0x7feffc19dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 >> 172.21.15.46:0/1440080443 conn(0x7feffc075330 msgr2=0x7feffc10f9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 shutdown_connections 2024-03-08T23:12:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.563+0000 7feffb200640 1 -- 172.21.15.46:0/1440080443 wait complete. 2024-03-08T23:12:33.584 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:12:33.585 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:12:33.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.815+0000 7f6c69a00640 1 -- 172.21.15.46:0/4014115055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64109560 msgr2=0x7f6c64109940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.815+0000 7f6c69a00640 1 --2- 172.21.15.46:0/4014115055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64109560 0x7f6c64109940 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6c54009900 tx=0x7f6c5402bc50 comp rx=0 tx=0).stop 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 -- 172.21.15.46:0/4014115055 shutdown_connections 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 --2- 172.21.15.46:0/4014115055 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c64103e60 0x7f6c64110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 --2- 172.21.15.46:0/4014115055 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c641034a0 0x7f6c64103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 --2- 172.21.15.46:0/4014115055 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64109560 0x7f6c64109940 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 -- 172.21.15.46:0/4014115055 >> 172.21.15.46:0/4014115055 conn(0x7f6c640fef50 msgr2=0x7f6c64101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 -- 172.21.15.46:0/4014115055 shutdown_connections 2024-03-08T23:12:33.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.816+0000 7f6c69a00640 1 -- 172.21.15.46:0/4014115055 wait complete. 2024-03-08T23:12:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.817+0000 7f6c69a00640 1 Processor -- start 2024-03-08T23:12:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.817+0000 7f6c69a00640 1 -- start start 2024-03-08T23:12:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 0x7f6c6419fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c64109560 0x7f6c64199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c64113840 con 0x7f6c64103e60 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c641136c0 con 0x7f6c64109560 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c69a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c641139c0 con 0x7f6c641034a0 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c62a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c5a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 0x7f6c6419fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c62a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53038/0 (socket says 172.21.15.46:53038) 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c62a00640 1 -- 172.21.15.46:0/417454851 learned_addr learned my addr 172.21.15.46:0/417454851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.818+0000 7f6c63400640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c64109560 0x7f6c64199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 -- 172.21.15.46:0/417454851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c64109560 msgr2=0x7f6c64199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c64109560 0x7f6c64199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 -- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 msgr2=0x7f6c6419fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:33.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 0x7f6c6419fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 -- 172.21.15.46:0/417454851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6419a280 con 0x7f6c641034a0 2024-03-08T23:12:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c5a000640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 0x7f6c6419fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:33.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c62a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c54004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54041070 con 0x7f6c641034a0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c540054e0 con 0x7f6c641034a0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c54003cd0 con 0x7f6c641034a0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.819+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c6419a510 con 0x7f6c641034a0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.820+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c641a6810 con 0x7f6c641034a0 2024-03-08T23:12:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.821+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c64068530 con 0x7f6c641034a0 2024-03-08T23:12:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.822+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6c54044070 con 0x7f6c641034a0 2024-03-08T23:12:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.824+0000 7f6c5be00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 0x7f6c480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.824+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f6c540f0d50 con 0x7f6c641034a0 2024-03-08T23:12:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.824+0000 7f6c5a000640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 0x7f6c480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.825+0000 7f6c5a000640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 0x7f6c480a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c64108ba0 tx=0x7f6c4c005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:33.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:33.828+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6c540bd1b0 con 0x7f6c641034a0 2024-03-08T23:12:33.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:33 smithi046 ceph-mon[31859]: pgmap v146: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:34.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:33 smithi192 ceph-mon[36700]: pgmap v146: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:34.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.075+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c6410b980 con 0x7f6c641034a0 2024-03-08T23:12:34.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.076+0000 7f6c5be00640 1 -- 172.21.15.46:0/417454851 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11340 (secure 0 0 0) 0x7f6c540c1250 con 0x7f6c641034a0 2024-03-08T23:12:34.080 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:12:34.080 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:10:43.831330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:12:34.080 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 27 2024-03-08T23:12:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 msgr2=0x7f6c480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 0x7f6c480a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f6c64108ba0 tx=0x7f6c4c005f90 comp rx=0 tx=0).stop 2024-03-08T23:12:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 msgr2=0x7f6c6419f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6c54006010 tx=0x7f6c54004270 comp rx=0 tx=0).stop 2024-03-08T23:12:34.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 shutdown_connections 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6c480a2970 0x7f6c480a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6c64109560 0x7f6c64199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6c64103e60 0x7f6c6419fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 --2- 172.21.15.46:0/417454851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6c641034a0 0x7f6c6419f950 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 >> 172.21.15.46:0/417454851 conn(0x7f6c640fef50 msgr2=0x7f6c64100640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.079+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 shutdown_connections 2024-03-08T23:12:34.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.080+0000 7f6c69a00640 1 -- 172.21.15.46:0/417454851 wait complete. 2024-03-08T23:12:34.104 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:34.104 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:12:34.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:33 smithi062 ceph-mon[35761]: pgmap v146: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:34.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.328+0000 7f9143200640 1 -- 172.21.15.46:0/3820617448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91441035e0 msgr2=0x7f91441039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.328+0000 7f9143200640 1 --2- 172.21.15.46:0/3820617448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91441035e0 0x7f91441039c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9134009be0 tx=0x7f913402bef0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 -- 172.21.15.46:0/3820617448 shutdown_connections 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 --2- 172.21.15.46:0/3820617448 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441086b0 0x7f9144110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 --2- 172.21.15.46:0/3820617448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9144103f90 0x7f9144107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 --2- 172.21.15.46:0/3820617448 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f91441035e0 0x7f91441039c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 -- 172.21.15.46:0/3820617448 >> 172.21.15.46:0/3820617448 conn(0x7f91440fee00 msgr2=0x7f9144101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 -- 172.21.15.46:0/3820617448 shutdown_connections 2024-03-08T23:12:34.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.329+0000 7f9143200640 1 -- 172.21.15.46:0/3820617448 wait complete. 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 Processor -- start 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 -- start start 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441035e0 0x7f914419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9144199ae0 0x7f9144199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9144113760 con 0x7f9144199ae0 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441135e0 con 0x7f91441035e0 2024-03-08T23:12:34.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.330+0000 7f9143200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441138e0 con 0x7f91441086b0 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53060/0 (socket says 172.21.15.46:53060) 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9142800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9144199ae0 0x7f9144199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 -- 172.21.15.46:0/3105101970 learned_addr learned my addr 172.21.15.46:0/3105101970 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141e00640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441035e0 0x7f914419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 -- 172.21.15.46:0/3105101970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441035e0 msgr2=0x7f914419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441035e0 0x7f914419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 -- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9144199ae0 msgr2=0x7f9144199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9144199ae0 0x7f9144199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 -- 172.21.15.46:0/3105101970 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914419a600 con 0x7f91441086b0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.331+0000 7f9141400640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f913000e9c0 tx=0x7f913000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.332+0000 7f913aa00640 1 -- 172.21.15.46:0/3105101970 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913000ccc0 con 0x7f91441086b0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.332+0000 7f913aa00640 1 -- 172.21.15.46:0/3105101970 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9130005500 con 0x7f91441086b0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.332+0000 7f913aa00640 1 -- 172.21.15.46:0/3105101970 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130010690 con 0x7f91441086b0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.332+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914419a8f0 con 0x7f91441086b0 2024-03-08T23:12:34.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.332+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91441a6aa0 con 0x7f91441086b0 2024-03-08T23:12:34.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.335+0000 7f914b5e1740 1 Processor -- start 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.335+0000 7f914b5e1740 1 -- start start 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 0x55cf55bdc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f913aa00640 1 -- 172.21.15.46:0/3105101970 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f91300046c0 con 0x7f91441086b0 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 0x55cf55bdee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55b37dd0 con 0x55cf55bd3e50 2024-03-08T23:12:34.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55ae0c70 con 0x55cf55bd3530 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f914b5e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55b33e70 con 0x55cf55bdca30 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9141e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 0x55cf55bdc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9141e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 0x55cf55bdc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50192/0 (socket says 172.21.15.46:50192) 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9141e00640 1 -- 172.21.15.46:0/1996891059 learned_addr learned my addr 172.21.15.46:0/1996891059 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9141400640 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 0x55cf55bdee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 -- 172.21.15.46:0/1996891059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 msgr2=0x55cf55bdee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 0x55cf55bdee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 -- 172.21.15.46:0/1996891059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 msgr2=0x55cf55bdc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 0x55cf55bdc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9142800640 1 -- 172.21.15.46:0/1996891059 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf55b9a2b0 con 0x55cf55bd3530 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.336+0000 7f9141400640 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 0x55cf55bdee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.337+0000 7f9142800640 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd3910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f913c009b80 tx=0x7f913c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a46a3c7d63ac28f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.337+0000 7f912f000640 1 -- 172.21.15.46:0/1996891059 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c004050 con 0x55cf55bd3530 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.337+0000 7f912f000640 1 -- 172.21.15.46:0/1996891059 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f913c005500 con 0x55cf55bd3530 2024-03-08T23:12:34.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.337+0000 7f912f000640 1 -- 172.21.15.46:0/1996891059 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c004d00 con 0x55cf55bd3530 2024-03-08T23:12:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f913aa00640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 0x7f91280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- 172.21.15.46:0/1996891059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 msgr2=0x55cf55bd3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd3910 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f913c009b80 tx=0x7f913c011810 comp rx=0 tx=0).stop 2024-03-08T23:12:34.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f913aa00640 1 -- 172.21.15.46:0/3105101970 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f91300cf4a0 con 0x7f91441086b0 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- 172.21.15.46:0/1996891059 shutdown_connections 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bdca30 0x55cf55bdee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bd3e50 0x55cf55bdc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1996891059 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd3910 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f9141e00640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 0x7f91280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- 172.21.15.46:0/1996891059 >> 172.21.15.46:0/1996891059 conn(0x55cf55bcdc60 msgr2=0x55cf55bce040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- 172.21.15.46:0/1996891059 shutdown_connections 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- 172.21.15.46:0/1996891059 wait complete. 2024-03-08T23:12:34.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.338+0000 7f914b5e1740 1 -- start start 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 Processor -- start 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f9141e00640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 0x7f91280a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9134036040 tx=0x7f91340333e0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bdc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bd9780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55bdf820 con 0x55cf55bdca30 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55af40c0 con 0x55cf55bd3530 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cf55bdf9a0 con 0x55cf55bd3e50 2024-03-08T23:12:34.350 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.339+0000 7f914b5e1740 10 client.0 _ll_register_callbacks cb 0x55cf55b2eec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50196/0 (socket says 172.21.15.46:50196) 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 -- 172.21.15.46:0/1578054416 learned_addr learned my addr 172.21.15.46:0/1578054416 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9141e00640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bdc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9141400640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bd9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 -- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 msgr2=0x55cf55bd9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bd9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.351 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 -- 172.21.15.46:0/1578054416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 msgr2=0x55cf55bdc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bdc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 -- 172.21.15.46:0/1578054416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf5584c3f0 con 0x55cf55bdca30 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9141400640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bd9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9142800640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f913c004920 tx=0x7f913c004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958b2a0070baca95 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.340+0000 7f9123400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:34.352 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913c029070 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f913c003a00 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f913c005500 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f913c035050 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f914b5e1740 10 client.16425 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.341+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cf559167d0 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f913c005500 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f913c0051c0 con 0x55cf55bdca30 2024-03-08T23:12:34.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 10 client.16425 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 10 client.16425 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 20 client.16425 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55becb60 0x55cf55beef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 4 client.16425 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cf55b0c4f0 con 0x55cf55becb60 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.342+0000 7f9141e00640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55becb60 0x55cf55beef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.343+0000 7f9141e00640 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55becb60 0x55cf55beef50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6a186e192b6aa23 server_cookie=b46944f484b9d26c in_seq=0 out_seq=0 2024-03-08T23:12:34.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.343+0000 7f9123400640 10 client.16425 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f9123400640 1 -- 172.21.15.46:0/1578054416 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cf55b0c4f0 con 0x55cf55becb60 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f9123400640 10 client.16425 handle_command_reply: tid=0 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 client.16425 shutdown 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 msgr2=0x55cf55bd9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f913c004920 tx=0x7f913c004280 comp rx=0 tx=0).stop 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 shutdown_connections 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55becb60 0x55cf55beef50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bd9d90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bd9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f9123400640 0 client.16425 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 --2- 172.21.15.46:0/1578054416 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bdc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 >> 172.21.15.46:0/1578054416 conn(0x55cf55bcdc60 msgr2=0x55cf55bcf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.344+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 shutdown_connections 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.345+0000 7f914b5e1740 1 -- 172.21.15.46:0/1578054416 wait complete. 2024-03-08T23:12:34.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.345+0000 7f914b5e1740 20 client.16425 trim_cache size 0 max 16384 2024-03-08T23:12:34.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 Processor -- start 2024-03-08T23:12:34.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 -- start start 2024-03-08T23:12:34.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bdc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441b2a70 con 0x55cf55bdca30 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441b05b0 con 0x55cf55bd3530 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f914b5e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441ae650 con 0x55cf55bd3e50 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.358+0000 7f9142800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bdc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53082/0 (socket says 172.21.15.46:53082) 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 -- 172.21.15.46:0/2000948721 learned_addr learned my addr 172.21.15.46:0/2000948721 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 -- 172.21.15.46:0/2000948721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 msgr2=0x55cf55bd8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 -- 172.21.15.46:0/2000948721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 msgr2=0x55cf55bdc3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bdc3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 -- 172.21.15.46:0/2000948721 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cf55bcd790 con 0x55cf55bd3e50 2024-03-08T23:12:34.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f9141400640 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f91300980c0 tx=0x7f9130009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6504f022ba50c156 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f912e600640 1 -- 172.21.15.46:0/2000948721 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913009b280 con 0x55cf55bd3e50 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f912e600640 1 -- 172.21.15.46:0/2000948721 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91300d2e20 con 0x55cf55bd3e50 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.359+0000 7f912e600640 1 -- 172.21.15.46:0/2000948721 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913009b0a0 con 0x55cf55bd3e50 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 -- 172.21.15.46:0/2000948721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 msgr2=0x55cf55bdbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f91300980c0 tx=0x7f9130009f50 comp rx=0 tx=0).stop 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 -- 172.21.15.46:0/2000948721 shutdown_connections 2024-03-08T23:12:34.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55bdc3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55bdbeb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 --2- 172.21.15.46:0/2000948721 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55bd8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 -- 172.21.15.46:0/2000948721 >> 172.21.15.46:0/2000948721 conn(0x55cf55bcdc60 msgr2=0x55cf55bd8050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 -- 172.21.15.46:0/2000948721 shutdown_connections 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.360+0000 7f914b5e1740 1 -- 172.21.15.46:0/2000948721 wait complete. 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 -- start start 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 Processor -- start 2024-03-08T23:12:34.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55be3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55be2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441aec50 con 0x55cf55bdca30 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441aff50 con 0x55cf55bd3530 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91441af290 con 0x55cf55bd3e50 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f914b5e1740 10 client.0 _ll_register_callbacks cb 0x7f913c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f9142800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55be2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.361+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53090/0 (socket says 172.21.15.46:53090) 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 -- 172.21.15.46:0/3158715292 learned_addr learned my addr 172.21.15.46:0/3158715292 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141e00640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55be3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 -- 172.21.15.46:0/3158715292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 msgr2=0x55cf55be3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55be3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 -- 172.21.15.46:0/3158715292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 msgr2=0x55cf55be2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55be2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 -- 172.21.15.46:0/3158715292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cf55be2990 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9141400640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9130097ba0 tx=0x7f91300cc7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63579c528c53157e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9123400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91300b3020 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f91300d2e20 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.362+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f91300a3a50 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f913009b0a0 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 10 client.26569 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55cf55c43990 con 0x55cf55bd3e50 2024-03-08T23:12:34.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91300d2e20 con 0x55cf55bd3e50 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f913009acb0 con 0x55cf55bd3e50 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 10 client.26569 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 10 client.26569 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 20 client.26569 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55c44000 0x55cf55c46420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 4 client.26569 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55cf55b45310 con 0x55cf55c44000 2024-03-08T23:12:34.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.363+0000 7f9141e00640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55c44000 0x55cf55c46420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f9141e00640 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55c44000 0x55cf55c46420 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d56e3d969b653c6b server_cookie=f371294046d5b9fe in_seq=0 out_seq=0 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f9123400640 10 client.26569 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f9123400640 1 -- 172.21.15.46:0/3158715292 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55cf55b45310 con 0x55cf55c44000 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f9123400640 10 client.26569 handle_command_reply: tid=0 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f914b5e1740 1 client.26569 shutdown 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 msgr2=0x55cf55be4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9130097ba0 tx=0x7f91300cc7a0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 shutdown_connections 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.364+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55cf55c44000 0x55cf55c46420 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f9123400640 0 client.26569 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cf55bdca30 0x55cf55be2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cf55bd3e50 0x55cf55be4130 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 --2- 172.21.15.46:0/3158715292 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cf55bd3530 0x55cf55be3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 >> 172.21.15.46:0/3158715292 conn(0x55cf55bcdc60 msgr2=0x55cf55bcf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 shutdown_connections 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 1 -- 172.21.15.46:0/3158715292 wait complete. 2024-03-08T23:12:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f914b5e1740 20 client.26569 trim_cache size 0 max 16384 2024-03-08T23:12:34.377 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 msgr2=0x7f91280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 0x7f91280a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9134036040 tx=0x7f91340333e0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 msgr2=0x7f914419fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.365+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f913000e9c0 tx=0x7f913000ee80 comp rx=0 tx=0).stop 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 shutdown_connections 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f91280a2970 0x7f91280a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9144199ae0 0x7f9144199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f91441086b0 0x7f914419fe20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 --2- 172.21.15.46:0/3105101970 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f91441035e0 0x7f914419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 >> 172.21.15.46:0/3105101970 conn(0x7f91440fee00 msgr2=0x7f91441011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 shutdown_connections 2024-03-08T23:12:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.366+0000 7f9143200640 1 -- 172.21.15.46:0/3105101970 wait complete. 2024-03-08T23:12:34.389 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:34.389 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 -- 172.21.15.46:0/3726523819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8112270 msgr2=0x7f0fc8114660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 --2- 172.21.15.46:0/3726523819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8112270 0x7f0fc8114660 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009ca0 tx=0x7f0fc002c190 comp rx=0 tx=0).stop 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 -- 172.21.15.46:0/3726523819 shutdown_connections 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 --2- 172.21.15.46:0/3726523819 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8112270 0x7f0fc8114660 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 --2- 172.21.15.46:0/3726523819 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8073fb0 0x7f0fc8111d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 --2- 172.21.15.46:0/3726523819 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc8073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 -- 172.21.15.46:0/3726523819 >> 172.21.15.46:0/3726523819 conn(0x7f0fc80fcd50 msgr2=0x7f0fc80ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 -- 172.21.15.46:0/3726523819 shutdown_connections 2024-03-08T23:12:34.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.610+0000 7f0fd0600640 1 -- 172.21.15.46:0/3726523819 wait complete. 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.611+0000 7f0fd0600640 1 Processor -- start 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.611+0000 7f0fd0600640 1 -- start start 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 0x7f0fc819e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8112270 0x7f0fc81a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8117c70 con 0x7f0fc8073fb0 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8117af0 con 0x7f0fc8073690 2024-03-08T23:12:34.617 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fd0600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8117df0 con 0x7f0fc8112270 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcd200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 0x7f0fc819e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:50700/0 (socket says 172.21.15.46:50700) 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 -- 172.21.15.46:0/287208863 learned_addr learned my addr 172.21.15.46:0/287208863 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fce600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8112270 0x7f0fc81a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 -- 172.21.15.46:0/287208863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8112270 msgr2=0x7f0fc81a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8112270 0x7f0fc81a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.612+0000 7f0fcdc00640 1 -- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 msgr2=0x7f0fc819e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 0x7f0fc819e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fcdc00640 1 -- 172.21.15.46:0/287208863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc81a31e0 con 0x7f0fc8073690 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fcd200640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 0x7f0fc819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e9f0 tx=0x7f0fbc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fc6000640 1 -- 172.21.15.46:0/287208863 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc00cd80 con 0x7f0fc8073690 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fc6000640 1 -- 172.21.15.46:0/287208863 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fbc010910 con 0x7f0fc8073690 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc81a34d0 con 0x7f0fc8073690 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fc6000640 1 -- 172.21.15.46:0/287208863 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc018990 con 0x7f0fc8073690 2024-03-08T23:12:34.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.613+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc81aad60 con 0x7f0fc8073690 2024-03-08T23:12:34.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.616+0000 7f0fd3437740 1 Processor -- start 2024-03-08T23:12:34.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.616+0000 7f0fc6000640 1 -- 172.21.15.46:0/287208863 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0fbc010ac0 con 0x7f0fc8073690 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 -- start start 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e8310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435eac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561743543dd0 con 0x5617435df350 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617434ecc70 con 0x5617435e8850 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fd3437740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56174353fe70 con 0x5617435dfc70 2024-03-08T23:12:34.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50244/0 (socket says 172.21.15.46:50244) 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fce600640 1 -- 172.21.15.46:0/983194193 learned_addr learned my addr 172.21.15.46:0/983194193 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.617+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fcd200640 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435eac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 -- 172.21.15.46:0/983194193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 msgr2=0x5617435e8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 -- 172.21.15.46:0/983194193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 msgr2=0x5617435eac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435eac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 -- 172.21.15.46:0/983194193 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617435a62b0 con 0x5617435df350 2024-03-08T23:12:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fce600640 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0004af0 tx=0x7f0fc0005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391dfed2dc146d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.618+0000 7f0fc6000640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 0x7f0fb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fbaa00640 1 -- 172.21.15.46:0/983194193 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0043070 con 0x5617435df350 2024-03-08T23:12:34.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fc6000640 1 -- 172.21.15.46:0/287208863 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0fbc0cf500 con 0x7f0fc8073690 2024-03-08T23:12:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fbaa00640 1 -- 172.21.15.46:0/983194193 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0003e50 con 0x5617435df350 2024-03-08T23:12:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 0x7f0fb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fbaa00640 1 -- 172.21.15.46:0/983194193 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0039040 con 0x5617435df350 2024-03-08T23:12:34.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fcdc00640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 0x7f0fb00a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0cc9a0 tx=0x7f0fbc0d50e0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fd3437740 1 -- 172.21.15.46:0/983194193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 msgr2=0x5617435df730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.619+0000 7f0fd3437740 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0004af0 tx=0x7f0fc0005f80 comp rx=0 tx=0).stop 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 -- 172.21.15.46:0/983194193 shutdown_connections 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435eac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 --2- 172.21.15.46:0/983194193 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435df350 0x5617435df730 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 -- 172.21.15.46:0/983194193 >> 172.21.15.46:0/983194193 conn(0x5617435d9a80 msgr2=0x5617435d9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 -- 172.21.15.46:0/983194193 shutdown_connections 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 -- 172.21.15.46:0/983194193 wait complete. 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 -- start start 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.620+0000 7f0fd3437740 1 Processor -- start 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435e5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617435eb5e0 con 0x5617435e8850 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617435000c0 con 0x5617435df350 2024-03-08T23:12:34.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617435eb760 con 0x5617435dfc70 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fd3437740 10 client.0 _ll_register_callbacks cb 0x56174353aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435e5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fce600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fce600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:50726/0 (socket says 172.21.15.46:50726) 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fce600640 1 -- 172.21.15.46:0/1439656163 learned_addr learned my addr 172.21.15.46:0/1439656163 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.621+0000 7f0fce600640 1 -- 172.21.15.46:0/1439656163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 msgr2=0x5617435e54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fce600640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fce600640 1 -- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 msgr2=0x5617435e5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fce600640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435e5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fce600640 1 -- 172.21.15.46:0/1439656163 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617432583f0 con 0x5617435df350 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fce600640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0037040 tx=0x7f0fc002c190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1127805c2c1a81d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0faf400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0039070 con 0x5617435df350 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0fc0004250 con 0x5617435df350 2024-03-08T23:12:34.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc0003e50 con 0x5617435df350 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0043040 con 0x5617435df350 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fd3437740 10 client.26594 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.622+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5617433227d0 con 0x5617435df350 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0003e50 con 0x5617435df350 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f0fc00344b0 con 0x5617435df350 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 10 client.26594 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 10 client.26594 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:12:34.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 20 client.26594 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5617435f8680 0x5617435faa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 4 client.26594 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617435184f0 con 0x5617435f8680 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.623+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5617435f8680 0x5617435faa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.624+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5617435f8680 0x5617435faa70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3612842b1afefd7 server_cookie=9a43021f61772562 in_seq=0 out_seq=0 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.624+0000 7f0faf400640 10 client.26594 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0faf400640 1 -- 172.21.15.46:0/1439656163 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5617435184f0 con 0x5617435f8680 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0faf400640 10 client.26594 handle_command_reply: tid=0 2024-03-08T23:12:34.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 client.26594 shutdown 2024-03-08T23:12:34.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 msgr2=0x5617435e8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0037040 tx=0x7f0fc002c190 comp rx=0 tx=0).stop 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 shutdown_connections 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435e5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5617435f8680 0x5617435faa70 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435e54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1439656163 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435e8160 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 >> 172.21.15.46:0/1439656163 conn(0x5617435d9a80 msgr2=0x5617435db7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0faf400640 0 client.26594 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 shutdown_connections 2024-03-08T23:12:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 1 -- 172.21.15.46:0/1439656163 wait complete. 2024-03-08T23:12:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.625+0000 7f0fd3437740 20 client.26594 trim_cache size 0 max 16384 2024-03-08T23:12:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.638+0000 7f0fd3437740 1 Processor -- start 2024-03-08T23:12:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 -- start start 2024-03-08T23:12:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435dfc70 0x5617435e7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435e8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b7080 con 0x5617435dfc70 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b4bc0 con 0x5617435e8850 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fd3437740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b2c60 con 0x5617435df350 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fcd200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435dfc70 0x5617435e7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fce600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fce600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53138/0 (socket says 172.21.15.46:53138) 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fce600640 1 -- 172.21.15.46:0/527750848 learned_addr learned my addr 172.21.15.46:0/527750848 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.639+0000 7f0fce600640 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435e8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 -- 172.21.15.46:0/527750848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 msgr2=0x5617435e8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435e8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 -- 172.21.15.46:0/527750848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435dfc70 msgr2=0x5617435e7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435dfc70 0x5617435e7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 -- 172.21.15.46:0/527750848 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617435de440 con 0x5617435df350 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fce600640 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0004af0 tx=0x7f0fc0005d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b45c354151eaafa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.645 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fba000640 1 -- 172.21.15.46:0/527750848 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0039040 con 0x5617435df350 2024-03-08T23:12:34.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fba000640 1 -- 172.21.15.46:0/527750848 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0003d70 con 0x5617435df350 2024-03-08T23:12:34.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.640+0000 7f0fba000640 1 -- 172.21.15.46:0/527750848 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0034530 con 0x5617435df350 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 -- 172.21.15.46:0/527750848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 msgr2=0x5617435d8910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0004af0 tx=0x7f0fc0005d80 comp rx=0 tx=0).stop 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 -- 172.21.15.46:0/527750848 shutdown_connections 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435e8850 0x5617435e8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435dfc70 0x5617435e7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 --2- 172.21.15.46:0/527750848 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435df350 0x5617435d8910 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 -- 172.21.15.46:0/527750848 >> 172.21.15.46:0/527750848 conn(0x5617435d9a80 msgr2=0x5617435d8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 -- 172.21.15.46:0/527750848 shutdown_connections 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.641+0000 7f0fd3437740 1 -- 172.21.15.46:0/527750848 wait complete. 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 -- start start 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 Processor -- start 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435efef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435ee290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b3260 con 0x5617435e8850 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b4560 con 0x5617435df350 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc81b38a0 con 0x5617435dfc70 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.642+0000 7f0fd3437740 10 client.0 _ll_register_callbacks cb 0x7f0fbc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50280/0 (socket says 172.21.15.46:50280) 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 -- 172.21.15.46:0/1151017951 learned_addr learned my addr 172.21.15.46:0/1151017951 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fcd200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435ee290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435efef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 -- 172.21.15.46:0/1151017951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 msgr2=0x5617435ee290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435ee290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 -- 172.21.15.46:0/1151017951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 msgr2=0x5617435efef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435efef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 -- 172.21.15.46:0/1151017951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56174364e820 con 0x5617435e8850 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0fce600640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0002230 tx=0x7f0fc002c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c666d11af6bc9ece server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.643+0000 7f0faf400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:34.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0039030 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc0003d70 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f0fc0004690 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0032670 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 10 client.16449 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56174364f9d0 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0003d70 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f0fc0048730 con 0x5617435e8850 2024-03-08T23:12:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 10 client.16449 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 10 client.16449 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 20 client.16449 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.644+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56174364ffb0 0x5617436523d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.645+0000 7f0fd3437740 4 client.16449 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.645+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561743551310 con 0x56174364ffb0 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.645+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56174364ffb0 0x5617436523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.645+0000 7f0fcd200640 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56174364ffb0 0x5617436523d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=74b68944a5037b43 server_cookie=c3709eb21fc64e6a in_seq=0 out_seq=0 2024-03-08T23:12:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.645+0000 7f0faf400640 10 client.16449 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0faf400640 1 -- 172.21.15.46:0/1151017951 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561743551310 con 0x56174364ffb0 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0faf400640 10 client.16449 handle_command_reply: tid=0 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 client.16449 shutdown 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 msgr2=0x5617435ee9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0002230 tx=0x7f0fc002c930 comp rx=0 tx=0).stop 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 shutdown_connections 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5617435e8850 0x5617435ee9d0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56174364ffb0 0x5617436523d0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5617435dfc70 0x5617435ee290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 --2- 172.21.15.46:0/1151017951 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5617435df350 0x5617435efef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 >> 172.21.15.46:0/1151017951 conn(0x5617435d9a80 msgr2=0x5617435e0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0faf400640 0 client.16449 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 shutdown_connections 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 1 -- 172.21.15.46:0/1151017951 wait complete. 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.646+0000 7f0fd3437740 20 client.16449 trim_cache size 0 max 16384 2024-03-08T23:12:34.657 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 msgr2=0x7f0fb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 0x7f0fb00a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0cc9a0 tx=0x7f0fbc0d50e0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 msgr2=0x7f0fc819e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e9f0 tx=0x7f0fbc00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 shutdown_connections 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb00a2970 0x7f0fb00a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fc8112270 0x7f0fc81a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fc8073fb0 0x7f0fc819e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 --2- 172.21.15.46:0/287208863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fc8073690 0x7f0fc819e1f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 >> 172.21.15.46:0/287208863 conn(0x7f0fc80fcd50 msgr2=0x7f0fc80fe650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 shutdown_connections 2024-03-08T23:12:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.647+0000 7f0fd0600640 1 -- 172.21.15.46:0/287208863 wait complete. 2024-03-08T23:12:34.670 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:12:34.671 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:12:34.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.890+0000 7f7660c00640 1 -- 172.21.15.46:0/679396230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100350 msgr2=0x7f765c100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.890+0000 7f7660c00640 1 --2- 172.21.15.46:0/679396230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100350 0x7f765c100730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f764c009cd0 tx=0x7f764c02c170 comp rx=0 tx=0).stop 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 -- 172.21.15.46:0/679396230 shutdown_connections 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 --2- 172.21.15.46:0/679396230 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c105750 0x7f765c105c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 --2- 172.21.15.46:0/679396230 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c100c70 0x7f765c105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 --2- 172.21.15.46:0/679396230 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100350 0x7f765c100730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 -- 172.21.15.46:0/679396230 >> 172.21.15.46:0/679396230 conn(0x7f765c077350 msgr2=0x7f765c077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 -- 172.21.15.46:0/679396230 shutdown_connections 2024-03-08T23:12:34.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.891+0000 7f7660c00640 1 -- 172.21.15.46:0/679396230 wait complete. 2024-03-08T23:12:34.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.892+0000 7f7660c00640 1 Processor -- start 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 -- start start 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 0x7f765c19e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 0x7f765c19e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c105750 0x7f765c1a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c117d30 con 0x7f765c105750 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c117bb0 con 0x7f765c100c70 2024-03-08T23:12:34.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.893+0000 7f7660c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c117eb0 con 0x7f765c100350 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7659e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 0x7f765c19e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f765a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c105750 0x7f765c1a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 0x7f765c19e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7659e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 0x7f765c19e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53162/0 (socket says 172.21.15.46:53162) 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7659e00640 1 -- 172.21.15.46:0/1050579694 learned_addr learned my addr 172.21.15.46:0/1050579694 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 -- 172.21.15.46:0/1050579694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 msgr2=0x7f765c19e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 0x7f765c19e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 -- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c105750 msgr2=0x7f765c1a2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c105750 0x7f765c1a2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 -- 172.21.15.46:0/1050579694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c1a31c0 con 0x7f765c100c70 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.894+0000 7f7651400640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 0x7f765c19e710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9f0 tx=0x7f764400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.895+0000 7f7653400640 1 -- 172.21.15.46:0/1050579694 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764400ccc0 con 0x7f765c100c70 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.895+0000 7f7653400640 1 -- 172.21.15.46:0/1050579694 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7644010910 con 0x7f765c100c70 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.895+0000 7f7653400640 1 -- 172.21.15.46:0/1050579694 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7644018960 con 0x7f765c100c70 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.895+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c1a34b0 con 0x7f765c100c70 2024-03-08T23:12:34.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.895+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c1aad40 con 0x7f765c100c70 2024-03-08T23:12:34.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.898+0000 7f7653400640 1 -- 172.21.15.46:0/1050579694 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7644010ac0 con 0x7f765c100c70 2024-03-08T23:12:34.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.898+0000 7f7663a27740 1 Processor -- start 2024-03-08T23:12:34.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.898+0000 7f7663a27740 1 -- start start 2024-03-08T23:12:34.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dc9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dd2850 0x55a553dd4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553d2ddd0 con 0x55a553dc9c70 2024-03-08T23:12:34.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553cd6c70 con 0x55a553dd2850 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7663a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553d29e70 con 0x55a553dc9350 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50296/0 (socket says 172.21.15.46:50296) 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 -- 172.21.15.46:0/1403314547 learned_addr learned my addr 172.21.15.46:0/1403314547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f765a800640 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dc9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7651400640 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dd2850 0x55a553dd4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 -- 172.21.15.46:0/1403314547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 msgr2=0x55a553dc9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dc9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 -- 172.21.15.46:0/1403314547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dd2850 msgr2=0x55a553dd4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dd2850 0x55a553dd4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.899+0000 7f7659e00640 1 -- 172.21.15.46:0/1403314547 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a553d902b0 con 0x55a553dc9c70 2024-03-08T23:12:34.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.900+0000 7f7659e00640 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f764c009bf0 tx=0x7f764c009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efd7d430c61dc95 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.900+0000 7f764b000640 1 -- 172.21.15.46:0/1403314547 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c045020 con 0x55a553dc9c70 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.900+0000 7f764b000640 1 -- 172.21.15.46:0/1403314547 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764c03a040 con 0x55a553dc9c70 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.900+0000 7f764b000640 1 -- 172.21.15.46:0/1403314547 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c039040 con 0x55a553dc9c70 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7653400640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 0x7f76400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7653400640 1 -- 172.21.15.46:0/1050579694 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f76440cf520 con 0x7f765c100c70 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f765a800640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 0x7f76400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 -- 172.21.15.46:0/1403314547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 msgr2=0x55a553dd2310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f764c009bf0 tx=0x7f764c009880 comp rx=0 tx=0).stop 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 -- 172.21.15.46:0/1403314547 shutdown_connections 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dd2850 0x55a553dd4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd2310 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 --2- 172.21.15.46:0/1403314547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dc9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 -- 172.21.15.46:0/1403314547 >> 172.21.15.46:0/1403314547 conn(0x55a553dc3a80 msgr2=0x55a553dc3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 -- 172.21.15.46:0/1403314547 shutdown_connections 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.901+0000 7f7663a27740 1 -- 172.21.15.46:0/1403314547 wait complete. 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.902+0000 7f765a800640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 0x7f76400a4eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7654009ce0 tx=0x7f7654005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.902+0000 7f7663a27740 1 -- start start 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.902+0000 7f7663a27740 1 Processor -- start 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.902+0000 7f7663a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.902+0000 7f7663a27740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 0x55a553dcf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7663a27740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7663a27740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553dd55e0 con 0x55a553dd2850 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7663a27740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553cea0c0 con 0x55a553dc9c70 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7663a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a553dd5760 con 0x55a553dc9350 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7663a27740 10 client.0 _ll_register_callbacks cb 0x55a553d24ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50298/0 (socket says 172.21.15.46:50298) 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 -- 172.21.15.46:0/3360940958 learned_addr learned my addr 172.21.15.46:0/3360940958 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7651400640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 0x55a553dcf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7659e00640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 -- 172.21.15.46:0/3360940958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 msgr2=0x55a553dd21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dd21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 -- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 msgr2=0x55a553dcf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 0x55a553dcf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f765a800640 1 -- 172.21.15.46:0/3360940958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a553a423f0 con 0x55a553dd2850 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7659e00640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dd21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.903+0000 7f7651400640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 0x55a553dcf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f765a800640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7654004610 tx=0x7f7654004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad69dc1f4c63a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f763f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765401e470 con 0x55a553dd2850 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f765401e610 con 0x55a553dd2850 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f764c03a040 con 0x55a553dd2850 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f765401e810 con 0x55a553dd2850 2024-03-08T23:12:34.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f7663a27740 10 client.16461 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.904+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a553b0c7d0 con 0x55a553dd2850 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764c03a040 con 0x55a553dd2850 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f7654027070 con 0x55a553dd2850 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 10 client.16461 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 20 client.16461 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553de28e0 0x55a553de4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 4 client.16461 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:34.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a553d024f0 con 0x55a553de28e0 2024-03-08T23:12:34.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.905+0000 7f7659e00640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553de28e0 0x55a553de4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.906+0000 7f7659e00640 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553de28e0 0x55a553de4d20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a90c7345bc989d2e server_cookie=bee780d4a5db0d95 in_seq=0 out_seq=0 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.906+0000 7f763f400640 10 client.16461 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f763f400640 1 -- 172.21.15.46:0/3360940958 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a553d024f0 con 0x55a553de28e0 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f763f400640 10 client.16461 handle_command_reply: tid=0 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 client.16461 shutdown 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 msgr2=0x55a553dcfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f7654004610 tx=0x7f7654004730 comp rx=0 tx=0).stop 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 shutdown_connections 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553de28e0 0x55a553de4d20 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553dcfb20 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f763f400640 0 client.16461 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9c70 0x55a553dcf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 --2- 172.21.15.46:0/3360940958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9350 0x55a553dd21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 >> 172.21.15.46:0/3360940958 conn(0x55a553dc3a80 msgr2=0x55a553dc57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 shutdown_connections 2024-03-08T23:12:34.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.907+0000 7f7663a27740 1 -- 172.21.15.46:0/3360940958 wait complete. 2024-03-08T23:12:34.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.908+0000 7f7663a27740 20 client.16461 trim_cache size 0 max 16384 2024-03-08T23:12:34.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 Processor -- start 2024-03-08T23:12:34.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 -- start start 2024-03-08T23:12:34.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dd2850 0x55a553dcde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b70a0 con 0x55a553dc9c70 2024-03-08T23:12:34.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b4be0 con 0x55a553dc9350 2024-03-08T23:12:34.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7663a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b2c80 con 0x55a553dd2850 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7651400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7651400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50312/0 (socket says 172.21.15.46:50312) 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7651400640 1 -- 172.21.15.46:0/341098546 learned_addr learned my addr 172.21.15.46:0/341098546 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f7659e00640 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.921+0000 7f765a800640 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dd2850 0x55a553dcde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 -- 172.21.15.46:0/341098546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dd2850 msgr2=0x55a553dcde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dd2850 0x55a553dcde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 -- 172.21.15.46:0/341098546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 msgr2=0x55a553dd1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 -- 172.21.15.46:0/341098546 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a553dc9aa0 con 0x55a553dc9c70 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f7651400640 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f764409b570 tx=0x7f76440d5060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3733ea9f86122c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f764a600640 1 -- 172.21.15.46:0/341098546 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764409b3b0 con 0x55a553dc9c70 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f764a600640 1 -- 172.21.15.46:0/341098546 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764409b870 con 0x55a553dc9c70 2024-03-08T23:12:34.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.922+0000 7f764a600640 1 -- 172.21.15.46:0/341098546 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764409b1d0 con 0x55a553dc9c70 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- 172.21.15.46:0/341098546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 msgr2=0x55a553dd21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f764409b570 tx=0x7f76440d5060 comp rx=0 tx=0).stop 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- 172.21.15.46:0/341098546 shutdown_connections 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dd2850 0x55a553dcde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dc9c70 0x55a553dd21f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 --2- 172.21.15.46:0/341098546 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- 172.21.15.46:0/341098546 >> 172.21.15.46:0/341098546 conn(0x55a553dc3a80 msgr2=0x55a553dcf540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- 172.21.15.46:0/341098546 shutdown_connections 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- 172.21.15.46:0/341098546 wait complete. 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.923+0000 7f7663a27740 1 -- start start 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 Processor -- start 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9c70 0x55a553dd7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b3280 con 0x55a553dd2850 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b4580 con 0x55a553dc9350 2024-03-08T23:12:34.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1b38c0 con 0x55a553dc9c70 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7663a27740 10 client.0 _ll_register_callbacks cb 0x7f7644005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f765a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f765a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50320/0 (socket says 172.21.15.46:50320) 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f765a800640 1 -- 172.21.15.46:0/1992259610 learned_addr learned my addr 172.21.15.46:0/1992259610 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.924+0000 7f7651400640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9c70 0x55a553dd7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f7659e00640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 -- 172.21.15.46:0/1992259610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9c70 msgr2=0x55a553dd7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9c70 0x55a553dd7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 -- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 msgr2=0x55a553dd7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 -- 172.21.15.46:0/1992259610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a553e38340 con 0x55a553dd2850 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f765a800640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7654012cb0 tx=0x7f76540043d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6eb05dfa3f3d9ec server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f763f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f765401ec80 con 0x55a553dd2850 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f764409b870 con 0x55a553dd2850 2024-03-08T23:12:34.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.925+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f7654029440 con 0x55a553dd2850 2024-03-08T23:12:34.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f765401ee20 con 0x55a553dd2850 2024-03-08T23:12:34.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 10 client.16467 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:34.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a553e395e0 con 0x55a553dd2850 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764409b870 con 0x55a553dd2850 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f765403f030 con 0x55a553dd2850 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 10 client.16467 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 20 client.16467 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553e39d00 0x55a553e3c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 4 client.16467 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:12:34.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.926+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a553d3b310 con 0x55a553e39d00 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f7659e00640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553e39d00 0x55a553e3c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f7659e00640 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553e39d00 0x55a553e3c120 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da06b439f42d36e5 server_cookie=282c95586d502197 in_seq=0 out_seq=0 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f763f400640 10 client.16467 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f763f400640 1 -- 172.21.15.46:0/1992259610 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a553d3b310 con 0x55a553e39d00 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f763f400640 10 client.16467 handle_command_reply: tid=0 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f7663a27740 1 client.16467 shutdown 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 msgr2=0x55a553e37e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.927+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7654012cb0 tx=0x7f76540043d0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 shutdown_connections 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a553e39d00 0x55a553e3c120 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a553dd2850 0x55a553e37e00 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a553dc9c70 0x55a553dd7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 --2- 172.21.15.46:0/1992259610 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a553dc9350 0x55a553dd7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f763f400640 0 client.16467 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 >> 172.21.15.46:0/1992259610 conn(0x55a553dc3a80 msgr2=0x55a553dc4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 shutdown_connections 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 1 -- 172.21.15.46:0/1992259610 wait complete. 2024-03-08T23:12:34.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7663a27740 20 client.16467 trim_cache size 0 max 16384 2024-03-08T23:12:34.940 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 msgr2=0x7f76400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.928+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 0x7f76400a4eb0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7654009ce0 tx=0x7f7654005f90 comp rx=0 tx=0).stop 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 msgr2=0x7f765c19e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 0x7f765c19e710 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f764400e9f0 tx=0x7f764400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 shutdown_connections 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f76400a29f0 0x7f76400a4eb0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f765c105750 0x7f765c1a2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f765c100c70 0x7f765c19e710 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 --2- 172.21.15.46:0/1050579694 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f765c100350 0x7f765c19e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 >> 172.21.15.46:0/1050579694 conn(0x7f765c077350 msgr2=0x7f765c103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 shutdown_connections 2024-03-08T23:12:34.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:34.929+0000 7f7660c00640 1 -- 172.21.15.46:0/1050579694 wait complete. 2024-03-08T23:12:34.948 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:34 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/417454851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:35.042 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:34 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/417454851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:35.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:34 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/417454851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:12:35.952 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:12:35.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:12:35.953 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:12:35.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:35 smithi046 ceph-mon[31859]: pgmap v147: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:36.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:35 smithi192 ceph-mon[36700]: pgmap v147: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:36.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:35 smithi062 ceph-mon[35761]: pgmap v147: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 -- 172.21.15.46:0/760955050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa40410b750 msgr2=0x7fa40410dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 --2- 172.21.15.46:0/760955050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa40410b750 0x7fa40410dbd0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc009ca0 tx=0x7fa3fc02c190 comp rx=0 tx=0).stop 2024-03-08T23:12:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 -- 172.21.15.46:0/760955050 shutdown_connections 2024-03-08T23:12:36.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 --2- 172.21.15.46:0/760955050 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa40410b750 0x7fa40410dbd0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 --2- 172.21.15.46:0/760955050 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102ca0 0x7fa40410b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 --2- 172.21.15.46:0/760955050 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa404102380 0x7fa404102760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 -- 172.21.15.46:0/760955050 >> 172.21.15.46:0/760955050 conn(0x7fa4040fbb40 msgr2=0x7fa4040fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 -- 172.21.15.46:0/760955050 shutdown_connections 2024-03-08T23:12:36.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.191+0000 7fa409a00640 1 -- 172.21.15.46:0/760955050 wait complete. 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.192+0000 7fa409a00640 1 Processor -- start 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 -- start start 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa404102ca0 0x7fa40419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa40410b750 0x7fa404197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041114f0 con 0x7fa404102ca0 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa404111370 con 0x7fa40410b750 2024-03-08T23:12:36.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa409a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa404111670 con 0x7fa404102380 2024-03-08T23:12:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa403e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa404102ca0 0x7fa40419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa403e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53212/0 (socket says 172.21.15.46:53212) 2024-03-08T23:12:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.193+0000 7fa403e00640 1 -- 172.21.15.46:0/2619880618 learned_addr learned my addr 172.21.15.46:0/2619880618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa409000640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa40410b750 0x7fa404197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 -- 172.21.15.46:0/2619880618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa40410b750 msgr2=0x7fa404197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa40410b750 0x7fa404197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 -- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa404102ca0 msgr2=0x7fa40419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa404102ca0 0x7fa40419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 -- 172.21.15.46:0/2619880618 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa404198040 con 0x7fa404102380 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.194+0000 7fa403e00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9f0 tx=0x7fa3f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.195+0000 7fa400c00640 1 -- 172.21.15.46:0/2619880618 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f400cd80 con 0x7fa404102380 2024-03-08T23:12:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.195+0000 7fa400c00640 1 -- 172.21.15.46:0/2619880618 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f4010910 con 0x7fa404102380 2024-03-08T23:12:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.195+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa404198330 con 0x7fa404102380 2024-03-08T23:12:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.195+0000 7fa400c00640 1 -- 172.21.15.46:0/2619880618 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4018990 con 0x7fa404102380 2024-03-08T23:12:36.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.195+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4041a4500 con 0x7fa404102380 2024-03-08T23:12:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 Processor -- start 2024-03-08T23:12:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 -- start start 2024-03-08T23:12:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa400c00640 1 -- 172.21.15.46:0/2619880618 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa3f4010ac0 con 0x7fa404102380 2024-03-08T23:12:36.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd4db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd56760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bcb1a50 con 0x55d52bd56ca0 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bc5ab00 con 0x55d52bd4d7a0 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.198+0000 7fa40dddf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bcadaf0 con 0x55d52bd4e0c0 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd56760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50348/0 (socket says 172.21.15.46:50348) 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 -- 172.21.15.46:0/1776108663 learned_addr learned my addr 172.21.15.46:0/1776108663 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa409000640 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd4db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 -- 172.21.15.46:0/1776108663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 msgr2=0x55d52bd56760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd56760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 -- 172.21.15.46:0/1776108663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 msgr2=0x55d52bd4db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd4db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 -- 172.21.15.46:0/1776108663 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d52bc781e0 con 0x55d52bd56ca0 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa409000640 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.199+0000 7fa403400640 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009c20 tx=0x7fa3f0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a7a8ee6dea2e7d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa3f9600640 1 -- 172.21.15.46:0/1776108663 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f001e070 con 0x55d52bd56ca0 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa3f9600640 1 -- 172.21.15.46:0/1776108663 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f0009dd0 con 0x55d52bd56ca0 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa3f9600640 1 -- 172.21.15.46:0/1776108663 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0018730 con 0x55d52bd56ca0 2024-03-08T23:12:36.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa400c00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 0x7fa3e80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa400c00640 1 -- 172.21.15.46:0/2619880618 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa3f40cf480 con 0x7fa404102380 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.200+0000 7fa409000640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 0x7fa3e80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 -- 172.21.15.46:0/1776108663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 msgr2=0x55d52bd59090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009c20 tx=0x7fa3f0011810 comp rx=0 tx=0).stop 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 -- 172.21.15.46:0/1776108663 shutdown_connections 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd56ca0 0x55d52bd59090 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd56760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 --2- 172.21.15.46:0/1776108663 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd4db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 -- 172.21.15.46:0/1776108663 >> 172.21.15.46:0/1776108663 conn(0x55d52bd47ed0 msgr2=0x55d52bd482b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 -- 172.21.15.46:0/1776108663 shutdown_connections 2024-03-08T23:12:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa409000640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 0x7fa3e80a4eb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc004af0 tx=0x7fa3fc005c50 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.201+0000 7fa40dddf740 1 -- 172.21.15.46:0/1776108663 wait complete. 2024-03-08T23:12:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 -- start start 2024-03-08T23:12:36.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 Processor -- start 2024-03-08T23:12:36.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd54180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bd546c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bd59a90 con 0x55d52bd4e0c0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bc6df60 con 0x55d52bd56ca0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d52bd59c10 con 0x55d52bd4d7a0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.202+0000 7fa40dddf740 10 client.0 _ll_register_callbacks cb 0x55d52bc59b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd54180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53228/0 (socket says 172.21.15.46:53228) 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 -- 172.21.15.46:0/2203044977 learned_addr learned my addr 172.21.15.46:0/2203044977 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa409000640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bd546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 -- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 msgr2=0x55d52bd546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bd546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 -- 172.21.15.46:0/2203044977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 msgr2=0x55d52bd54180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd54180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 -- 172.21.15.46:0/2203044977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d52ba8e7e0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403400640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd54180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa403e00640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9c0 tx=0x7fa3f40974d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afcb65e865f43038 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.203+0000 7fa3ee000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f409b370 con 0x55d52bd4d7a0 2024-03-08T23:12:36.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa3f40ccaf0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3f0009dd0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f409b190 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 10 client.26617 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d52baeef70 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f0009dd0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa3f40cf110 con 0x55d52bd4d7a0 2024-03-08T23:12:36.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 10 client.26617 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 10 client.26617 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 20 client.26617 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bd66ed0 0x55d52bd692c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 4 client.26617 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.204+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d52bcbef90 con 0x55d52bd66ed0 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.205+0000 7fa403400640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bd66ed0 0x55d52bd692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.205+0000 7fa403400640 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bd66ed0 0x55d52bd692c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43bc30c0e159e538 server_cookie=b37850d8f2292f02 in_seq=0 out_seq=0 2024-03-08T23:12:36.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.205+0000 7fa3ee000640 10 client.26617 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa3ee000640 1 -- 172.21.15.46:0/2203044977 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d52bcbef90 con 0x55d52bd66ed0 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa3ee000640 10 client.26617 handle_command_reply: tid=0 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 client.26617 shutdown 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 msgr2=0x55d52bd53c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9c0 tx=0x7fa3f40974d0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 shutdown_connections 2024-03-08T23:12:36.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bd66ed0 0x55d52bd692c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bd546c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd54180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2203044977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4d7a0 0x55d52bd53c40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa3ee000640 0 client.26617 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.206+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 >> 172.21.15.46:0/2203044977 conn(0x55d52bd47ed0 msgr2=0x55d52bd49c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.207+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 shutdown_connections 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.207+0000 7fa40dddf740 1 -- 172.21.15.46:0/2203044977 wait complete. 2024-03-08T23:12:36.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.207+0000 7fa40dddf740 20 client.26617 trim_cache size 0 max 16384 2024-03-08T23:12:36.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 Processor -- start 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 -- start start 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd52c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 0x55d52bd51400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041ae710 con 0x55d52bd4e0c0 2024-03-08T23:12:36.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041ac250 con 0x55d52bd4d7a0 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa40dddf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041aa2f0 con 0x55d52bd56ca0 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.224+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50370/0 (socket says 172.21.15.46:50370) 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 -- 172.21.15.46:0/2935604384 learned_addr learned my addr 172.21.15.46:0/2935604384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa409000640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 0x55d52bd51400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403e00640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd52c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 -- 172.21.15.46:0/2935604384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 msgr2=0x55d52bd51400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 0x55d52bd51400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 -- 172.21.15.46:0/2935604384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 msgr2=0x55d52bd52c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd52c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 -- 172.21.15.46:0/2935604384 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d52bd4def0 con 0x55d52bd4e0c0 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403e00640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd52c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa409000640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 0x55d52bd51400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.225+0000 7fa403400640 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009c20 tx=0x7fa3f0005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ef843407e483d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa3f8c00640 1 -- 172.21.15.46:0/2935604384 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f001e040 con 0x55d52bd4e0c0 2024-03-08T23:12:36.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa3f8c00640 1 -- 172.21.15.46:0/2935604384 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f002f070 con 0x55d52bd4e0c0 2024-03-08T23:12:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa3f8c00640 1 -- 172.21.15.46:0/2935604384 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f0019670 con 0x55d52bd4e0c0 2024-03-08T23:12:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 -- 172.21.15.46:0/2935604384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 msgr2=0x55d52bd50ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0009c20 tx=0x7fa3f0005f10 comp rx=0 tx=0).stop 2024-03-08T23:12:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 -- 172.21.15.46:0/2935604384 shutdown_connections 2024-03-08T23:12:36.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd56ca0 0x55d52bd51400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4e0c0 0x55d52bd50ec0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 --2- 172.21.15.46:0/2935604384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd4d7a0 0x55d52bd52c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 -- 172.21.15.46:0/2935604384 >> 172.21.15.46:0/2935604384 conn(0x55d52bd47ed0 msgr2=0x55d52bd52510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 -- 172.21.15.46:0/2935604384 shutdown_connections 2024-03-08T23:12:36.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.226+0000 7fa40dddf740 1 -- 172.21.15.46:0/2935604384 wait complete. 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.227+0000 7fa40dddf740 1 -- start start 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.227+0000 7fa40dddf740 1 Processor -- start 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd5d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bdbd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041aa8f0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041abbf0 con 0x55d52bd56ca0 2024-03-08T23:12:36.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4041aaf30 con 0x55d52bd4e0c0 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa40dddf740 10 client.0 _ll_register_callbacks cb 0x7fa3f0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50374/0 (socket says 172.21.15.46:50374) 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 -- 172.21.15.46:0/3880984470 learned_addr learned my addr 172.21.15.46:0/3880984470 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa409000640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bdbd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403400640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd5d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 -- 172.21.15.46:0/3880984470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 msgr2=0x55d52bd5d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd5d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 -- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 msgr2=0x55d52bdbd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bdbd3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.228+0000 7fa403e00640 1 -- 172.21.15.46:0/3880984470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d52b986170 con 0x55d52bd4d7a0 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa403e00640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40981b0 tx=0x7fa3f4005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bbee0c9f25faaa server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f409b370 con 0x55d52bd4d7a0 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa3f40ccaf0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa3f002f070 con 0x55d52bd4d7a0 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3f40d92d0 con 0x55d52bd4d7a0 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa40dddf740 10 client.16485 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55d52bc7d050 con 0x55d52bd4d7a0 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3f002f070 con 0x55d52bd4d7a0 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.229+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa3f40afb60 con 0x55d52bd4d7a0 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 10 client.16485 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:36.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 20 client.16485 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bdbefe0 0x55d52bdc1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 4 client.16485 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:12:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d52bcb92e0 con 0x55d52bdbefe0 2024-03-08T23:12:36.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa403400640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bdbefe0 0x55d52bdc1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa403400640 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bdbefe0 0x55d52bdc1400 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7dc1c526821c4d8 server_cookie=b2d1ab85fb63eb59 in_seq=0 out_seq=0 2024-03-08T23:12:36.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.230+0000 7fa3ee000640 10 client.16485 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa3ee000640 1 -- 172.21.15.46:0/3880984470 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d52bcb92e0 con 0x55d52bdbefe0 2024-03-08T23:12:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa3ee000640 10 client.16485 handle_command_reply: tid=0 2024-03-08T23:12:36.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 client.16485 shutdown 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 msgr2=0x55d52bd5cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40981b0 tx=0x7fa3f4005b20 comp rx=0 tx=0).stop 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 shutdown_connections 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d52bdbefe0 0x55d52bdc1400 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d52bd56ca0 0x55d52bdbd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d52bd4e0c0 0x55d52bd5d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 --2- 172.21.15.46:0/3880984470 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d52bd4d7a0 0x55d52bd5cfd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 >> 172.21.15.46:0/3880984470 conn(0x55d52bd47ed0 msgr2=0x55d52bd52da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa3ee000640 0 client.16485 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 shutdown_connections 2024-03-08T23:12:36.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 1 -- 172.21.15.46:0/3880984470 wait complete. 2024-03-08T23:12:36.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.232+0000 7fa40dddf740 20 client.16485 trim_cache size 0 max 16384 2024-03-08T23:12:36.247 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:36.248 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:12:36.248 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "e86cddce-2b0c-49bc-872e-bd92a4e2a3de", 2024-03-08T23:12:36.248 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:12:36.248 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 msgr2=0x7fa3e80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 0x7fa3e80a4eb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa3fc004af0 tx=0x7fa3fc005c50 comp rx=0 tx=0).stop 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 msgr2=0x7fa40419d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa3f400e9f0 tx=0x7fa3f400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 shutdown_connections 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa3e80a29f0 0x7fa3e80a4eb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa40410b750 0x7fa404197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa404102ca0 0x7fa40419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 --2- 172.21.15.46:0/2619880618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa404102380 0x7fa40419d6b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 >> 172.21.15.46:0/2619880618 conn(0x7fa4040fbb40 msgr2=0x7fa4040fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 shutdown_connections 2024-03-08T23:12:36.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.233+0000 7fa409a00640 1 -- 172.21.15.46:0/2619880618 wait complete. 2024-03-08T23:12:36.257 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:12:36.258 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:12:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 -- 172.21.15.46:0/3998131585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073690 msgr2=0x7fa4ec073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/3998131585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073690 0x7fa4ec073a70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0009900 tx=0x7fa4e002bcd0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 -- 172.21.15.46:0/3998131585 shutdown_connections 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/3998131585 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec112170 0x7fa4ec114560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/3998131585 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec111c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/3998131585 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073690 0x7fa4ec073a70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.570+0000 7fa4f2c00640 1 -- 172.21.15.46:0/3998131585 >> 172.21.15.46:0/3998131585 conn(0x7fa4ec0fcc50 msgr2=0x7fa4ec0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.571+0000 7fa4f2c00640 1 -- 172.21.15.46:0/3998131585 shutdown_connections 2024-03-08T23:12:36.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.571+0000 7fa4f2c00640 1 -- 172.21.15.46:0/3998131585 wait complete. 2024-03-08T23:12:36.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 Processor -- start 2024-03-08T23:12:36.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 -- start start 2024-03-08T23:12:36.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec112170 0x7fa4ec1a2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec117bc0 con 0x7fa4ec112170 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec117a40 con 0x7fa4ec073690 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.572+0000 7fa4f2c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec117d40 con 0x7fa4ec073fb0 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:50862/0 (socket says 172.21.15.46:50862) 2024-03-08T23:12:36.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 -- 172.21.15.46:0/2995310222 learned_addr learned my addr 172.21.15.46:0/2995310222 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f2200640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec112170 0x7fa4ec1a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 -- 172.21.15.46:0/2995310222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 msgr2=0x7fa4ec19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 -- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec112170 msgr2=0x7fa4ec1a2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec112170 0x7fa4ec1a2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f1800640 1 -- 172.21.15.46:0/2995310222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec1a3070 con 0x7fa4ec073690 2024-03-08T23:12:36.579 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.573+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:36.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4f1800640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000c00 tx=0x7fa4e00043c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4ea000640 1 -- 172.21.15.46:0/2995310222 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0042020 con 0x7fa4ec073690 2024-03-08T23:12:36.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4ea000640 1 -- 172.21.15.46:0/2995310222 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e00054e0 con 0x7fa4ec073690 2024-03-08T23:12:36.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ec1a3300 con 0x7fa4ec073690 2024-03-08T23:12:36.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4ea000640 1 -- 172.21.15.46:0/2995310222 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e0003b20 con 0x7fa4ec073690 2024-03-08T23:12:36.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.574+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ec1aabf0 con 0x7fa4ec073690 2024-03-08T23:12:36.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.579+0000 7fa4f7039740 1 Processor -- start 2024-03-08T23:12:36.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.579+0000 7fa4f7039740 1 -- start start 2024-03-08T23:12:36.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.579+0000 7fa4ea000640 1 -- 172.21.15.46:0/2995310222 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa4e0045020 con 0x7fa4ec073690 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fbdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fc6c90 0x55fee2fc9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2f7f410 con 0x55fee2fbd790 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2f21d50 con 0x55fee2fc6c90 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f7039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2ecab80 con 0x55fee2fbe0b0 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f2200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fbdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53260/0 (socket says 172.21.15.46:53260) 2024-03-08T23:12:36.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.580+0000 7fa4f1800640 1 -- 172.21.15.46:0/1816436544 learned_addr learned my addr 172.21.15.46:0/1816436544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fc6c90 0x55fee2fc9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f1800640 1 -- 172.21.15.46:0/1816436544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fc6c90 msgr2=0x55fee2fc9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fc6c90 0x55fee2fc9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f1800640 1 -- 172.21.15.46:0/1816436544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 msgr2=0x55fee2fbdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fbdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.581+0000 7fa4f1800640 1 -- 172.21.15.46:0/1816436544 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fee2d007d0 con 0x55fee2fbe0b0 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.582+0000 7fa4ea000640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 0x7fa4d40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.582+0000 7fa4ea000640 1 -- 172.21.15.46:0/2995310222 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa4e00f0e90 con 0x7fa4ec073690 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.582+0000 7fa4f2200640 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fbdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:12:36.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.582+0000 7fa4f2200640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 0x7fa4d40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.583+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00ee270 tx=0x7fa4e002bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a900c852c0bed9e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.583+0000 7fa4dea00640 1 -- 172.21.15.46:0/1816436544 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00b95a0 con 0x55fee2fbe0b0 2024-03-08T23:12:36.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.583+0000 7fa4dea00640 1 -- 172.21.15.46:0/1816436544 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e00bda00 con 0x55fee2fbe0b0 2024-03-08T23:12:36.588 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.583+0000 7fa4dea00640 1 -- 172.21.15.46:0/1816436544 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00bcc60 con 0x55fee2fbe0b0 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 -- 172.21.15.46:0/1816436544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 msgr2=0x55fee2fc6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f2200640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 0x7fa4d40a4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40092f0 tx=0x7fa4e4009280 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa4e00ee270 tx=0x7fa4e002bcd0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 -- 172.21.15.46:0/1816436544 shutdown_connections 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fc6c90 0x55fee2fc9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc6750 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1816436544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fbdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 -- 172.21.15.46:0/1816436544 >> 172.21.15.46:0/1816436544 conn(0x55fee2fb7ec0 msgr2=0x55fee2fb82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 -- 172.21.15.46:0/1816436544 shutdown_connections 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.584+0000 7fa4f7039740 1 -- 172.21.15.46:0/1816436544 wait complete. 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.585+0000 7fa4f7039740 1 -- start start 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.585+0000 7fa4f7039740 1 Processor -- start 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc4200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fc4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2eddee0 con 0x55fee2fbd790 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2f1ddf0 con 0x55fee2fbe0b0 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fee2fc9840 con 0x55fee2fc6c90 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f7039740 10 client.0 _ll_register_callbacks cb 0x55fee2ee48d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50422/0 (socket says 172.21.15.46:50422) 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f1800640 1 -- 172.21.15.46:0/1737607800 learned_addr learned my addr 172.21.15.46:0/1737607800 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.586+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f2200640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fc4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f1800640 1 -- 172.21.15.46:0/1737607800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 msgr2=0x55fee2fc4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fc4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f1800640 1 -- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 msgr2=0x55fee2fc4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.587+0000 7fa4f1800640 1 -- 172.21.15.46:0/1737607800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fee2c363f0 con 0x55fee2fbd790 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4f1800640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005370 tx=0x7fa4e00f44d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ced3f846415db2f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4d3400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:12:36.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00bce40 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa4e0003cc0 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4e00bda00 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4e00bd0d0 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4f7039740 10 client.16497 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.588+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fee2bf60f0 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e00bda00 con 0x55fee2fbd790 2024-03-08T23:12:36.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa4e002cc50 con 0x55fee2fbd790 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4f7039740 10 client.16497 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4f7039740 10 client.16497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4f7039740 20 client.16497 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.589+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee2fd6a50 0x55fee2fd8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.590+0000 7fa4f7039740 4 client.16497 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.590+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fee2f2f290 con 0x55fee2fd6a50 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.590+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee2fd6a50 0x55fee2fd8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.591+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee2fd6a50 0x55fee2fd8e40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ecc3f52ab73914d server_cookie=cc799d6556b8a44 in_seq=0 out_seq=0 2024-03-08T23:12:36.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.591+0000 7fa4d3400640 10 client.16497 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4d3400640 1 -- 172.21.15.46:0/1737607800 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fee2f2f290 con 0x55fee2fd6a50 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4d3400640 10 client.16497 handle_command_reply: tid=0 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 client.16497 shutdown 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 msgr2=0x55fee2fc3cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005370 tx=0x7fa4e00f44d0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 shutdown_connections 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee2fd6a50 0x55fee2fd8e40 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fc4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fc4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 --2- 172.21.15.46:0/1737607800 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc3cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 >> 172.21.15.46:0/1737607800 conn(0x55fee2fb7ec0 msgr2=0x55fee2fb9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.592+0000 7fa4d3400640 0 client.16497 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.593+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 shutdown_connections 2024-03-08T23:12:36.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.593+0000 7fa4f7039740 1 -- 172.21.15.46:0/1737607800 wait complete. 2024-03-08T23:12:36.601 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.593+0000 7fa4f7039740 20 client.16497 trim_cache size 0 max 16384 2024-03-08T23:12:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 Processor -- start 2024-03-08T23:12:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 -- start start 2024-03-08T23:12:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fcd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fcde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b6ec0 con 0x55fee2fbd790 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b4a00 con 0x55fee2fbe0b0 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f7039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b2aa0 con 0x55fee2fc6c90 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f2200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fcde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.613+0000 7fa4f1800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50428/0 (socket says 172.21.15.46:50428) 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 -- 172.21.15.46:0/72952016 learned_addr learned my addr 172.21.15.46:0/72952016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f2200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fcde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53268/0 (socket says 172.21.15.46:53268) 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fcd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 -- 172.21.15.46:0/72952016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 msgr2=0x55fee2fcde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fcde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 -- 172.21.15.46:0/72952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 msgr2=0x55fee2fcd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fcd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 -- 172.21.15.46:0/72952016 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fee2fdde00 con 0x55fee2fbd790 2024-03-08T23:12:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fcd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:12:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4f1800640 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005ab0 tx=0x7fa4e00dd2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82689d163aa1fbab server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.614+0000 7fa4de000640 1 -- 172.21.15.46:0/72952016 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00bce40 con 0x55fee2fbd790 2024-03-08T23:12:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4de000640 1 -- 172.21.15.46:0/72952016 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e00ac020 con 0x55fee2fbd790 2024-03-08T23:12:36.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4de000640 1 -- 172.21.15.46:0/72952016 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e00b95a0 con 0x55fee2fbd790 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 -- 172.21.15.46:0/72952016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 msgr2=0x55fee2fc63f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0005ab0 tx=0x7fa4e00dd2c0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 -- 172.21.15.46:0/72952016 shutdown_connections 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee2fcde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee2fcd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 --2- 172.21.15.46:0/72952016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee2fc63f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.615+0000 7fa4f7039740 1 -- 172.21.15.46:0/72952016 >> 172.21.15.46:0/72952016 conn(0x55fee2fb7ec0 msgr2=0x55fee2fbefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.616+0000 7fa4f7039740 1 -- 172.21.15.46:0/72952016 shutdown_connections 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.616+0000 7fa4f7039740 1 -- 172.21.15.46:0/72952016 wait complete. 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.616+0000 7fa4f7039740 1 -- start start 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.616+0000 7fa4f7039740 1 Processor -- start 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee3020e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee3025d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b30a0 con 0x55fee2fbd790 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b43a0 con 0x55fee2fbe0b0 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ec1b36e0 con 0x55fee2fc6c90 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f7039740 10 client.0 _ll_register_callbacks cb 0x7fa4e00f4810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:50876/0 (socket says 172.21.15.46:50876) 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 -- 172.21.15.46:0/3718556987 learned_addr learned my addr 172.21.15.46:0/3718556987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:12:36.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f1800640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee3020e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f2200640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee3025d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 -- 172.21.15.46:0/3718556987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 msgr2=0x55fee3025d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee3025d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 -- 172.21.15.46:0/3718556987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 msgr2=0x55fee3020e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee3020e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.617+0000 7fa4f0e00640 1 -- 172.21.15.46:0/3718556987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fee3026300 con 0x55fee2fbe0b0 2024-03-08T23:12:36.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4f0e00640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55fee2fb9050 tx=0x7fa4d80039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e59f142ac772248 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4d3400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d8025020 con 0x55fee2fbe0b0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4e00ac020 con 0x55fee2fbe0b0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa4d80212b0 con 0x55fee2fbe0b0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4d8030050 con 0x55fee2fbe0b0 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4f7039740 10 client.26654 fetch_fsmap learned FSMap version 27 2024-03-08T23:12:36.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55fee302d750 con 0x55fee2fbe0b0 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.618+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e00ac020 con 0x55fee2fbe0b0 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 27) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa4d8010430 con 0x55fee2fbe0b0 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 10 client.26654 fetch_fsmap finished waiting for FSMap version 27 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 10 client.26654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 20 client.26654 populate_metadata read hostname 'smithi046' 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee302df10 0x55fee3030330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:12:36.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 4 client.26654 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fee2f295e0 con 0x55fee302df10 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f1800640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee302df10 0x55fee3030330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4f1800640 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee302df10 0x55fee3030330 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8e7b1b3631230 server_cookie=12e6e53f9e0e17e5 in_seq=0 out_seq=0 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.619+0000 7fa4d3400640 10 client.26654 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4d3400640 1 -- 172.21.15.46:0/3718556987 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55fee2f295e0 con 0x55fee302df10 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4d3400640 10 client.26654 handle_command_reply: tid=0 2024-03-08T23:12:36.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 client.26654 shutdown 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 msgr2=0x55fee30213b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x55fee2fb9050 tx=0x7fa4d80039c0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 shutdown_connections 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55fee302df10 0x55fee3030330 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55fee2fc6c90 0x55fee3025d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55fee2fbe0b0 0x55fee30213b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 --2- 172.21.15.46:0/3718556987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55fee2fbd790 0x55fee3020e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 >> 172.21.15.46:0/3718556987 conn(0x55fee2fb7ec0 msgr2=0x55fee2fc2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4d3400640 0 client.26654 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:12:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 shutdown_connections 2024-03-08T23:12:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 1 -- 172.21.15.46:0/3718556987 wait complete. 2024-03-08T23:12:36.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.620+0000 7fa4f7039740 20 client.26654 trim_cache size 0 max 16384 2024-03-08T23:12:36.634 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "e86cddce-2b0c-49bc-872e-bd92a4e2a3de": { 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "e86cddce-2b0c-49bc-872e-bd92a4e2a3de", 2024-03-08T23:12:36.635 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:12:36.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:12:36.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:12:36.636 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 msgr2=0x7fa4d40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 0x7fa4d40a4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa4e40092f0 tx=0x7fa4e4009280 comp rx=0 tx=0).stop 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 msgr2=0x7fa4ec19e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fa4e0000c00 tx=0x7fa4e00043c0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 shutdown_connections 2024-03-08T23:12:36.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa4d40a29f0 0x7fa4d40a4eb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa4ec112170 0x7fa4ec1a2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa4ec073fb0 0x7fa4ec19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 --2- 172.21.15.46:0/2995310222 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa4ec073690 0x7fa4ec19e080 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 >> 172.21.15.46:0/2995310222 conn(0x7fa4ec0fcc50 msgr2=0x7fa4ec0fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 shutdown_connections 2024-03-08T23:12:36.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:12:36.621+0000 7fa4f2c00640 1 -- 172.21.15.46:0/2995310222 wait complete. 2024-03-08T23:12:36.642 INFO:tasks.cephfs.filesystem:scrub status for tag:e86cddce-2b0c-49bc-872e-bd92a4e2a3de - {'path': '/', 'tag': 'e86cddce-2b0c-49bc-872e-bd92a4e2a3de', 'options': 'recursive,force'} 2024-03-08T23:12:37.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:12:37.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:37.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: pgmap v148: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:37.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: pgmap v148: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:38.064 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:38.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: pgmap v148: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:38.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:12:39.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:39 smithi046 ceph-mon[31859]: pgmap v149: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:40.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:39 smithi192 ceph-mon[36700]: pgmap v149: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:40.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:39 smithi062 ceph-mon[35761]: pgmap v149: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:41.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:41 smithi046 ceph-mon[31859]: pgmap v150: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:42.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:41 smithi192 ceph-mon[36700]: pgmap v150: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:42.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:41 smithi062 ceph-mon[35761]: pgmap v150: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:43.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:43 smithi046 ceph-mon[31859]: pgmap v151: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:44.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:43 smithi192 ceph-mon[36700]: pgmap v151: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:43 smithi062 ceph-mon[35761]: pgmap v151: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:45.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:45 smithi046 ceph-mon[31859]: pgmap v152: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:46.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:45 smithi192 ceph-mon[36700]: pgmap v152: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:46.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:45 smithi062 ceph-mon[35761]: pgmap v152: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:48.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:47 smithi192 ceph-mon[36700]: pgmap v153: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:48.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:48.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:48.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:47 smithi062 ceph-mon[35761]: pgmap v153: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:48.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:48.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:48.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:47 smithi046 ceph-mon[31859]: pgmap v153: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:12:48.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:48.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:49.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:48 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:12:49.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:48 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:12:49.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:48 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:12:50.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:49 smithi192 ceph-mon[36700]: pgmap v154: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:50.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:49 smithi062 ceph-mon[35761]: pgmap v154: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:50.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:49 smithi046 ceph-mon[31859]: pgmap v154: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:12:52.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:51 smithi192 ceph-mon[36700]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:52.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:51 smithi062 ceph-mon[35761]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:52.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:51 smithi046 ceph-mon[31859]: pgmap v155: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:54.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:53 smithi192 ceph-mon[36700]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:54.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:53 smithi062 ceph-mon[35761]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:54.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:53 smithi046 ceph-mon[31859]: pgmap v156: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:12:56.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:55 smithi192 ceph-mon[36700]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:56.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:55 smithi062 ceph-mon[35761]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:56.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:55 smithi046 ceph-mon[31859]: pgmap v157: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:58.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:57 smithi192 ceph-mon[36700]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:58.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:57 smithi062 ceph-mon[35761]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:58.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:57 smithi046 ceph-mon[31859]: pgmap v158: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:12:59.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:58 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:12:59.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:58 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:12:59.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:58 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:00.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:12:59 smithi192 ceph-mon[36700]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:00.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:12:59 smithi062 ceph-mon[35761]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:00.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:12:59 smithi046 ceph-mon[31859]: pgmap v159: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:01.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:00 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:01.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:00 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:01.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:00 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:02.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:01 smithi192 ceph-mon[36700]: pgmap v160: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:02.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:01 smithi062 ceph-mon[35761]: pgmap v160: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:02.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:01 smithi046 ceph-mon[31859]: pgmap v160: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:03.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:02 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:13:03.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:02 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:13:03.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:02 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:13:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:03 smithi062 ceph-mon[35761]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:03 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:13:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:03 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:13:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:03 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:13:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:03 smithi046 ceph-mon[31859]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:03 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:13:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:03 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:13:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:03 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:13:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:03 smithi192 ceph-mon[36700]: pgmap v161: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:03 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:13:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:03 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:13:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:03 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:13:06.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:05 smithi062 ceph-mon[35761]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:06.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:05 smithi046 ceph-mon[31859]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:06.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:05 smithi192 ceph-mon[36700]: pgmap v162: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:06.643 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:13:06.644 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:13:06.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 -- 172.21.15.46:0/1633980027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 msgr2=0x7f44c8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 --2- 172.21.15.46:0/1633980027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c8103930 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009890 tx=0x7f44b002bb70 comp rx=0 tx=0).stop 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 -- 172.21.15.46:0/1633980027 shutdown_connections 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 --2- 172.21.15.46:0/1633980027 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8103e70 0x7f44c8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 --2- 172.21.15.46:0/1633980027 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c8103930 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 --2- 172.21.15.46:0/1633980027 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8109570 0x7f44c8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 -- 172.21.15.46:0/1633980027 >> 172.21.15.46:0/1633980027 conn(0x7f44c80fef40 msgr2=0x7f44c8101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.884+0000 7f44cca00640 1 -- 172.21.15.46:0/1633980027 shutdown_connections 2024-03-08T23:13:06.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.885+0000 7f44cca00640 1 -- 172.21.15.46:0/1633980027 wait complete. 2024-03-08T23:13:06.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.885+0000 7f44cca00640 1 Processor -- start 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 -- start start 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8103e70 0x7f44c819ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 0x7f44c8199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c8113850 con 0x7f44c8109570 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81136d0 con 0x7f44c81034b0 2024-03-08T23:13:06.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44cca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81139d0 con 0x7f44c8103e70 2024-03-08T23:13:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.886+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 0x7f44c8199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 0x7f44c8199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54292/0 (socket says 172.21.15.46:54292) 2024-03-08T23:13:06.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44bd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8103e70 0x7f44c819ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38708/0 (socket says 172.21.15.46:38708) 2024-03-08T23:13:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c6800640 1 -- 172.21.15.46:0/3811457598 learned_addr learned my addr 172.21.15.46:0/3811457598 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 -- 172.21.15.46:0/3811457598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8103e70 msgr2=0x7f44c819ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8103e70 0x7f44c819ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 -- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 msgr2=0x7f44c8199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 0x7f44c8199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 -- 172.21.15.46:0/3811457598 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c819a3f0 con 0x7f44c81034b0 2024-03-08T23:13:06.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f44b800e9f0 tx=0x7f44b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.887+0000 7f44bf400640 1 -- 172.21.15.46:0/3811457598 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b800ccc0 con 0x7f44c81034b0 2024-03-08T23:13:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.888+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c819a6e0 con 0x7f44c81034b0 2024-03-08T23:13:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.888+0000 7f44bf400640 1 -- 172.21.15.46:0/3811457598 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44b8005020 con 0x7f44c81034b0 2024-03-08T23:13:06.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.888+0000 7f44bf400640 1 -- 172.21.15.46:0/3811457598 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b80049e0 con 0x7f44c81034b0 2024-03-08T23:13:06.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.888+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c81a69f0 con 0x7f44c81034b0 2024-03-08T23:13:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44bf400640 1 -- 172.21.15.46:0/3811457598 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f44b8005560 con 0x7f44c81034b0 2024-03-08T23:13:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 Processor -- start 2024-03-08T23:13:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 -- start start 2024-03-08T23:13:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4ea0 0x555c511cd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c51186410 con 0x555c511c4580 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c51128d50 con 0x555c511c4ea0 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.891+0000 7f44cf67f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c510d1b80 con 0x555c511cda80 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54296/0 (socket says 172.21.15.46:54296) 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 -- 172.21.15.46:0/3594569227 learned_addr learned my addr 172.21.15.46:0/3594569227 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:06.907 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4ea0 0x555c511cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44bd400640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 -- 172.21.15.46:0/3594569227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 msgr2=0x555c511cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 -- 172.21.15.46:0/3594569227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4ea0 msgr2=0x555c511cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4ea0 0x555c511cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44c6800640 1 -- 172.21.15.46:0/3594569227 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c50f077d0 con 0x555c511c4580 2024-03-08T23:13:06.908 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44bd400640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.892+0000 7f44bf400640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 0x7f44ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44bf400640 1 -- 172.21.15.46:0/3811457598 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f44b80cf1d0 con 0x7f44c81034b0 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44c6800640 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e550 tx=0x7f44c00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bbd08b2e0452b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 0x7f44ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44b7000640 1 -- 172.21.15.46:0/3594569227 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c000d990 con 0x555c511c4580 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44b7000640 1 -- 172.21.15.46:0/3594569227 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c001c070 con 0x555c511c4580 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.893+0000 7f44b7000640 1 -- 172.21.15.46:0/3594569227 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0016380 con 0x555c511c4580 2024-03-08T23:13:06.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- 172.21.15.46:0/3594569227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 msgr2=0x555c511c4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f44c000e550 tx=0x7f44c00095c0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44c5e00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 0x7f44ac0a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x555c511c4050 tx=0x7f44b80d2c80 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- 172.21.15.46:0/3594569227 shutdown_connections 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4ea0 0x555c511cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3594569227 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4580 0x555c511c4960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- 172.21.15.46:0/3594569227 >> 172.21.15.46:0/3594569227 conn(0x555c511becb0 msgr2=0x555c511bf090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- 172.21.15.46:0/3594569227 shutdown_connections 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- 172.21.15.46:0/3594569227 wait complete. 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.894+0000 7f44cf67f740 1 -- start start 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 Processor -- start 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511caa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c510e4ee0 con 0x555c511c4ea0 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c51124df0 con 0x555c511c4580 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c511d0870 con 0x555c511cda80 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44cf67f740 10 client.0 _ll_register_callbacks cb 0x555c510eb8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44bd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.895+0000 7f44c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511caa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54312/0 (socket says 172.21.15.46:54312) 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 -- 172.21.15.46:0/3978047382 learned_addr learned my addr 172.21.15.46:0/3978047382 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44c6800640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 -- 172.21.15.46:0/3978047382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 msgr2=0x555c511cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 -- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 msgr2=0x555c511caa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511caa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 -- 172.21.15.46:0/3978047382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c50e3d3f0 con 0x555c511c4ea0 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44c6800640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44bd400640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009890 tx=0x7f44b0034f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7623e53804adab9d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.896+0000 7f44ab400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0043070 con 0x555c511c4ea0 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f44b0004450 con 0x555c511c4ea0 2024-03-08T23:13:06.914 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44c001c070 con 0x555c511c4ea0 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44b004f050 con 0x555c511c4ea0 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44cf67f740 10 client.16515 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555c50dfd0f0 con 0x555c511c4ea0 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.897+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c001c070 con 0x555c511c4ea0 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f44b0032640 con 0x555c511c4ea0 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 10 client.16515 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 10 client.16515 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 20 client.16515 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:06.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c511ddb30 0x555c511dff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 4 client.16515 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c51136290 con 0x555c511ddb30 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.898+0000 7f44c6800640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c511ddb30 0x555c511dff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.899+0000 7f44ab400640 10 client.16515 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.899+0000 7f44c6800640 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c511ddb30 0x555c511dff20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a68b34ade034baac server_cookie=16c79ed13bf97614 in_seq=0 out_seq=0 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.899+0000 7f44ab400640 1 -- 172.21.15.46:0/3978047382 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c51136290 con 0x555c511ddb30 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.899+0000 7f44ab400640 10 client.16515 handle_command_reply: tid=0 2024-03-08T23:13:06.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.899+0000 7f44cf67f740 1 client.16515 shutdown 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 msgr2=0x555c511caf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f44b0009890 tx=0x7f44b0034f90 comp rx=0 tx=0).stop 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 shutdown_connections 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c511ddb30 0x555c511dff20 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c511cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c511caf60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 --2- 172.21.15.46:0/3978047382 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511caa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44ab400640 0 client.16515 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:06.917 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 >> 172.21.15.46:0/3978047382 conn(0x555c511becb0 msgr2=0x555c511bdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 shutdown_connections 2024-03-08T23:13:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 1 -- 172.21.15.46:0/3978047382 wait complete. 2024-03-08T23:13:06.918 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.900+0000 7f44cf67f740 20 client.16515 trim_cache size 0 max 16384 2024-03-08T23:13:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 Processor -- start 2024-03-08T23:13:06.919 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 -- start start 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511cd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81b2ba0 con 0x555c511cda80 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81b0710 con 0x555c511c4580 2024-03-08T23:13:06.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.917+0000 7f44cf67f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81ae7b0 con 0x555c511c4ea0 2024-03-08T23:13:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36534/0 (socket says 172.21.15.46:36534) 2024-03-08T23:13:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 -- 172.21.15.46:0/1108965832 learned_addr learned my addr 172.21.15.46:0/1108965832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:06.922 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44c6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54314/0 (socket says 172.21.15.46:54314) 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44c6800640 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 -- 172.21.15.46:0/1108965832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 msgr2=0x555c511cd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511cd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 -- 172.21.15.46:0/1108965832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 msgr2=0x555c511d4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 -- 172.21.15.46:0/1108965832 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c511cfe70 con 0x555c511c4ea0 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44c6800640 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:06.923 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.918+0000 7f44bd400640 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44b0033000 tx=0x7f44b0005b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c7c1471fa5606e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44b6600640 1 -- 172.21.15.46:0/1108965832 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0043070 con 0x555c511c4ea0 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44b6600640 1 -- 172.21.15.46:0/1108965832 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44b0032840 con 0x555c511c4ea0 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44b6600640 1 -- 172.21.15.46:0/1108965832 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b0039040 con 0x555c511c4ea0 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 -- 172.21.15.46:0/1108965832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 msgr2=0x555c511d4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44b0033000 tx=0x7f44b0005b10 comp rx=0 tx=0).stop 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 -- 172.21.15.46:0/1108965832 shutdown_connections 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511cda80 0x555c511d4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511c4ea0 0x555c511d4ab0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 --2- 172.21.15.46:0/1108965832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c511cd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.919+0000 7f44cf67f740 1 -- 172.21.15.46:0/1108965832 >> 172.21.15.46:0/1108965832 conn(0x555c511becb0 msgr2=0x555c511c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.920+0000 7f44cf67f740 1 -- 172.21.15.46:0/1108965832 shutdown_connections 2024-03-08T23:13:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.920+0000 7f44cf67f740 1 -- 172.21.15.46:0/1108965832 wait complete. 2024-03-08T23:13:06.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.920+0000 7f44cf67f740 1 -- start start 2024-03-08T23:13:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.920+0000 7f44cf67f740 1 Processor -- start 2024-03-08T23:13:06.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c5122e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c512283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81aedb0 con 0x555c511c4ea0 2024-03-08T23:13:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81b00b0 con 0x555c511c4580 2024-03-08T23:13:06.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c81af3f0 con 0x555c511cda80 2024-03-08T23:13:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44cf67f740 10 client.0 _ll_register_callbacks cb 0x7f44c0009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38738/0 (socket says 172.21.15.46:38738) 2024-03-08T23:13:06.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 -- 172.21.15.46:0/602224094 learned_addr learned my addr 172.21.15.46:0/602224094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44bd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c5122e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c512283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 -- 172.21.15.46:0/602224094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 msgr2=0x555c512283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c512283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 -- 172.21.15.46:0/602224094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 msgr2=0x555c5122e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c5122e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.921+0000 7f44c6800640 1 -- 172.21.15.46:0/602224094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c51228aa0 con 0x555c511c4580 2024-03-08T23:13:06.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44c6800640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f44c000efd0 tx=0x7f44c0026040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0d8575e7344420 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44ab400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0038020 con 0x555c511c4580 2024-03-08T23:13:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f44c0003970 con 0x555c511c4580 2024-03-08T23:13:06.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f44b0032840 con 0x555c511c4580 2024-03-08T23:13:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44c0034070 con 0x555c511c4580 2024-03-08T23:13:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44cf67f740 10 client.26672 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.922+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x555c512357a0 con 0x555c511c4580 2024-03-08T23:13:06.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44b0032840 con 0x555c511c4580 2024-03-08T23:13:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f44c000b040 con 0x555c511c4580 2024-03-08T23:13:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 10 client.26672 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 10 client.26672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 20 client.26672 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:06.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c51235e10 0x555c51238230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 4 client.26672 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:13:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555c511305e0 con 0x555c51235e10 2024-03-08T23:13:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.923+0000 7f44bd400640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c51235e10 0x555c51238230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:06.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44bd400640 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c51235e10 0x555c51238230 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=238d01896e8c345d server_cookie=473f1f44e540a8a0 in_seq=0 out_seq=0 2024-03-08T23:13:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44ab400640 10 client.26672 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44ab400640 1 -- 172.21.15.46:0/602224094 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x555c511305e0 con 0x555c51235e10 2024-03-08T23:13:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44ab400640 10 client.26672 handle_command_reply: tid=0 2024-03-08T23:13:06.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 client.26672 shutdown 2024-03-08T23:13:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 msgr2=0x555c5122df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f44c000efd0 tx=0x7f44c0026040 comp rx=0 tx=0).stop 2024-03-08T23:13:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 shutdown_connections 2024-03-08T23:13:06.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555c51235e10 0x555c51238230 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c511cda80 0x555c512283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c511c4ea0 0x555c5122e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 --2- 172.21.15.46:0/602224094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c511c4580 0x555c5122df90 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44ab400640 0 client.26672 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:06.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 >> 172.21.15.46:0/602224094 conn(0x555c511becb0 msgr2=0x555c511ca9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 shutdown_connections 2024-03-08T23:13:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.924+0000 7f44cf67f740 1 -- 172.21.15.46:0/602224094 wait complete. 2024-03-08T23:13:06.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cf67f740 20 client.26672 trim_cache size 0 max 16384 2024-03-08T23:13:06.940 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:06.940 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:13:06.941 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:13:06.941 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:06.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 msgr2=0x7f44ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 0x7f44ac0a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x555c511c4050 tx=0x7f44b80d2c80 comp rx=0 tx=0).stop 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 msgr2=0x7f44c819fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f44b800e9f0 tx=0x7f44b800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 shutdown_connections 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f44ac0a2970 0x7f44ac0a4e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44c8109570 0x7f44c8199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44c8103e70 0x7f44c819ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 --2- 172.21.15.46:0/3811457598 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44c81034b0 0x7f44c819fa70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 >> 172.21.15.46:0/3811457598 conn(0x7f44c80fef40 msgr2=0x7f44c8100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 shutdown_connections 2024-03-08T23:13:06.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:06.925+0000 7f44cca00640 1 -- 172.21.15.46:0/3811457598 wait complete. 2024-03-08T23:13:06.946 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:13:06.946 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:13:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 -- 172.21.15.46:0/3354203720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c080698c0 msgr2=0x7f5c08069d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/3354203720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c080698c0 0x7f5c08069d00 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009930 tx=0x7f5bf002bc30 comp rx=0 tx=0).stop 2024-03-08T23:13:07.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 -- 172.21.15.46:0/3354203720 shutdown_connections 2024-03-08T23:13:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/3354203720 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c0806a240 0x7f5c0810e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/3354203720 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c080698c0 0x7f5c08069d00 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/3354203720 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c08107340 0x7f5c08107720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 -- 172.21.15.46:0/3354203720 >> 172.21.15.46:0/3354203720 conn(0x7f5c080771d0 msgr2=0x7f5c080775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 -- 172.21.15.46:0/3354203720 shutdown_connections 2024-03-08T23:13:07.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.172+0000 7f5c0ec00640 1 -- 172.21.15.46:0/3354203720 wait complete. 2024-03-08T23:13:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 Processor -- start 2024-03-08T23:13:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 -- start start 2024-03-08T23:13:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 0x7f5c0819a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c08107340 0x7f5c0819e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c08111630 con 0x7f5c08107340 2024-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c081114b0 con 0x7f5c0806a240 2024-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.174+0000 7f5c0ec00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c081117b0 con 0x7f5c080698c0 2024-03-08T23:13:07.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 0x7f5c0819a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36550/0 (socket says 172.21.15.46:36550) 2024-03-08T23:13:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 0x7f5c0819a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38762/0 (socket says 172.21.15.46:38762) 2024-03-08T23:13:07.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 -- 172.21.15.46:0/577977532 learned_addr learned my addr 172.21.15.46:0/577977532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c0cc00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c08107340 0x7f5c0819e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 -- 172.21.15.46:0/577977532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 msgr2=0x7f5c0819a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 0x7f5c0819a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 -- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c08107340 msgr2=0x7f5c0819e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c08107340 0x7f5c0819e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.175+0000 7f5c07e00640 1 -- 172.21.15.46:0/577977532 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c0819ee20 con 0x7f5c080698c0 2024-03-08T23:13:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c07e00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800ebc0 tx=0x7f5bf800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8010630 con 0x7f5c080698c0 2024-03-08T23:13:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c0819f110 con 0x7f5c080698c0 2024-03-08T23:13:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5bf8010cb0 con 0x7f5c080698c0 2024-03-08T23:13:07.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8018bd0 con 0x7f5c080698c0 2024-03-08T23:13:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.176+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c081a69a0 con 0x7f5c080698c0 2024-03-08T23:13:07.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.177+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c081029e0 con 0x7f5c080698c0 2024-03-08T23:13:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.179+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5bf80107d0 con 0x7f5c080698c0 2024-03-08T23:13:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.181+0000 7f5c04c00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 0x7f5bec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.181+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f5bf80cf7c0 con 0x7f5c080698c0 2024-03-08T23:13:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.181+0000 7f5c07400640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 0x7f5bec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.182+0000 7f5c07400640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 0x7f5bec0a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005ab0 tx=0x7f5bf0005fd0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.185+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5bf809bb10 con 0x7f5c080698c0 2024-03-08T23:13:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.425+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c0819b180 con 0x7f5c080698c0 2024-03-08T23:13:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.426+0000 7f5c04c00640 1 -- 172.21.15.46:0/577977532 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11336 (secure 0 0 0) 0x7f5bf809fbb0 con 0x7f5c080698c0 2024-03-08T23:13:07.429 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:13:07.429 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:12:58.834752+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:13:07.430 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 29 2024-03-08T23:13:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 msgr2=0x7f5bec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 0x7f5bec0a4e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0005ab0 tx=0x7f5bf0005fd0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 msgr2=0x7f5c08199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f5bf800ebc0 tx=0x7f5bf800c6a0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 shutdown_connections 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5bec0a2970 0x7f5bec0a4e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5c08107340 0x7f5c0819e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5c0806a240 0x7f5c0819a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 --2- 172.21.15.46:0/577977532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5c080698c0 0x7f5c08199e30 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.429+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 >> 172.21.15.46:0/577977532 conn(0x7f5c080771d0 msgr2=0x7f5c0810c520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.430+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 shutdown_connections 2024-03-08T23:13:07.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.430+0000 7f5c0ec00640 1 -- 172.21.15.46:0/577977532 wait complete. 2024-03-08T23:13:07.452 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:07.452 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:13:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 -- 172.21.15.46:0/1638517865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470075820 msgr2=0x7ff470075c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 --2- 172.21.15.46:0/1638517865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470075820 0x7ff470075c00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff460009ca0 tx=0x7ff46002c0f0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 -- 172.21.15.46:0/1638517865 shutdown_connections 2024-03-08T23:13:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 --2- 172.21.15.46:0/1638517865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470112010 0x7ff470114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 --2- 172.21.15.46:0/1638517865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470076140 0x7ff470111ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 --2- 172.21.15.46:0/1638517865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470075820 0x7ff470075c00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 -- 172.21.15.46:0/1638517865 >> 172.21.15.46:0/1638517865 conn(0x7ff4700fee00 msgr2=0x7ff470101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 -- 172.21.15.46:0/1638517865 shutdown_connections 2024-03-08T23:13:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.661+0000 7ff474c00640 1 -- 172.21.15.46:0/1638517865 wait complete. 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 Processor -- start 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 -- start start 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470076140 0x7ff47019e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470112010 0x7ff4701a2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470117b70 con 0x7ff470076140 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701179f0 con 0x7ff470112010 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.663+0000 7ff474c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff470117cf0 con 0x7ff470075820 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff466a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470076140 0x7ff47019e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36570/0 (socket says 172.21.15.46:36570) 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 -- 172.21.15.46:0/3853514054 learned_addr learned my addr 172.21.15.46:0/3853514054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46fe00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470112010 0x7ff4701a2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 -- 172.21.15.46:0/3853514054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470112010 msgr2=0x7ff4701a2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470112010 0x7ff4701a2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 -- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470076140 msgr2=0x7ff47019e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470076140 0x7ff47019e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 -- 172.21.15.46:0/3853514054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4701a3050 con 0x7ff470075820 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.664+0000 7ff46f400640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff460005e30 tx=0x7ff460032f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.665+0000 7ff46cc00640 1 -- 172.21.15.46:0/3853514054 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460042020 con 0x7ff470075820 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.665+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4701a32e0 con 0x7ff470075820 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.665+0000 7ff46cc00640 1 -- 172.21.15.46:0/3853514054 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff460034640 con 0x7ff470075820 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.665+0000 7ff46cc00640 1 -- 172.21.15.46:0/3853514054 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff460039040 con 0x7ff470075820 2024-03-08T23:13:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.665+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4701aabd0 con 0x7ff470075820 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 Processor -- start 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 -- start start 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff46cc00640 1 -- 172.21.15.46:0/3853514054 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff460045020 con 0x7ff470075820 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 0x55a6e6eb9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6e1ddd0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6dc6c70 con 0x55a6e6eb9530 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.668+0000 7ff478f64740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6e19e70 con 0x55a6e6ec2a30 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54366/0 (socket says 172.21.15.46:54366) 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff466a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 0x55a6e6eb9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46f400640 1 -- 172.21.15.46:0/2401513179 learned_addr learned my addr 172.21.15.46:0/2401513179 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 -- 172.21.15.46:0/2401513179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 msgr2=0x55a6e6ec4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 -- 172.21.15.46:0/2401513179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 msgr2=0x55a6e6ec24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 -- 172.21.15.46:0/2401513179 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6e6e802b0 con 0x55a6e6eb9530 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.669+0000 7ff46fe00640 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 0x55a6e6eb9910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff468009b50 tx=0x7ff468011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0da9ad1fc6cf94d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.670+0000 7ff45fe00640 1 -- 172.21.15.46:0/2401513179 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468004050 con 0x55a6e6eb9530 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.670+0000 7ff45fe00640 1 -- 172.21.15.46:0/2401513179 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff468005500 con 0x55a6e6eb9530 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.670+0000 7ff45fe00640 1 -- 172.21.15.46:0/2401513179 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468004c10 con 0x55a6e6eb9530 2024-03-08T23:13:07.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.670+0000 7ff46cc00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 0x7ff4540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.670+0000 7ff46cc00640 1 -- 172.21.15.46:0/3853514054 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff4600f0d30 con 0x7ff470075820 2024-03-08T23:13:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff46f400640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 0x7ff4540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 -- 172.21.15.46:0/2401513179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 msgr2=0x55a6e6eb9910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 0x55a6e6eb9910 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7ff468009b50 tx=0x7ff468011810 comp rx=0 tx=0).stop 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 -- 172.21.15.46:0/2401513179 shutdown_connections 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 --2- 172.21.15.46:0/2401513179 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9530 0x55a6e6eb9910 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 -- 172.21.15.46:0/2401513179 >> 172.21.15.46:0/2401513179 conn(0x55a6e6eb3c60 msgr2=0x55a6e6eb4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 -- 172.21.15.46:0/2401513179 shutdown_connections 2024-03-08T23:13:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff478f64740 1 -- 172.21.15.46:0/2401513179 wait complete. 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.671+0000 7ff46f400640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 0x7ff4540a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff46002c700 tx=0x7ff4600f47e0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 -- start start 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 Processor -- start 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ebf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ebfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6ec5820 con 0x55a6e6eb9e50 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6dda0c0 con 0x55a6e6ec2a30 2024-03-08T23:13:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6e6ec59a0 con 0x55a6e6eb9530 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.672+0000 7ff478f64740 10 client.0 _ll_register_callbacks cb 0x55a6e6e14ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ebf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36580/0 (socket says 172.21.15.46:36580) 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff46fe00640 1 -- 172.21.15.46:0/3855675883 learned_addr learned my addr 172.21.15.46:0/3855675883 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff46fe00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ebfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 -- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 msgr2=0x55a6e6ec23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 -- 172.21.15.46:0/3855675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 msgr2=0x55a6e6ebfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ebfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 -- 172.21.15.46:0/3855675883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6e6b323f0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff46fe00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ebfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff466a00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ebf750 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff4701012d0 tx=0x7ff458012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566ce2883366b435 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.673+0000 7ff45cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45800f920 con 0x55a6e6eb9e50 2024-03-08T23:13:07.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff45800fac0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff468005500 con 0x55a6e6eb9e50 2024-03-08T23:13:07.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff45800fcc0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff478f64740 10 client.16539 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a6e6bfc7d0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.674+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff468005500 con 0x55a6e6eb9e50 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff45801c420 con 0x55a6e6eb9e50 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 10 client.16539 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 10 client.16539 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 20 client.16539 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6ed2b50 0x55a6e6ed4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 4 client.16539 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6e6df24f0 con 0x55a6e6ed2b50 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.675+0000 7ff46fe00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6ed2b50 0x55a6e6ed4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.676+0000 7ff46fe00640 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6ed2b50 0x55a6e6ed4f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8b0595580c74f server_cookie=5ba337cf02be736e in_seq=0 out_seq=0 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.676+0000 7ff45cc00640 10 client.16539 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.676+0000 7ff45cc00640 1 -- 172.21.15.46:0/3855675883 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a6e6df24f0 con 0x55a6e6ed2b50 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.676+0000 7ff45cc00640 10 client.16539 handle_command_reply: tid=0 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.676+0000 7ff478f64740 1 client.16539 shutdown 2024-03-08T23:13:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 msgr2=0x55a6e6ebf750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ebf750 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7ff4701012d0 tx=0x7ff458012b60 comp rx=0 tx=0).stop 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 shutdown_connections 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6ed2b50 0x55a6e6ed4f70 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ebfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ebf750 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 --2- 172.21.15.46:0/3855675883 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 >> 172.21.15.46:0/3855675883 conn(0x55a6e6eb3c60 msgr2=0x55a6e6eb59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff45cc00640 0 client.16539 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 shutdown_connections 2024-03-08T23:13:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 1 -- 172.21.15.46:0/3855675883 wait complete. 2024-03-08T23:13:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.677+0000 7ff478f64740 20 client.16539 trim_cache size 0 max 16384 2024-03-08T23:13:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 Processor -- start 2024-03-08T23:13:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 -- start start 2024-03-08T23:13:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9e50 0x55a6e6eb4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ed9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b6e40 con 0x55a6e6eb9530 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b49b0 con 0x55a6e6ec2a30 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.690+0000 7ff478f64740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b2a50 con 0x55a6e6eb9e50 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54384/0 (socket says 172.21.15.46:54384) 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 -- 172.21.15.46:0/1212462795 learned_addr learned my addr 172.21.15.46:0/1212462795 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ed9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff466a00640 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9e50 0x55a6e6eb4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 -- 172.21.15.46:0/1212462795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9e50 msgr2=0x55a6e6eb4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9e50 0x55a6e6eb4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 -- 172.21.15.46:0/1212462795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 msgr2=0x55a6e6ed9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ed9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 -- 172.21.15.46:0/1212462795 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6e6ec0040 con 0x55a6e6eb9530 2024-03-08T23:13:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff468014040 tx=0x7ff468011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4841eff320da05f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.691+0000 7ff45f400640 1 -- 172.21.15.46:0/1212462795 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4680037f0 con 0x55a6e6eb9530 2024-03-08T23:13:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff45f400640 1 -- 172.21.15.46:0/1212462795 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff46802b030 con 0x55a6e6eb9530 2024-03-08T23:13:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff45f400640 1 -- 172.21.15.46:0/1212462795 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff468027b10 con 0x55a6e6eb9530 2024-03-08T23:13:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 -- 172.21.15.46:0/1212462795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 msgr2=0x55a6e6ebf710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff468014040 tx=0x7ff468011810 comp rx=0 tx=0).stop 2024-03-08T23:13:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 -- 172.21.15.46:0/1212462795 shutdown_connections 2024-03-08T23:13:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ed9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6eb9e50 0x55a6e6eb4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 --2- 172.21.15.46:0/1212462795 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ebf710 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 -- 172.21.15.46:0/1212462795 >> 172.21.15.46:0/1212462795 conn(0x55a6e6eb3c60 msgr2=0x55a6e6ebefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 -- 172.21.15.46:0/1212462795 shutdown_connections 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.692+0000 7ff478f64740 1 -- 172.21.15.46:0/1212462795 wait complete. 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 -- start start 2024-03-08T23:13:07.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 Processor -- start 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b3050 con 0x55a6e6eb9530 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b4350 con 0x55a6e6eb9e50 2024-03-08T23:13:07.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.693+0000 7ff478f64740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4701b3690 con 0x55a6e6ec2a30 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff478f64740 10 client.0 _ll_register_callbacks cb 0x7ff468009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38832/0 (socket says 172.21.15.46:38832) 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff46fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54392/0 (socket says 172.21.15.46:54392) 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 -- 172.21.15.46:0/1189291429 learned_addr learned my addr 172.21.15.46:0/1189291429 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 -- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 msgr2=0x55a6e6ec7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 -- 172.21.15.46:0/1189291429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 msgr2=0x55a6e6ec9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 -- 172.21.15.46:0/1189291429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6e6eb9c80 con 0x55a6e6eb9e50 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff466a00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55a6e6ec0d20 tx=0x7ff458009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46265f92ea01280c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.694+0000 7ff45cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45802f060 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff46802b030 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ff45801ca80 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff458036050 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 10 client.26696 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55a6e6f298e0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff46802b030 con 0x55a6e6eb9e50 2024-03-08T23:13:07.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7ff4580035f0 con 0x55a6e6eb9e50 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 10 client.26696 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 10 client.26696 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 20 client.26696 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6f29f80 0x55a6e6f2c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 4 client.26696 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.695+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a6e6e2b310 con 0x55a6e6f29f80 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6f29f80 0x55a6e6f2c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff46fe00640 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6f29f80 0x55a6e6f2c3a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ebd926764b77442a server_cookie=227fa51dca6479b6 in_seq=0 out_seq=0 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff45cc00640 10 client.26696 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff45cc00640 1 -- 172.21.15.46:0/1189291429 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a6e6e2b310 con 0x55a6e6f29f80 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff45cc00640 10 client.26696 handle_command_reply: tid=0 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.696+0000 7ff478f64740 1 client.26696 shutdown 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 msgr2=0x55a6e6ec9dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55a6e6ec0d20 tx=0x7ff458009920 comp rx=0 tx=0).stop 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 shutdown_connections 2024-03-08T23:13:07.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a6e6f29f80 0x55a6e6f2c3a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a6e6ec2a30 0x55a6e6ec7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a6e6eb9e50 0x55a6e6ec9dc0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff45cc00640 0 client.26696 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 --2- 172.21.15.46:0/1189291429 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a6e6eb9530 0x55a6e6ec9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 >> 172.21.15.46:0/1189291429 conn(0x55a6e6eb3c60 msgr2=0x55a6e6ed9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 shutdown_connections 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 1 -- 172.21.15.46:0/1189291429 wait complete. 2024-03-08T23:13:07.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.697+0000 7ff478f64740 20 client.26696 trim_cache size 0 max 16384 2024-03-08T23:13:07.711 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 msgr2=0x7ff4540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 0x7ff4540a4e30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff46002c700 tx=0x7ff4600f47e0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 msgr2=0x7ff47019e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff460005e30 tx=0x7ff460032f20 comp rx=0 tx=0).stop 2024-03-08T23:13:07.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 shutdown_connections 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4540a2970 0x7ff4540a4e30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff470112010 0x7ff4701a2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff470076140 0x7ff47019e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 --2- 172.21.15.46:0/3853514054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff470075820 0x7ff47019e0b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 >> 172.21.15.46:0/3853514054 conn(0x7ff4700fee00 msgr2=0x7ff470113730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 shutdown_connections 2024-03-08T23:13:07.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.698+0000 7ff474c00640 1 -- 172.21.15.46:0/3853514054 wait complete. 2024-03-08T23:13:07.717 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:07.717 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:13:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3686219105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 msgr2=0x7f6cf8075c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3686219105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf8075c00 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009930 tx=0x7f6ce802bc20 comp rx=0 tx=0).stop 2024-03-08T23:13:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3686219105 shutdown_connections 2024-03-08T23:13:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3686219105 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8112010 0x7f6cf8114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3686219105 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8076140 0x7f6cf8111ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3686219105 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf8075c00 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3686219105 >> 172.21.15.46:0/3686219105 conn(0x7f6cf80fee00 msgr2=0x7f6cf8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3686219105 shutdown_connections 2024-03-08T23:13:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.927+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3686219105 wait complete. 2024-03-08T23:13:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.928+0000 7f6cf7e00640 1 Processor -- start 2024-03-08T23:13:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.928+0000 7f6cf7e00640 1 -- start start 2024-03-08T23:13:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf819e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8112010 0x7f6cf81a2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8117bc0 con 0x7f6cf8076140 2024-03-08T23:13:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8117a40 con 0x7f6cf8112010 2024-03-08T23:13:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf8117d40 con 0x7f6cf8075820 2024-03-08T23:13:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54414/0 (socket says 172.21.15.46:54414) 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf6000640 1 -- 172.21.15.46:0/3060866144 learned_addr learned my addr 172.21.15.46:0/3060866144 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf819e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.929+0000 7f6cf7400640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8112010 0x7f6cf81a2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 -- 172.21.15.46:0/3060866144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 msgr2=0x7f6cf819e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf819e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 -- 172.21.15.46:0/3060866144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8112010 msgr2=0x7f6cf81a2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8112010 0x7f6cf81a2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 -- 172.21.15.46:0/3060866144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf81a3450 con 0x7f6cf8076140 2024-03-08T23:13:07.938 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf6000640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400e9f0 tx=0x7f6ce400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cef400640 1 -- 172.21.15.46:0/3060866144 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce400cd80 con 0x7f6cf8076140 2024-03-08T23:13:07.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cef400640 1 -- 172.21.15.46:0/3060866144 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ce4010910 con 0x7f6cf8076140 2024-03-08T23:13:07.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf81a3740 con 0x7f6cf8076140 2024-03-08T23:13:07.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.930+0000 7f6cef400640 1 -- 172.21.15.46:0/3060866144 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce4018990 con 0x7f6cf8076140 2024-03-08T23:13:07.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.931+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf81aafd0 con 0x7f6cf8076140 2024-03-08T23:13:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.934+0000 7f6d00587740 1 Processor -- start 2024-03-08T23:13:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.934+0000 7f6d00587740 1 -- start start 2024-03-08T23:13:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c4464f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cef400640 1 -- 172.21.15.46:0/3060866144 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6ce4010ac0 con 0x7f6cf8076140 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c446a30 0x55ed4c448e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c3a1dd0 con 0x55ed4c43d530 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c34ac70 con 0x55ed4c446a30 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6d00587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c39de70 con 0x55ed4c43de50 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54420/0 (socket says 172.21.15.46:54420) 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 -- 172.21.15.46:0/743125735 learned_addr learned my addr 172.21.15.46:0/743125735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf6000640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c446a30 0x55ed4c448e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c4464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 -- 172.21.15.46:0/743125735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 msgr2=0x55ed4c4464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c4464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 -- 172.21.15.46:0/743125735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c446a30 msgr2=0x55ed4c448e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c446a30 0x55ed4c448e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf7400640 1 -- 172.21.15.46:0/743125735 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed4c4042b0 con 0x55ed4c43d530 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.935+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c4464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.936+0000 7f6cf7400640 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009f30 tx=0x7f6cf00118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ebe3581989263a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.936+0000 7f6ce3e00640 1 -- 172.21.15.46:0/743125735 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0004d20 con 0x55ed4c43d530 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.936+0000 7f6ce3e00640 1 -- 172.21.15.46:0/743125735 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0004050 con 0x55ed4c43d530 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.936+0000 7f6ce3e00640 1 -- 172.21.15.46:0/743125735 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf0005280 con 0x55ed4c43d530 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.936+0000 7f6cef400640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 0x7f6cdc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6cef400640 1 -- 172.21.15.46:0/3060866144 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f6ce40cf460 con 0x7f6cf8076140 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 0x7f6cdc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 -- 172.21.15.46:0/743125735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 msgr2=0x55ed4c43d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0009f30 tx=0x7f6cf00118d0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 -- 172.21.15.46:0/743125735 shutdown_connections 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c446a30 0x55ed4c448e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c4464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 --2- 172.21.15.46:0/743125735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c43d910 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 -- 172.21.15.46:0/743125735 >> 172.21.15.46:0/743125735 conn(0x55ed4c437c60 msgr2=0x55ed4c438040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 -- 172.21.15.46:0/743125735 shutdown_connections 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.937+0000 7f6d00587740 1 -- 172.21.15.46:0/743125735 wait complete. 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 0x7f6cdc0a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009900 tx=0x7f6ce8005a20 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 -- start start 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 Processor -- start 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 0x55ed4c443710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c446a30 0x55ed4c443c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c449820 con 0x55ed4c43d530 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c35e0c0 con 0x55ed4c43de50 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.938+0000 7f6d00587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed4c4499a0 con 0x55ed4c446a30 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6d00587740 10 client.0 _ll_register_callbacks cb 0x55ed4c398ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54428/0 (socket says 172.21.15.46:54428) 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 -- 172.21.15.46:0/1391718159 learned_addr learned my addr 172.21.15.46:0/1391718159 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf7400640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c446a30 0x55ed4c443c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6000640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 0x55ed4c443710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 -- 172.21.15.46:0/1391718159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c446a30 msgr2=0x55ed4c443c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c446a30 0x55ed4c443c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 -- 172.21.15.46:0/1391718159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 msgr2=0x55ed4c443710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 0x55ed4c443710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6a00640 1 -- 172.21.15.46:0/1391718159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed4c0b63f0 con 0x55ed4c43d530 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.939+0000 7f6cf6000640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 0x55ed4c443710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6ce802efd0 tx=0x7f6ce8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21fc046a491c2acc server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8044070 con 0x55ed4c43d530 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f6ce8004c10 con 0x55ed4c43d530 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6cf0004050 con 0x55ed4c43d530 2024-03-08T23:13:07.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ce802cbb0 con 0x55ed4c43d530 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6d00587740 10 client.16557 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ed4c1807d0 con 0x55ed4c43d530 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.940+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cf0004050 con 0x55ed4c43d530 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6ce8003890 con 0x55ed4c43d530 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 10 client.16557 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 10 client.16557 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:13:07.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 20 client.16557 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c456a80 0x55ed4c458e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 4 client.16557 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed4c3764f0 con 0x55ed4c456a80 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.941+0000 7f6cf6000640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c456a80 0x55ed4c458e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6ce0c00640 10 client.16557 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6cf6000640 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c456a80 0x55ed4c458e90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e761a20fd843a80 server_cookie=a5cc278965413fdb in_seq=0 out_seq=0 2024-03-08T23:13:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6ce0c00640 1 -- 172.21.15.46:0/1391718159 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed4c3764f0 con 0x55ed4c456a80 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6ce0c00640 10 client.16557 handle_command_reply: tid=0 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6d00587740 1 client.16557 shutdown 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 msgr2=0x55ed4c446380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6ce802efd0 tx=0x7f6ce8009260 comp rx=0 tx=0).stop 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.942+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 shutdown_connections 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c456a80 0x55ed4c458e90 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c446a30 0x55ed4c443c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43de50 0x55ed4c443710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6ce0c00640 0 client.16557 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 --2- 172.21.15.46:0/1391718159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c43d530 0x55ed4c446380 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 >> 172.21.15.46:0/1391718159 conn(0x55ed4c437c60 msgr2=0x55ed4c4399b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 shutdown_connections 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 1 -- 172.21.15.46:0/1391718159 wait complete. 2024-03-08T23:13:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.943+0000 7f6d00587740 20 client.16557 trim_cache size 0 max 16384 2024-03-08T23:13:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.955+0000 7f6d00587740 1 Processor -- start 2024-03-08T23:13:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 -- start start 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c442870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c445e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b6f70 con 0x55ed4c446a30 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b4ae0 con 0x55ed4c43d530 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6d00587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b2b80 con 0x55ed4c43de50 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6cf7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6cf7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54442/0 (socket says 172.21.15.46:54442) 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6cf7400640 1 -- 172.21.15.46:0/631387759 learned_addr learned my addr 172.21.15.46:0/631387759 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6cf6000640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c445e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.956+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c442870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 -- 172.21.15.46:0/631387759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 msgr2=0x55ed4c442870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c442870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 -- 172.21.15.46:0/631387759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 msgr2=0x55ed4c446380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 -- 172.21.15.46:0/631387759 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed4c443fe0 con 0x55ed4c43de50 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf7400640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6cf6000640 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c445e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40cc8f0 tx=0x7f6ce40d1040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790c42204363834e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6ce3400640 1 -- 172.21.15.46:0/631387759 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce409b210 con 0x55ed4c43de50 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6ce3400640 1 -- 172.21.15.46:0/631387759 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ce409b8b0 con 0x55ed4c43de50 2024-03-08T23:13:07.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.957+0000 7f6ce3400640 1 -- 172.21.15.46:0/631387759 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40980b0 con 0x55ed4c43de50 2024-03-08T23:13:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- 172.21.15.46:0/631387759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 msgr2=0x55ed4c445e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c445e40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6ce40cc8f0 tx=0x7f6ce40d1040 comp rx=0 tx=0).stop 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- 172.21.15.46:0/631387759 shutdown_connections 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c446380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c445e40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 --2- 172.21.15.46:0/631387759 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c442870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- 172.21.15.46:0/631387759 >> 172.21.15.46:0/631387759 conn(0x55ed4c437c60 msgr2=0x55ed4c438040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- 172.21.15.46:0/631387759 shutdown_connections 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- 172.21.15.46:0/631387759 wait complete. 2024-03-08T23:13:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 -- start start 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.958+0000 7f6d00587740 1 Processor -- start 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c44df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c44c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b3180 con 0x55ed4c446a30 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b4480 con 0x55ed4c43d530 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf81b37c0 con 0x55ed4c43de50 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6d00587740 10 client.0 _ll_register_callbacks cb 0x7f6cf0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6cf6000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6cf7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c44c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6cf6000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36648/0 (socket says 172.21.15.46:36648) 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6cf6000640 1 -- 172.21.15.46:0/625279334 learned_addr learned my addr 172.21.15.46:0/625279334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.959+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c44df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 -- 172.21.15.46:0/625279334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 msgr2=0x55ed4c44df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c44df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 -- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 msgr2=0x55ed4c44c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c44c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 -- 172.21.15.46:0/625279334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed4c44d0a0 con 0x55ed4c43de50 2024-03-08T23:13:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf7400640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c44c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6cf6000640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6ce409b5b0 tx=0x7f6ce409b210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd38a3412ded8b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6ce0c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce409f950 con 0x55ed4c43de50 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f6ce4097f20 con 0x55ed4c43de50 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ce409b8b0 con 0x55ed4c43de50 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ce40cf140 con 0x55ed4c43de50 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6d00587740 10 client.26701 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.960+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ed4c4adfb0 con 0x55ed4c43de50 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ce409b8b0 con 0x55ed4c43de50 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f6ce409ad00 con 0x55ed4c43de50 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 10 client.26701 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 10 client.26701 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 20 client.26701 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c4ae6b0 0x55ed4c4b0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 4 client.26701 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ed4c3af310 con 0x55ed4c4ae6b0 2024-03-08T23:13:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.961+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c4ae6b0 0x55ed4c4b0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6cf6a00640 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c4ae6b0 0x55ed4c4b0ad0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b97ca107e4706fc4 server_cookie=17b249f315611333 in_seq=0 out_seq=0 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6ce0c00640 10 client.26701 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6ce0c00640 1 -- 172.21.15.46:0/625279334 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ed4c3af310 con 0x55ed4c4ae6b0 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6ce0c00640 10 client.26701 handle_command_reply: tid=0 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 client.26701 shutdown 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 msgr2=0x55ed4c44c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6ce409b5b0 tx=0x7f6ce409b210 comp rx=0 tx=0).stop 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 shutdown_connections 2024-03-08T23:13:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ed4c446a30 0x55ed4c44c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55ed4c4ae6b0 0x55ed4c4b0ad0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ed4c43de50 0x55ed4c44c330 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 --2- 172.21.15.46:0/625279334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ed4c43d530 0x55ed4c44df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6ce0c00640 0 client.26701 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 >> 172.21.15.46:0/625279334 conn(0x55ed4c437c60 msgr2=0x55ed4c436ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 shutdown_connections 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 1 -- 172.21.15.46:0/625279334 wait complete. 2024-03-08T23:13:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.962+0000 7f6d00587740 20 client.26701 trim_cache size 0 max 16384 2024-03-08T23:13:07.974 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:07.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 msgr2=0x7f6cdc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 0x7f6cdc0a4e30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009900 tx=0x7f6ce8005a20 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 msgr2=0x7f6cf819e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f6ce400e9f0 tx=0x7f6ce400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 shutdown_connections 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6cdc0a2970 0x7f6cdc0a4e30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6cf8112010 0x7f6cf81a2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6cf8076140 0x7f6cf819e9a0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 --2- 172.21.15.46:0/3060866144 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6cf8075820 0x7f6cf819e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:07.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 >> 172.21.15.46:0/3060866144 conn(0x7f6cf80fee00 msgr2=0x7f6cf81011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:07.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 shutdown_connections 2024-03-08T23:13:07.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:07.963+0000 7f6cf7e00640 1 -- 172.21.15.46:0/3060866144 wait complete. 2024-03-08T23:13:07.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:07 smithi046 ceph-mon[31859]: pgmap v163: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:07.976 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:07 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/577977532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:07.984 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:07.984 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:13:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:07 smithi062 ceph-mon[35761]: pgmap v163: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:07 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/577977532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 -- 172.21.15.46:0/2687394896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c10d7d0 msgr2=0x7fb76c10dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 --2- 172.21.15.46:0/2687394896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c10d7d0 0x7fb76c10dbb0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fb75c009900 tx=0x7fb75c02bcd0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 -- 172.21.15.46:0/2687394896 shutdown_connections 2024-03-08T23:13:08.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 --2- 172.21.15.46:0/2687394896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c1081d0 0x7fb76c114620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 --2- 172.21.15.46:0/2687394896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c107830 0x7fb76c107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 --2- 172.21.15.46:0/2687394896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c10d7d0 0x7fb76c10dbb0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 -- 172.21.15.46:0/2687394896 >> 172.21.15.46:0/2687394896 conn(0x7fb76c077110 msgr2=0x7fb76c106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 -- 172.21.15.46:0/2687394896 shutdown_connections 2024-03-08T23:13:08.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.201+0000 7fb770e00640 1 -- 172.21.15.46:0/2687394896 wait complete. 2024-03-08T23:13:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 Processor -- start 2024-03-08T23:13:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 -- start start 2024-03-08T23:13:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 0x7fb76c19e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 0x7fb76c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 0x7fb76c1a2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c117be0 con 0x7fb76c107830 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c117a60 con 0x7fb76c10d7d0 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.202+0000 7fb770e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c117d60 con 0x7fb76c1081d0 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 0x7fb76c19e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 0x7fb76c19e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54476/0 (socket says 172.21.15.46:54476) 2024-03-08T23:13:08.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76b400640 1 -- 172.21.15.46:0/2127193895 learned_addr learned my addr 172.21.15.46:0/2127193895 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 0x7fb76c1a2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 0x7fb76c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 -- 172.21.15.46:0/2127193895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 msgr2=0x7fb76c19e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 0x7fb76c19e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 -- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 msgr2=0x7fb76c19e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 0x7fb76c19e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 -- 172.21.15.46:0/2127193895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c1a3030 con 0x7fb76c10d7d0 2024-03-08T23:13:08.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 0x7fb76c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.203+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 0x7fb76c1a2900 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb764005fd0 tx=0x7fb76400ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.204+0000 7fb763e00640 1 -- 172.21.15.46:0/2127193895 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76400cd80 con 0x7fb76c10d7d0 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.204+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c1a3320 con 0x7fb76c10d7d0 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.204+0000 7fb763e00640 1 -- 172.21.15.46:0/2127193895 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb764005020 con 0x7fb76c10d7d0 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.204+0000 7fb763e00640 1 -- 172.21.15.46:0/2127193895 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb764010720 con 0x7fb76c10d7d0 2024-03-08T23:13:08.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.204+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c1aabb0 con 0x7fb76c10d7d0 2024-03-08T23:13:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.207+0000 7fb77521a740 1 Processor -- start 2024-03-08T23:13:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.207+0000 7fb763e00640 1 -- 172.21.15.46:0/2127193895 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb7640051d0 con 0x7fb76c10d7d0 2024-03-08T23:13:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.207+0000 7fb77521a740 1 -- start start 2024-03-08T23:13:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x5629313f8910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x562931403e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56293135cdd0 con 0x562931401a30 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562931305c70 con 0x5629313f8530 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb77521a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562931358e70 con 0x5629313f8e50 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb76b400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb76b400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36682/0 (socket says 172.21.15.46:36682) 2024-03-08T23:13:08.214 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb76aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x5629313f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb76b400640 1 -- 172.21.15.46:0/3145802974 learned_addr learned my addr 172.21.15.46:0/3145802974 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.208+0000 7fb76aa00640 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x562931403e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 -- 172.21.15.46:0/3145802974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 msgr2=0x5629313f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x5629313f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 -- 172.21.15.46:0/3145802974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 msgr2=0x562931403e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x562931403e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb763e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 0x7fb7500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 -- 172.21.15.46:0/3145802974 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629313bf2b0 con 0x5629313f8e50 2024-03-08T23:13:08.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76aa00640 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x562931403e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb763e00640 1 -- 172.21.15.46:0/2127193895 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb7640cf760 con 0x7fb76c10d7d0 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 0x7fb7500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb76b400640 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005be0 tx=0x7fb75c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3010e8960be8b1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb760c00640 1 -- 172.21.15.46:0/3145802974 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c043070 con 0x5629313f8e50 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb760c00640 1 -- 172.21.15.46:0/3145802974 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb75c02c920 con 0x5629313f8e50 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.209+0000 7fb760c00640 1 -- 172.21.15.46:0/3145802974 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c004b70 con 0x5629313f8e50 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 -- 172.21.15.46:0/3145802974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 msgr2=0x5629314014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb75c005be0 tx=0x7fb75c009560 comp rx=0 tx=0).stop 2024-03-08T23:13:08.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb76be00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 0x7fb7500a4eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7640ccb90 tx=0x7fb76401c000 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 -- 172.21.15.46:0/3145802974 shutdown_connections 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x562931403e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x5629314014f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 --2- 172.21.15.46:0/3145802974 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x5629313f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 -- 172.21.15.46:0/3145802974 >> 172.21.15.46:0/3145802974 conn(0x5629313f2c60 msgr2=0x5629313f3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 -- 172.21.15.46:0/3145802974 shutdown_connections 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.210+0000 7fb77521a740 1 -- 172.21.15.46:0/3145802974 wait complete. 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 -- start start 2024-03-08T23:13:08.217 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 Processor -- start 2024-03-08T23:13:08.222 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x562931401360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562931401a30 0x5629313fec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629314047c0 con 0x5629313f8e50 2024-03-08T23:13:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629313190c0 con 0x5629313f8530 2024-03-08T23:13:08.223 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562931404940 con 0x562931401a30 2024-03-08T23:13:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.211+0000 7fb77521a740 10 client.0 _ll_register_callbacks cb 0x562931353ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54500/0 (socket says 172.21.15.46:54500) 2024-03-08T23:13:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 -- 172.21.15.46:0/853160690 learned_addr learned my addr 172.21.15.46:0/853160690 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:08.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76aa00640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562931401a30 0x5629313fec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76aa00640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x562931401360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 -- 172.21.15.46:0/853160690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562931401a30 msgr2=0x5629313fec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562931401a30 0x5629313fec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 -- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 msgr2=0x562931401360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x562931401360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 -- 172.21.15.46:0/853160690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629310713f0 con 0x5629313f8e50 2024-03-08T23:13:08.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76aa00640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x562931401360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:08.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb76b400640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006010 tx=0x7fb75c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a87316d796c0e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.212+0000 7fb755600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c04f070 con 0x5629313f8e50 2024-03-08T23:13:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb75c0052f0 con 0x5629313f8e50 2024-03-08T23:13:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb75c02c920 con 0x5629313f8e50 2024-03-08T23:13:08.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb75c05b050 con 0x5629313f8e50 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb77521a740 10 client.16575 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x56293113b7d0 con 0x5629313f8e50 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.213+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb75c02c920 con 0x5629313f8e50 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb75c048ca0 con 0x5629313f8e50 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 10 client.16575 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:08.228 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 10 client.16575 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 20 client.16575 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629314118c0 0x562931413cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 4 client.16575 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:08.229 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629313314f0 con 0x5629314118c0 2024-03-08T23:13:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.214+0000 7fb76aa00640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629314118c0 0x562931413cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.215+0000 7fb76aa00640 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629314118c0 0x562931413cb0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=510836bca063a1b4 server_cookie=f70dfdb592e30684 in_seq=0 out_seq=0 2024-03-08T23:13:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.215+0000 7fb755600640 10 client.16575 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:08.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.215+0000 7fb755600640 1 -- 172.21.15.46:0/853160690 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5629313314f0 con 0x5629314118c0 2024-03-08T23:13:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.215+0000 7fb755600640 10 client.16575 handle_command_reply: tid=0 2024-03-08T23:13:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.215+0000 7fb77521a740 1 client.16575 shutdown 2024-03-08T23:13:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 msgr2=0x5629313fe6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb75c006010 tx=0x7fb75c009260 comp rx=0 tx=0).stop 2024-03-08T23:13:08.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 shutdown_connections 2024-03-08T23:13:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5629314118c0 0x562931413cb0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562931401a30 0x5629313fec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8e50 0x5629313fe6f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 --2- 172.21.15.46:0/853160690 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8530 0x562931401360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb755600640 0 client.16575 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 >> 172.21.15.46:0/853160690 conn(0x5629313f2c60 msgr2=0x5629313f49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 shutdown_connections 2024-03-08T23:13:08.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 1 -- 172.21.15.46:0/853160690 wait complete. 2024-03-08T23:13:08.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.216+0000 7fb77521a740 20 client.16575 trim_cache size 0 max 16384 2024-03-08T23:13:08.240 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 Processor -- start 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 -- start start 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x562931401360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562931401a30 0x5629313fcfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b6d70 con 0x5629313f8530 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b48e0 con 0x562931401a30 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.229+0000 7fb77521a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b2980 con 0x5629313f8e50 2024-03-08T23:13:08.241 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54502/0 (socket says 172.21.15.46:54502) 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 -- 172.21.15.46:0/2330137818 learned_addr learned my addr 172.21.15.46:0/2330137818 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76b400640 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x562931401360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562931401a30 0x5629313fcfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 -- 172.21.15.46:0/2330137818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 msgr2=0x562931401360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x562931401360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 -- 172.21.15.46:0/2330137818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562931401a30 msgr2=0x5629313fcfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562931401a30 0x5629313fcfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 -- 172.21.15.46:0/2330137818 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562931418b90 con 0x5629313f8530 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.230+0000 7fb76aa00640 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x5629313f2530 tx=0x7fb758005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3e20af7eb00359 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb757e00640 1 -- 172.21.15.46:0/2330137818 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb758005050 con 0x5629313f8530 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb757e00640 1 -- 172.21.15.46:0/2330137818 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb758004730 con 0x5629313f8530 2024-03-08T23:13:08.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb757e00640 1 -- 172.21.15.46:0/2330137818 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75800f420 con 0x5629313f8530 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 -- 172.21.15.46:0/2330137818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 msgr2=0x562931400e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x5629313f2530 tx=0x7fb758005f90 comp rx=0 tx=0).stop 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 -- 172.21.15.46:0/2330137818 shutdown_connections 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562931401a30 0x5629313fcfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8e50 0x562931401360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.231+0000 7fb77521a740 1 --2- 172.21.15.46:0/2330137818 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5629313f8530 0x562931400e20 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.232+0000 7fb77521a740 1 -- 172.21.15.46:0/2330137818 >> 172.21.15.46:0/2330137818 conn(0x5629313f2c60 msgr2=0x5629313fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.232+0000 7fb77521a740 1 -- 172.21.15.46:0/2330137818 shutdown_connections 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.232+0000 7fb77521a740 1 -- 172.21.15.46:0/2330137818 wait complete. 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.232+0000 7fb77521a740 1 -- start start 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.232+0000 7fb77521a740 1 Processor -- start 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8530 0x5629314071f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x5629314085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b2f80 con 0x562931401a30 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b4280 con 0x5629313f8e50 2024-03-08T23:13:08.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c1b35c0 con 0x5629313f8530 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb77521a740 10 client.0 _ll_register_callbacks cb 0x7fb758005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38938/0 (socket says 172.21.15.46:38938) 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 -- 172.21.15.46:0/3493394617 learned_addr learned my addr 172.21.15.46:0/3493394617 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8530 0x5629314071f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76aa00640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x5629314085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 -- 172.21.15.46:0/3493394617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8530 msgr2=0x5629314071f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8530 0x5629314071f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 -- 172.21.15.46:0/3493394617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 msgr2=0x5629314085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x5629314085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.233+0000 7fb76b400640 1 -- 172.21.15.46:0/3493394617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562931408df0 con 0x5629313f8e50 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb76b400640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5629313ffcc0 tx=0x7fb75c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6168f3185cbd3919 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb755600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb75c04f040 con 0x5629313f8e50 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fb75c004ae0 con 0x5629313f8e50 2024-03-08T23:13:08.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb758004730 con 0x5629313f8e50 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb75c065050 con 0x5629313f8e50 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb77521a740 10 client.26732 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.234+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x562931468640 con 0x5629313f8e50 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb758004730 con 0x5629313f8e50 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fb75c02c920 con 0x5629313f8e50 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 10 client.26732 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 10 client.26732 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 20 client.26732 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562931468d60 0x56293146b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:08.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 4 client.26732 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56293136a310 con 0x562931468d60 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.235+0000 7fb76aa00640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562931468d60 0x56293146b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb76aa00640 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562931468d60 0x56293146b180 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab668e32066b6d75 server_cookie=ccafb96896af7427 in_seq=0 out_seq=0 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb755600640 10 client.26732 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb755600640 1 -- 172.21.15.46:0/3493394617 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56293136a310 con 0x562931468d60 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb755600640 10 client.26732 handle_command_reply: tid=0 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 client.26732 shutdown 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 msgr2=0x562931407730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.249 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5629313ffcc0 tx=0x7fb75c0046c0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 shutdown_connections 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562931468d60 0x56293146b180 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562931401a30 0x5629314085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb755600640 0 client.26732 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5629313f8e50 0x562931407730 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 --2- 172.21.15.46:0/3493394617 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5629313f8530 0x5629314071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 >> 172.21.15.46:0/3493394617 conn(0x5629313f2c60 msgr2=0x5629313f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 shutdown_connections 2024-03-08T23:13:08.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 1 -- 172.21.15.46:0/3493394617 wait complete. 2024-03-08T23:13:08.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.236+0000 7fb77521a740 20 client.26732 trim_cache size 0 max 16384 2024-03-08T23:13:08.251 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 msgr2=0x7fb7500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 0x7fb7500a4eb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb7640ccb90 tx=0x7fb76401c000 comp rx=0 tx=0).stop 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 msgr2=0x7fb76c1a2900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 0x7fb76c1a2900 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb764005fd0 tx=0x7fb76400ee60 comp rx=0 tx=0).stop 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 shutdown_connections 2024-03-08T23:13:08.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb7500a29f0 0x7fb7500a4eb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb76c10d7d0 0x7fb76c1a2900 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb76c1081d0 0x7fb76c19e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 --2- 172.21.15.46:0/2127193895 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb76c107830 0x7fb76c19e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 >> 172.21.15.46:0/2127193895 conn(0x7fb76c077110 msgr2=0x7fb76c078070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 shutdown_connections 2024-03-08T23:13:08.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:08.237+0000 7fb770e00640 1 -- 172.21.15.46:0/2127193895 wait complete. 2024-03-08T23:13:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:07 smithi192 ceph-mon[36700]: pgmap v163: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:07 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/577977532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:09.258 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:13:09.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:13:09.259 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:13:09.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 -- 172.21.15.46:0/2188612888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c076140 msgr2=0x7f483c111bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 --2- 172.21.15.46:0/2188612888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c076140 0x7f483c111bf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4828009c10 tx=0x7f482802bef0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 -- 172.21.15.46:0/2188612888 shutdown_connections 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 --2- 172.21.15.46:0/2188612888 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 0x7f483c114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 --2- 172.21.15.46:0/2188612888 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c076140 0x7f483c111bf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 --2- 172.21.15.46:0/2188612888 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c075820 0x7f483c075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 -- 172.21.15.46:0/2188612888 >> 172.21.15.46:0/2188612888 conn(0x7f483c0fef40 msgr2=0x7f483c101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 -- 172.21.15.46:0/2188612888 shutdown_connections 2024-03-08T23:13:09.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.497+0000 7f4842400640 1 -- 172.21.15.46:0/2188612888 wait complete. 2024-03-08T23:13:09.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 Processor -- start 2024-03-08T23:13:09.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 -- start start 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 0x7f483c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 0x7f483c1a2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c117ce0 con 0x7f483c112130 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c117b60 con 0x7f483c075820 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.498+0000 7f4842400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c117e60 con 0x7f483c076140 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 0x7f483c1a2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38952/0 (socket says 172.21.15.46:38952) 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 0x7f483c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 -- 172.21.15.46:0/1903167719 learned_addr learned my addr 172.21.15.46:0/1903167719 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 -- 172.21.15.46:0/1903167719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 msgr2=0x7f483c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 0x7f483c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 -- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 msgr2=0x7f483c1a2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 0x7f483c1a2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483b400640 1 -- 172.21.15.46:0/1903167719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483c1a32a0 con 0x7f483c075820 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.499+0000 7f483aa00640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 0x7f483c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f483b400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f482c00e9f0 tx=0x7f482c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f4833e00640 1 -- 172.21.15.46:0/1903167719 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00ccc0 con 0x7f483c075820 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f4833e00640 1 -- 172.21.15.46:0/1903167719 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f482c00ce60 con 0x7f483c075820 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f4833e00640 1 -- 172.21.15.46:0/1903167719 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c010690 con 0x7f483c075820 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483c1a3590 con 0x7f483c075820 2024-03-08T23:13:09.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.500+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483c1aae20 con 0x7f483c075820 2024-03-08T23:13:09.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.503+0000 7f4845171740 1 Processor -- start 2024-03-08T23:13:09.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.503+0000 7f4845171740 1 -- start start 2024-03-08T23:13:09.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.503+0000 7f4833e00640 1 -- 172.21.15.46:0/1903167719 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f482c0046c0 con 0x7f483c075820 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bca9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 0x557031bd35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 0x557031bd5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031b2ea50 con 0x557031bd3ae0 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031ad7b00 con 0x557031bcaf00 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f4845171740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031b2aaf0 con 0x557031bca5e0 2024-03-08T23:13:09.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 0x557031bd35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 0x557031bd5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 0x557031bd35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38954/0 (socket says 172.21.15.46:38954) 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483b400640 1 -- 172.21.15.46:0/724969483 learned_addr learned my addr 172.21.15.46:0/724969483 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483be00640 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bca9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 -- 172.21.15.46:0/724969483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 msgr2=0x557031bca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bca9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 -- 172.21.15.46:0/724969483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 msgr2=0x557031bd35a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 0x557031bd35a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.504+0000 7f483aa00640 1 -- 172.21.15.46:0/724969483 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557031af51e0 con 0x557031bd3ae0 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f483aa00640 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 0x557031bd5ed0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f482802c430 tx=0x7f4828033f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eedf5151534f539 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f4833e00640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 0x7f48200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f4830c00640 1 -- 172.21.15.46:0/724969483 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828043030 con 0x557031bd3ae0 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f4830c00640 1 -- 172.21.15.46:0/724969483 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4828003970 con 0x557031bd3ae0 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f4830c00640 1 -- 172.21.15.46:0/724969483 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482802cc90 con 0x557031bd3ae0 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f4833e00640 1 -- 172.21.15.46:0/1903167719 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f482c0cf580 con 0x7f483c075820 2024-03-08T23:13:09.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.505+0000 7f483be00640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 0x7f48200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 -- 172.21.15.46:0/724969483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 msgr2=0x557031bd5ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 0x557031bd5ed0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f482802c430 tx=0x7f4828033f70 comp rx=0 tx=0).stop 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 -- 172.21.15.46:0/724969483 shutdown_connections 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bd3ae0 0x557031bd5ed0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bcaf00 0x557031bd35a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 --2- 172.21.15.46:0/724969483 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bca9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 -- 172.21.15.46:0/724969483 >> 172.21.15.46:0/724969483 conn(0x557031bc4d10 msgr2=0x557031bc50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f483be00640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 0x7f48200a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f483c06a6b0 tx=0x7f4834009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 -- 172.21.15.46:0/724969483 shutdown_connections 2024-03-08T23:13:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.506+0000 7f4845171740 1 -- 172.21.15.46:0/724969483 wait complete. 2024-03-08T23:13:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 -- start start 2024-03-08T23:13:09.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 Processor -- start 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bd0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031bd6870 con 0x557031bcaf00 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031aeaf60 con 0x557031bd3ae0 2024-03-08T23:13:09.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557031bd69f0 con 0x557031bca5e0 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.507+0000 7f4845171740 10 client.0 _ll_register_callbacks cb 0x557031ad6b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483b400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bd0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38960/0 (socket says 172.21.15.46:38960) 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 -- 172.21.15.46:0/726635904 learned_addr learned my addr 172.21.15.46:0/726635904 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 -- 172.21.15.46:0/726635904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 msgr2=0x557031bd0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bd0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 -- 172.21.15.46:0/726635904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 msgr2=0x557031bd1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 -- 172.21.15.46:0/726635904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55703190b7e0 con 0x557031bd3ae0 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483b400640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bd0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483aa00640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f483be00640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4834009670 tx=0x7f4834004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f629f3a81fd33533 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.508+0000 7f4825600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483402d060 con 0x557031bd3ae0 2024-03-08T23:13:09.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4834018030 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4828003970 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4834034050 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4845171740 10 client.26744 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55703196bf70 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4828003970 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.509+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f483400f450 con 0x557031bd3ae0 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 10 client.26744 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 10 client.26744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:09.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 20 client.26744 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031be3e10 0x557031be6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 4 client.26744 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557031b3bf90 con 0x557031be3e10 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.510+0000 7f483b400640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031be3e10 0x557031be6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.511+0000 7f483b400640 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031be3e10 0x557031be6200 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc002d2a6f2da269 server_cookie=4fcbcb5359f5d916 in_seq=0 out_seq=0 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.511+0000 7f4825600640 10 client.26744 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.511+0000 7f4825600640 1 -- 172.21.15.46:0/726635904 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557031b3bf90 con 0x557031be3e10 2024-03-08T23:13:09.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.511+0000 7f4825600640 10 client.26744 handle_command_reply: tid=0 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.511+0000 7f4845171740 1 client.26744 shutdown 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 msgr2=0x557031bd1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4834009670 tx=0x7f4834004730 comp rx=0 tx=0).stop 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 shutdown_connections 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031be3e10 0x557031be6200 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bd3ae0 0x557031bd1550 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 --2- 172.21.15.46:0/726635904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bca5e0 0x557031bd0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4825600640 0 client.26744 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 >> 172.21.15.46:0/726635904 conn(0x557031bc4d10 msgr2=0x557031bc6a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 shutdown_connections 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 1 -- 172.21.15.46:0/726635904 wait complete. 2024-03-08T23:13:09.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.512+0000 7f4845171740 20 client.26744 trim_cache size 0 max 16384 2024-03-08T23:13:09.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 Processor -- start 2024-03-08T23:13:09.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 -- start start 2024-03-08T23:13:09.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bdac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b7020 con 0x557031bcaf00 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b4b90 con 0x557031bca5e0 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f4845171740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b2c30 con 0x557031bd3ae0 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.529+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:38974/0 (socket says 172.21.15.46:38974) 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 -- 172.21.15.46:0/2204004140 learned_addr learned my addr 172.21.15.46:0/2204004140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483aa00640 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483be00640 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bdac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 -- 172.21.15.46:0/2204004140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 msgr2=0x557031bdac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bdac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 -- 172.21.15.46:0/2204004140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 msgr2=0x557031bd11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 -- 172.21.15.46:0/2204004140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557031803170 con 0x557031bca5e0 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f483b400640 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f482c09b810 tx=0x7f482c086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebb61e5a548f39 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f4827e00640 1 -- 172.21.15.46:0/2204004140 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c0b4070 con 0x557031bca5e0 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f4827e00640 1 -- 172.21.15.46:0/2204004140 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f482c09bf80 con 0x557031bca5e0 2024-03-08T23:13:09.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.530+0000 7f4827e00640 1 -- 172.21.15.46:0/2204004140 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c099c10 con 0x557031bca5e0 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 -- 172.21.15.46:0/2204004140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 msgr2=0x557031bd3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f482c09b810 tx=0x7f482c086040 comp rx=0 tx=0).stop 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 -- 172.21.15.46:0/2204004140 shutdown_connections 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bdac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031bd11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 --2- 172.21.15.46:0/2204004140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031bd3200 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 -- 172.21.15.46:0/2204004140 >> 172.21.15.46:0/2204004140 conn(0x557031bc4d10 msgr2=0x557031bd0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 -- 172.21.15.46:0/2204004140 shutdown_connections 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.531+0000 7f4845171740 1 -- 172.21.15.46:0/2204004140 wait complete. 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 -- start start 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 Processor -- start 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031c34830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bda0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b3230 con 0x557031bcaf00 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b4530 con 0x557031bca5e0 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f483c1b3870 con 0x557031bd3ae0 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.532+0000 7f4845171740 10 client.0 _ll_register_callbacks cb 0x7f482c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:54580/0 (socket says 172.21.15.46:54580) 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 -- 172.21.15.46:0/26260681 learned_addr learned my addr 172.21.15.46:0/26260681 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483be00640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bda0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483b400640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031c34830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 -- 172.21.15.46:0/26260681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 msgr2=0x557031bda0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bda0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 -- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 msgr2=0x557031c34830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031c34830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.533+0000 7f483aa00640 1 -- 172.21.15.46:0/26260681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557031afa050 con 0x557031bcaf00 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f483aa00640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x557031bc5ea0 tx=0x7f4828005ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7b7263713bd168 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4825600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4828043040 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f482c09bf80 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4828003b70 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f482805a050 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4845171740 10 client.16599 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.534+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x557031c3bf30 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f482c09bf80 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f482803f440 con 0x557031bcaf00 2024-03-08T23:13:09.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 10 client.16599 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 20 client.16599 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031c3c340 0x557031c3e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 4 client.16599 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557031b362e0 con 0x557031c3c340 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.535+0000 7f483b400640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031c3c340 0x557031c3e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.536+0000 7f483b400640 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031c3c340 0x557031c3e730 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fccaef296b0f33ef server_cookie=f6c6a7107ced8b89 in_seq=0 out_seq=0 2024-03-08T23:13:09.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.536+0000 7f4825600640 10 client.16599 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4825600640 1 -- 172.21.15.46:0/26260681 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557031b362e0 con 0x557031c3c340 2024-03-08T23:13:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4825600640 10 client.16599 handle_command_reply: tid=0 2024-03-08T23:13:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 client.16599 shutdown 2024-03-08T23:13:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 msgr2=0x557031c34d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x557031bc5ea0 tx=0x7f4828005ec0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 shutdown_connections 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557031c3c340 0x557031c3e730 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557031bd3ae0 0x557031bda0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557031bcaf00 0x557031c34d70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4825600640 0 client.16599 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 --2- 172.21.15.46:0/26260681 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557031bca5e0 0x557031c34830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 >> 172.21.15.46:0/26260681 conn(0x557031bc4d10 msgr2=0x557031bc3580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 shutdown_connections 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.537+0000 7f4845171740 1 -- 172.21.15.46:0/26260681 wait complete. 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4845171740 20 client.16599 trim_cache size 0 max 16384 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:09.548 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:13:09.549 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "81b0e481-750e-44fe-96f2-76b719514bd2", 2024-03-08T23:13:09.549 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:13:09.549 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:09.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 msgr2=0x7f48200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 0x7f48200a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f483c06a6b0 tx=0x7f4834009260 comp rx=0 tx=0).stop 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 msgr2=0x7f483c19e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f482c00e9f0 tx=0x7f482c00edf0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 shutdown_connections 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48200a2970 0x7f48200a4e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f483c112130 0x7f483c1a2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f483c076140 0x7f483c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 --2- 172.21.15.46:0/1903167719 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f483c075820 0x7f483c19e2b0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 >> 172.21.15.46:0/1903167719 conn(0x7f483c0fef40 msgr2=0x7f483c101330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 shutdown_connections 2024-03-08T23:13:09.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.538+0000 7f4842400640 1 -- 172.21.15.46:0/1903167719 wait complete. 2024-03-08T23:13:09.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:13:09.563 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 -- 172.21.15.46:0/3523845313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 msgr2=0x7f4f7410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 --2- 172.21.15.46:0/3523845313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7410e190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009ca0 tx=0x7f4f6c02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 -- 172.21.15.46:0/3523845313 shutdown_connections 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 --2- 172.21.15.46:0/3523845313 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7410e190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 --2- 172.21.15.46:0/3523845313 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f74101ea0 0x7f4f74105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 --2- 172.21.15.46:0/3523845313 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f741018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 -- 172.21.15.46:0/3523845313 >> 172.21.15.46:0/3523845313 conn(0x7f4f740786f0 msgr2=0x7f4f74078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 -- 172.21.15.46:0/3523845313 shutdown_connections 2024-03-08T23:13:09.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.835+0000 7f4f79800640 1 -- 172.21.15.46:0/3523845313 wait complete. 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 Processor -- start 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 -- start start 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 0x7f4f7419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f74111660 con 0x7f4f7419e5e0 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741114e0 con 0x7f4f741014f0 2024-03-08T23:13:09.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.837+0000 7f4f79800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741117e0 con 0x7f4f741065c0 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39002/0 (socket says 172.21.15.46:39002) 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 -- 172.21.15.46:0/1633682896 learned_addr learned my addr 172.21.15.46:0/1633682896 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f73400640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 0x7f4f7419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72000640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 -- 172.21.15.46:0/1633682896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 msgr2=0x7f4f7419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 -- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 msgr2=0x7f4f7419ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 0x7f4f7419ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72a00640 1 -- 172.21.15.46:0/1633682896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7419f0d0 con 0x7f4f741014f0 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f73400640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 0x7f4f7419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.838+0000 7f4f72000640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f72a00640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400e9f0 tx=0x7f4f6400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f6b400640 1 -- 172.21.15.46:0/1633682896 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6400cd80 con 0x7f4f741014f0 2024-03-08T23:13:09.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f6b400640 1 -- 172.21.15.46:0/1633682896 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f64010910 con 0x7f4f741014f0 2024-03-08T23:13:09.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f6b400640 1 -- 172.21.15.46:0/1633682896 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64018990 con 0x7f4f741014f0 2024-03-08T23:13:09.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7419f3c0 con 0x7f4f741014f0 2024-03-08T23:13:09.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.839+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f741a6b60 con 0x7f4f741014f0 2024-03-08T23:13:09.846 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.843+0000 7f4f6b400640 1 -- 172.21.15.46:0/1633682896 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4f64018b30 con 0x7f4f741014f0 2024-03-08T23:13:09.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.844+0000 7f4f7c609740 1 Processor -- start 2024-03-08T23:13:09.847 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 -- start start 2024-03-08T23:13:09.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 0x561b318c4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4ea0 0x561b318cd540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b31886410 con 0x561b318c4580 2024-03-08T23:13:09.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b31828d50 con 0x561b318cda80 2024-03-08T23:13:09.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.845+0000 7f4f7c609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b317d1b80 con 0x561b318c4ea0 2024-03-08T23:13:09.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f6b400640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 0x7f4f580a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39006/0 (socket says 172.21.15.46:39006) 2024-03-08T23:13:09.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 -- 172.21.15.46:0/4037283987 learned_addr learned my addr 172.21.15.46:0/4037283987 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72a00640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4ea0 0x561b318cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f73400640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 0x561b318c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f6b400640 1 -- 172.21.15.46:0/1633682896 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4f640cf5d0 con 0x7f4f741014f0 2024-03-08T23:13:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 -- 172.21.15.46:0/4037283987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4ea0 msgr2=0x561b318cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4ea0 0x561b318cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.846+0000 7f4f72000640 1 -- 172.21.15.46:0/4037283987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 msgr2=0x561b318c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f72000640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 0x561b318c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f72000640 1 -- 172.21.15.46:0/4037283987 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b316077d0 con 0x561b318cda80 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f73400640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 0x561b318c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f73400640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 0x7f4f580a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f72000640 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c30 tx=0x7f4f600117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6694877da87c71c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.847+0000 7f4f5fe00640 1 -- 172.21.15.46:0/4037283987 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60004020 con 0x561b318cda80 2024-03-08T23:13:09.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.848+0000 7f4f5fe00640 1 -- 172.21.15.46:0/4037283987 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f600041c0 con 0x561b318cda80 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.848+0000 7f4f5fe00640 1 -- 172.21.15.46:0/4037283987 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60012780 con 0x561b318cda80 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.848+0000 7f4f7c609740 1 -- 172.21.15.46:0/4037283987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 msgr2=0x561b318cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.848+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c30 tx=0x7f4f600117b0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f73400640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 0x7f4f580a4eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c006010 tx=0x7f4f6c0095c0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 -- 172.21.15.46:0/4037283987 shutdown_connections 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318cfe70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4ea0 0x561b318cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4037283987 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4580 0x561b318c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 -- 172.21.15.46:0/4037283987 >> 172.21.15.46:0/4037283987 conn(0x561b318becb0 msgr2=0x561b318bf090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 -- 172.21.15.46:0/4037283987 shutdown_connections 2024-03-08T23:13:09.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.849+0000 7f4f7c609740 1 -- 172.21.15.46:0/4037283987 wait complete. 2024-03-08T23:13:09.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.850+0000 7f4f7c609740 1 -- start start 2024-03-08T23:13:09.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.850+0000 7f4f7c609740 1 Processor -- start 2024-03-08T23:13:09.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b318caf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b318cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b317e4ee0 con 0x561b318cda80 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b31824df0 con 0x561b318c4ea0 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b318d0870 con 0x561b318c4580 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f7c609740 10 client.0 _ll_register_callbacks cb 0x561b317eb8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.859 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36792/0 (socket says 172.21.15.46:36792) 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f72a00640 1 -- 172.21.15.46:0/4033194016 learned_addr learned my addr 172.21.15.46:0/4033194016 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f72000640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b318caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.851+0000 7f4f73400640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b318cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 -- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 msgr2=0x561b318caf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b318caf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 -- 172.21.15.46:0/4033194016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 msgr2=0x561b318cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b318cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 -- 172.21.15.46:0/4033194016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b3153d3f0 con 0x561b318c4580 2024-03-08T23:13:09.860 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f73400640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b318cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f72a00640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4f6409b860 tx=0x7f4f6409bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24915d2ef4b4a565 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6409b8d0 con 0x561b318c4580 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.852+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4f64010430 con 0x561b318c4580 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f600041c0 con 0x561b318c4580 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f640b3020 con 0x561b318c4580 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f7c609740 10 client.26749 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561b314fd0f0 con 0x561b318c4580 2024-03-08T23:13:09.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f600041c0 con 0x561b318c4580 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4f64097f40 con 0x561b318c4580 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f7c609740 10 client.26749 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.853+0000 7f4f7c609740 10 client.26749 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.854+0000 7f4f7c609740 20 client.26749 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.854+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b318ddbb0 0x561b318dffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.854+0000 7f4f7c609740 4 client.26749 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.854+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b31836290 con 0x561b318ddbb0 2024-03-08T23:13:09.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.854+0000 7f4f72000640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b318ddbb0 0x561b318dffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.855+0000 7f4f72000640 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b318ddbb0 0x561b318dffa0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a7060ac45f5ab24 server_cookie=a900f5c2580c0a9f in_seq=0 out_seq=0 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.855+0000 7f4f5cc00640 10 client.26749 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f5cc00640 1 -- 172.21.15.46:0/4033194016 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b31836290 con 0x561b318ddbb0 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f5cc00640 10 client.26749 handle_command_reply: tid=0 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 client.26749 shutdown 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 msgr2=0x561b318ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4f6409b860 tx=0x7f4f6409bfd0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 shutdown_connections 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b318ddbb0 0x561b318dffa0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b318cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b318caf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f5cc00640 0 client.26749 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 --2- 172.21.15.46:0/4033194016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318ca9f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.856+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 >> 172.21.15.46:0/4033194016 conn(0x561b318becb0 msgr2=0x561b318c0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.857+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 shutdown_connections 2024-03-08T23:13:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.857+0000 7f4f7c609740 1 -- 172.21.15.46:0/4033194016 wait complete. 2024-03-08T23:13:09.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.857+0000 7f4f7c609740 20 client.26749 trim_cache size 0 max 16384 2024-03-08T23:13:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 Processor -- start 2024-03-08T23:13:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 -- start start 2024-03-08T23:13:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 0x561b318d4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318d4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741b2c30 con 0x561b318c4ea0 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741b07a0 con 0x561b318cda80 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f7c609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741ae840 con 0x561b318c4580 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.882+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36798/0 (socket says 172.21.15.46:36798) 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 -- 172.21.15.46:0/3413370126 learned_addr learned my addr 172.21.15.46:0/3413370126 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f73400640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318d4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72000640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 0x561b318d4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 -- 172.21.15.46:0/3413370126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 msgr2=0x561b318d4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318d4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 -- 172.21.15.46:0/3413370126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 msgr2=0x561b318d4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 0x561b318d4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 -- 172.21.15.46:0/3413370126 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b318ba930 con 0x561b318c4580 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72000640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 0x561b318d4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.883+0000 7f4f72a00640 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4f640ccfc0 tx=0x7f4f64018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920d6f2f17c991f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.888 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.884+0000 7f4f5f400640 1 -- 172.21.15.46:0/3413370126 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6409b8d0 con 0x561b318c4580 2024-03-08T23:13:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.884+0000 7f4f5f400640 1 -- 172.21.15.46:0/3413370126 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f640a4070 con 0x561b318c4580 2024-03-08T23:13:09.889 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.884+0000 7f4f5f400640 1 -- 172.21.15.46:0/3413370126 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f640ccc40 con 0x561b318c4580 2024-03-08T23:13:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.884+0000 7f4f7c609740 1 -- 172.21.15.46:0/3413370126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 msgr2=0x561b318cd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.884+0000 7f4f7c609740 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4f640ccfc0 tx=0x7f4f64018420 comp rx=0 tx=0).stop 2024-03-08T23:13:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 -- 172.21.15.46:0/3413370126 shutdown_connections 2024-03-08T23:13:09.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318cda80 0x561b318d4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318c4ea0 0x561b318d4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 --2- 172.21.15.46:0/3413370126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b318cd120 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 -- 172.21.15.46:0/3413370126 >> 172.21.15.46:0/3413370126 conn(0x561b318becb0 msgr2=0x561b318c5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 -- 172.21.15.46:0/3413370126 shutdown_connections 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 -- 172.21.15.46:0/3413370126 wait complete. 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 -- start start 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.885+0000 7f4f7c609740 1 Processor -- start 2024-03-08T23:13:09.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b3192e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b31927fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741aee40 con 0x561b318cda80 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741b0140 con 0x561b318c4ea0 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741af480 con 0x561b318c4580 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f7c609740 10 client.0 _ll_register_callbacks cb 0x7f4f600095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f72a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:36800/0 (socket says 172.21.15.46:36800) 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f72a00640 1 -- 172.21.15.46:0/678434150 learned_addr learned my addr 172.21.15.46:0/678434150 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f72000640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b3192e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.886+0000 7f4f73400640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b31927fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 -- 172.21.15.46:0/678434150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 msgr2=0x561b3192dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 -- 172.21.15.46:0/678434150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 msgr2=0x561b31927fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b31927fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 -- 172.21.15.46:0/678434150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b319286b0 con 0x561b318c4ea0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72a00640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f72000640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b3192e0f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c00 tx=0x7f4f60004490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b24085ef8b3309c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.887+0000 7f4f5cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60004030 con 0x561b318c4ea0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f4f60005040 con 0x561b318c4ea0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f640a4070 con 0x561b318c4ea0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f60035050 con 0x561b318c4ea0 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f7c609740 10 client.26768 fetch_fsmap learned FSMap version 29 2024-03-08T23:13:09.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x561b31935330 con 0x561b318c4ea0 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f640a4070 con 0x561b318c4ea0 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.888+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f4f600039b0 con 0x561b318c4ea0 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 10 client.26768 fetch_fsmap finished waiting for FSMap version 29 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 10 client.26768 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 20 client.26768 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b31935800 0x561b31937c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 4 client.26768 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:13:09.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561b318305e0 con 0x561b31935800 2024-03-08T23:13:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.889+0000 7f4f72a00640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b31935800 0x561b31937c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f72a00640 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b31935800 0x561b31937c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=748ff8bddde8e0bb server_cookie=d9f49532290f6828 in_seq=0 out_seq=0 2024-03-08T23:13:09.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f5cc00640 10 client.26768 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f5cc00640 1 -- 172.21.15.46:0/678434150 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x561b318305e0 con 0x561b31935800 2024-03-08T23:13:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f5cc00640 10 client.26768 handle_command_reply: tid=0 2024-03-08T23:13:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 client.26768 shutdown 2024-03-08T23:13:09.899 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 msgr2=0x561b3192e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b3192e0f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009c00 tx=0x7f4f60004490 comp rx=0 tx=0).stop 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 shutdown_connections 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x561b31935800 0x561b31937c20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561b318cda80 0x561b31927fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561b318c4ea0 0x561b3192e0f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 --2- 172.21.15.46:0/678434150 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561b318c4580 0x561b3192dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.890+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 >> 172.21.15.46:0/678434150 conn(0x561b318becb0 msgr2=0x561b318ca990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.891+0000 7f4f5cc00640 0 client.26768 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.891+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 shutdown_connections 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.891+0000 7f4f7c609740 1 -- 172.21.15.46:0/678434150 wait complete. 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.891+0000 7f4f7c609740 20 client.26768 trim_cache size 0 max 16384 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:13:09.901 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: "81b0e481-750e-44fe-96f2-76b719514bd2": { 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "81b0e481-750e-44fe-96f2-76b719514bd2", 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:13:09.902 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 msgr2=0x7f4f580a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 0x7f4f580a4eb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c006010 tx=0x7f4f6c0095c0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 msgr2=0x7f4f74199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:09.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4f6400e9f0 tx=0x7f4f6400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 shutdown_connections 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4f580a29f0 0x7f4f580a4eb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4f7419e5e0 0x7f4f7419ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4f741065c0 0x7f4f7419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 --2- 172.21.15.46:0/1633682896 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4f741014f0 0x7f4f74199d20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 >> 172.21.15.46:0/1633682896 conn(0x7f4f740786f0 msgr2=0x7f4f740ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 shutdown_connections 2024-03-08T23:13:09.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:09.892+0000 7f4f79800640 1 -- 172.21.15.46:0/1633682896 wait complete. 2024-03-08T23:13:09.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:09 smithi046 ceph-mon[31859]: pgmap v164: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:09.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:09 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:09.910 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:09 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:09.912 INFO:tasks.cephfs.filesystem:scrub status for tag:81b0e481-750e-44fe-96f2-76b719514bd2 - {'path': '/', 'tag': '81b0e481-750e-44fe-96f2-76b719514bd2', 'options': 'recursive,force'} 2024-03-08T23:13:10.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:09 smithi062 ceph-mon[35761]: pgmap v164: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:10.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:09 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:10.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:09 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:10.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:09 smithi192 ceph-mon[36700]: pgmap v164: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:10.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:09 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:10.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:09 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:13:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:10 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:10 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:10 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:12.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:11 smithi062 ceph-mon[35761]: pgmap v165: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:12.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:11 smithi046 ceph-mon[31859]: pgmap v165: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:12.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:11 smithi192 ceph-mon[36700]: pgmap v165: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-03-08T23:13:14.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:13 smithi062 ceph-mon[35761]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:14.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:13 smithi046 ceph-mon[31859]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:14.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:13 smithi192 ceph-mon[36700]: pgmap v166: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-03-08T23:13:16.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:15 smithi062 ceph-mon[35761]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:16.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:15 smithi046 ceph-mon[31859]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:16.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:15 smithi192 ceph-mon[36700]: pgmap v167: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:18.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:17 smithi062 ceph-mon[35761]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:18.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:17 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:18.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:17 smithi046 ceph-mon[31859]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:18.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:17 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:18.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:17 smithi192 ceph-mon[36700]: pgmap v168: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-08T23:13:18.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:17 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:20.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:19 smithi062 ceph-mon[35761]: pgmap v169: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:13:20.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:19 smithi046 ceph-mon[31859]: pgmap v169: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:13:20.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:19 smithi192 ceph-mon[36700]: pgmap v169: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:13:22.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:21 smithi062 ceph-mon[35761]: pgmap v170: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:22.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:21 smithi046 ceph-mon[31859]: pgmap v170: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:22.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:21 smithi192 ceph-mon[36700]: pgmap v170: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:23.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:22 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:23.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:22 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:23.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:22 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:23.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:22 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:13:23.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:22 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:13:23.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:22 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:13:23.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:22 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:23.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:22 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:23.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:22 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:24.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:23 smithi046 ceph-mon[31859]: pgmap v171: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:24.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:23 smithi192 ceph-mon[36700]: pgmap v171: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:24.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:23 smithi062 ceph-mon[35761]: pgmap v171: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:26.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:25 smithi046 ceph-mon[31859]: pgmap v172: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:26.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:25 smithi192 ceph-mon[36700]: pgmap v172: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:26.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:25 smithi062 ceph-mon[35761]: pgmap v172: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:28.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:27 smithi046 ceph-mon[31859]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:28.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:27 smithi192 ceph-mon[36700]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:28.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:27 smithi062 ceph-mon[35761]: pgmap v173: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:30.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:29 smithi046 ceph-mon[31859]: pgmap v174: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:30.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:29 smithi192 ceph-mon[36700]: pgmap v174: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:30.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:29 smithi062 ceph-mon[35761]: pgmap v174: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:32.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:31 smithi046 ceph-mon[31859]: pgmap v175: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:32.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:31 smithi192 ceph-mon[36700]: pgmap v175: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:32.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:31 smithi062 ceph-mon[35761]: pgmap v175: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:34.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:33 smithi046 ceph-mon[31859]: pgmap v176: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:34.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:33 smithi192 ceph-mon[36700]: pgmap v176: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:34.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:33 smithi062 ceph-mon[35761]: pgmap v176: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:35.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:34 smithi046 ceph-mon[31859]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:35.292 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:34 smithi062 ceph-mon[35761]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:35.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:34 smithi192 ceph-mon[36700]: fsmap cephfs:3 {0=b=up:active,1=a=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-03-08T23:13:36.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:35 smithi046 ceph-mon[31859]: pgmap v177: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:36.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:35 smithi192 ceph-mon[36700]: pgmap v177: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:36.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:35 smithi062 ceph-mon[35761]: pgmap v177: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:38.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:37 smithi046 ceph-mon[31859]: pgmap v178: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:38.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:37 smithi192 ceph-mon[36700]: pgmap v178: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:38.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:37 smithi062 ceph-mon[35761]: pgmap v178: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:39.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:13:39.914 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:13:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 -- 172.21.15.46:0/710226761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444112010 msgr2=0x7fd444114400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 --2- 172.21.15.46:0/710226761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444112010 0x7fd444114400 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd43c009c10 tx=0x7fd43c02bee0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 -- 172.21.15.46:0/710226761 shutdown_connections 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 --2- 172.21.15.46:0/710226761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444112010 0x7fd444114400 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 --2- 172.21.15.46:0/710226761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444076140 0x7fd444111ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 --2- 172.21.15.46:0/710226761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444075820 0x7fd444075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 -- 172.21.15.46:0/710226761 >> 172.21.15.46:0/710226761 conn(0x7fd4440fee00 msgr2=0x7fd444101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 -- 172.21.15.46:0/710226761 shutdown_connections 2024-03-08T23:13:40.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.147+0000 7fd443e00640 1 -- 172.21.15.46:0/710226761 wait complete. 2024-03-08T23:13:40.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.148+0000 7fd443e00640 1 Processor -- start 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.148+0000 7fd443e00640 1 -- start start 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 0x7fd44419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444112010 0x7fd4441a2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444117bc0 con 0x7fd444076140 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444117a40 con 0x7fd444075820 2024-03-08T23:13:40.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd444117d40 con 0x7fd444112010 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd442a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 0x7fd44419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50760/0 (socket says 172.21.15.46:50760) 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd43be00640 1 -- 172.21.15.46:0/1715653147 learned_addr learned my addr 172.21.15.46:0/1715653147 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.149+0000 7fd443400640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444112010 0x7fd4441a2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 -- 172.21.15.46:0/1715653147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444112010 msgr2=0x7fd4441a2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444112010 0x7fd4441a2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 -- 172.21.15.46:0/1715653147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 msgr2=0x7fd44419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 0x7fd44419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 -- 172.21.15.46:0/1715653147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4441a3040 con 0x7fd444076140 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd442a00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 0x7fd44419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd43be00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e9f0 tx=0x7fd42c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd448a00640 1 -- 172.21.15.46:0/1715653147 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c00cd80 con 0x7fd444076140 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4441a3330 con 0x7fd444076140 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd448a00640 1 -- 172.21.15.46:0/1715653147 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd42c005500 con 0x7fd444076140 2024-03-08T23:13:40.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd448a00640 1 -- 172.21.15.46:0/1715653147 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c010690 con 0x7fd444076140 2024-03-08T23:13:40.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.150+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4441aabc0 con 0x7fd444076140 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd448a00640 1 -- 172.21.15.46:0/1715653147 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd42c0046c0 con 0x7fd444076140 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 Processor -- start 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 -- start start 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c79d410 con 0x55862c7e4c90 2024-03-08T23:13:40.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c73fd50 con 0x55862c7db790 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.154+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c6e8b80 con 0x55862c7dc0b0 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd442a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50766/0 (socket says 172.21.15.46:50766) 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40682/0 (socket says 172.21.15.46:40682) 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd43be00640 1 -- 172.21.15.46:0/3304655928 learned_addr learned my addr 172.21.15.46:0/3304655928 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 -- 172.21.15.46:0/3304655928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 msgr2=0x55862c7e4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd448a00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 0x7fd4280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 -- 172.21.15.46:0/3304655928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 msgr2=0x55862c7e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd443400640 1 -- 172.21.15.46:0/3304655928 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55862c51e7d0 con 0x55862c7db790 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd43be00640 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.155+0000 7fd448a00640 1 -- 172.21.15.46:0/1715653147 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd42c0cf5b0 con 0x7fd444076140 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.156+0000 7fd443400640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 0x7fd4280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.156+0000 7fd443400640 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd43c02c4f0 tx=0x7fd43c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3731778d447ab351 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.156+0000 7fd433e00640 1 -- 172.21.15.46:0/3304655928 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c043040 con 0x55862c7db790 2024-03-08T23:13:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.156+0000 7fd433e00640 1 -- 172.21.15.46:0/3304655928 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd43c02c880 con 0x55862c7db790 2024-03-08T23:13:40.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.156+0000 7fd433e00640 1 -- 172.21.15.46:0/3304655928 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43c004040 con 0x55862c7db790 2024-03-08T23:13:40.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 -- 172.21.15.46:0/3304655928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 msgr2=0x55862c7dbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd43c02c4f0 tx=0x7fd43c009840 comp rx=0 tx=0).stop 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd443400640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 0x7fd4280a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55862c7e76d0 tx=0x7fd43c04e000 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 -- 172.21.15.46:0/3304655928 shutdown_connections 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/3304655928 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7dbb70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 -- 172.21.15.46:0/3304655928 >> 172.21.15.46:0/3304655928 conn(0x55862c7d5ec0 msgr2=0x55862c7d62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 -- 172.21.15.46:0/3304655928 shutdown_connections 2024-03-08T23:13:40.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.157+0000 7fd44cb9a740 1 -- 172.21.15.46:0/3304655928 wait complete. 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 -- start start 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 Processor -- start 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7e1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c6fbee0 con 0x55862c7e4c90 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c73bdf0 con 0x55862c7db790 2024-03-08T23:13:40.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.158+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55862c7e7a80 con 0x55862c7dc0b0 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd44cb9a740 10 client.0 _ll_register_callbacks cb 0x55862c7028d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd443400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd443400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50770/0 (socket says 172.21.15.46:50770) 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd443400640 1 -- 172.21.15.46:0/1378023116 learned_addr learned my addr 172.21.15.46:0/1378023116 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd43be00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7e1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 -- 172.21.15.46:0/1378023116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 msgr2=0x55862c7e2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 -- 172.21.15.46:0/1378023116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 msgr2=0x55862c7e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd442a00640 1 -- 172.21.15.46:0/1378023116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55862c4543f0 con 0x55862c7db790 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.159+0000 7fd443400640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd442a00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7e1cc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd434009530 tx=0x7fd43400f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a62e36c3af6496a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd430c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43401e020 con 0x55862c7db790 2024-03-08T23:13:40.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd43c02c880 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd4340047a0 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd434026050 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd44cb9a740 10 client.26780 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55862c4140f0 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.160+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd43c02c880 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd434019070 con 0x55862c7db790 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 10 client.26780 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 10 client.26780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:40.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 20 client.26780 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c7f4ee0 0x55862c7f7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 4 client.26780 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55862c74d290 con 0x55862c7f4ee0 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.161+0000 7fd43be00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c7f4ee0 0x55862c7f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.162+0000 7fd43be00640 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c7f4ee0 0x55862c7f7310 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32b9cba8bced112f server_cookie=f3f077ee90e582cc in_seq=0 out_seq=0 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.162+0000 7fd430c00640 10 client.26780 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.162+0000 7fd430c00640 1 -- 172.21.15.46:0/1378023116 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55862c74d290 con 0x55862c7f4ee0 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.162+0000 7fd430c00640 10 client.26780 handle_command_reply: tid=0 2024-03-08T23:13:40.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 client.26780 shutdown 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 msgr2=0x55862c7e1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7e1cc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd434009530 tx=0x7fd43400f790 comp rx=0 tx=0).stop 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 shutdown_connections 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c7f4ee0 0x55862c7f7310 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7e4c90 0x55862c7e2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7dc0b0 0x55862c7e2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/1378023116 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c7e1cc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 >> 172.21.15.46:0/1378023116 conn(0x55862c7d5ec0 msgr2=0x55862c7d7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd430c00640 0 client.26780 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 shutdown_connections 2024-03-08T23:13:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 1 -- 172.21.15.46:0/1378023116 wait complete. 2024-03-08T23:13:40.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.163+0000 7fd44cb9a740 20 client.26780 trim_cache size 0 max 16384 2024-03-08T23:13:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 Processor -- start 2024-03-08T23:13:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 -- start start 2024-03-08T23:13:40.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c7e0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7e4c90 0x55862c7e09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b6dc0 con 0x55862c7dc0b0 2024-03-08T23:13:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b4960 con 0x55862c7e4c90 2024-03-08T23:13:40.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b2a00 con 0x55862c7db790 2024-03-08T23:13:40.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c7e0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd442a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd442a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43302/0 (socket says 172.21.15.46:43302) 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.180+0000 7fd442a00640 1 -- 172.21.15.46:0/2198490067 learned_addr learned my addr 172.21.15.46:0/2198490067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd443400640 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7e4c90 0x55862c7e09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 -- 172.21.15.46:0/2198490067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7e4c90 msgr2=0x55862c7e09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7e4c90 0x55862c7e09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 -- 172.21.15.46:0/2198490067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 msgr2=0x55862c7e0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c7e0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 -- 172.21.15.46:0/2198490067 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55862c7fc500 con 0x55862c7db790 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd442a00640 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd434019b00 tx=0x7fd434019b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b80d45e17e9b61aa server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd433400640 1 -- 172.21.15.46:0/2198490067 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43401e040 con 0x55862c7db790 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd433400640 1 -- 172.21.15.46:0/2198490067 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd434019370 con 0x55862c7db790 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.181+0000 7fd433400640 1 -- 172.21.15.46:0/2198490067 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd434005040 con 0x55862c7db790 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 -- 172.21.15.46:0/2198490067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 msgr2=0x55862c7e43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd434019b00 tx=0x7fd434019b30 comp rx=0 tx=0).stop 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 -- 172.21.15.46:0/2198490067 shutdown_connections 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7e4c90 0x55862c7e09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c7e0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/2198490067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7db790 0x55862c7e43f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 -- 172.21.15.46:0/2198490067 >> 172.21.15.46:0/2198490067 conn(0x55862c7d5ec0 msgr2=0x55862c7dcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 -- 172.21.15.46:0/2198490067 shutdown_connections 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.182+0000 7fd44cb9a740 1 -- 172.21.15.46:0/2198490067 wait complete. 2024-03-08T23:13:40.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 -- start start 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 Processor -- start 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7e4c90 0x55862c83f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b3000 con 0x55862c7dc0b0 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b4300 con 0x55862c7db790 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4441b3640 con 0x55862c7e4c90 2024-03-08T23:13:40.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.183+0000 7fd44cb9a740 10 client.0 _ll_register_callbacks cb 0x7fd42c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd442a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50790/0 (socket says 172.21.15.46:50790) 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 -- 172.21.15.46:0/4095046324 learned_addr learned my addr 172.21.15.46:0/4095046324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd442a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40720/0 (socket says 172.21.15.46:40720) 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd443400640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7e4c90 0x55862c83f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 -- 172.21.15.46:0/4095046324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7e4c90 msgr2=0x55862c83f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7e4c90 0x55862c83f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 -- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 msgr2=0x55862c844910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 -- 172.21.15.46:0/4095046324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55862c83fa70 con 0x55862c7dc0b0 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd442a00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd43be00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd42c098220 tx=0x7fd42c0d1a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4df14cbb033b81 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.184+0000 7fd430c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:40.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0b3070 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd42c0860b0 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd434019370 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd42c09b1b0 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd44cb9a740 10 client.16632 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55862c84c100 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd434019370 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.185+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd42c0ccbc0 con 0x55862c7dc0b0 2024-03-08T23:13:40.196 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 10 client.16632 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 10 client.16632 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 20 client.16632 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c84c830 0x55862c84ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 4 client.16632 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55862c7475e0 con 0x55862c84c830 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd442a00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c84c830 0x55862c84ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd442a00640 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c84c830 0x55862c84ec50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd91ccc467f19489 server_cookie=f2c02ffe8422f37 in_seq=0 out_seq=0 2024-03-08T23:13:40.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.186+0000 7fd430c00640 10 client.16632 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd430c00640 1 -- 172.21.15.46:0/4095046324 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55862c7475e0 con 0x55862c84c830 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd430c00640 10 client.16632 handle_command_reply: tid=0 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 client.16632 shutdown 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 msgr2=0x55862c844e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd42c098220 tx=0x7fd42c0d1a00 comp rx=0 tx=0).stop 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 shutdown_connections 2024-03-08T23:13:40.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55862c84c830 0x55862c84ec50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55862c7e4c90 0x55862c83f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55862c7dc0b0 0x55862c844e50 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 --2- 172.21.15.46:0/4095046324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55862c7db790 0x55862c844910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 >> 172.21.15.46:0/4095046324 conn(0x55862c7d5ec0 msgr2=0x55862c7e0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd430c00640 0 client.16632 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 shutdown_connections 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 1 -- 172.21.15.46:0/4095046324 wait complete. 2024-03-08T23:13:40.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.187+0000 7fd44cb9a740 20 client.16632 trim_cache size 0 max 16384 2024-03-08T23:13:40.200 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:40.200 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:13:40.200 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:13:40.200 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:40.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 msgr2=0x7fd4280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 0x7fd4280a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x55862c7e76d0 tx=0x7fd43c04e000 comp rx=0 tx=0).stop 2024-03-08T23:13:40.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 msgr2=0x7fd44419e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fd42c00e9f0 tx=0x7fd42c00edf0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 shutdown_connections 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd4280a2970 0x7fd4280a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd444112010 0x7fd4441a2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd444076140 0x7fd44419e5e0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 --2- 172.21.15.46:0/1715653147 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd444075820 0x7fd44419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 >> 172.21.15.46:0/1715653147 conn(0x7fd4440fee00 msgr2=0x7fd444113660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 shutdown_connections 2024-03-08T23:13:40.202 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.188+0000 7fd443e00640 1 -- 172.21.15.46:0/1715653147 wait complete. 2024-03-08T23:13:40.206 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:39 smithi046 ceph-mon[31859]: pgmap v179: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:40.209 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:13:40.209 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:13:40.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:39 smithi192 ceph-mon[36700]: pgmap v179: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:40.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:39 smithi062 ceph-mon[35761]: pgmap v179: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.429+0000 7f68c5200640 1 -- 172.21.15.46:0/84806529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c01063f0 msgr2=0x7f68c010dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.429+0000 7f68c5200640 1 --2- 172.21.15.46:0/84806529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c01063f0 0x7f68c010dfc0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f68b8009be0 tx=0x7f68b802bef0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 -- 172.21.15.46:0/84806529 shutdown_connections 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 --2- 172.21.15.46:0/84806529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c01063f0 0x7f68c010dfc0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 --2- 172.21.15.46:0/84806529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c0101cd0 0x7f68c0105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 --2- 172.21.15.46:0/84806529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0101320 0x7f68c0101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 -- 172.21.15.46:0/84806529 >> 172.21.15.46:0/84806529 conn(0x7f68c00fcbc0 msgr2=0x7f68c00fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 -- 172.21.15.46:0/84806529 shutdown_connections 2024-03-08T23:13:40.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.430+0000 7f68c5200640 1 -- 172.21.15.46:0/84806529 wait complete. 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 Processor -- start 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 -- start start 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0074070 0x7f68c01a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c01114e0 con 0x7f68c01063f0 2024-03-08T23:13:40.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0111360 con 0x7f68c0101320 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.431+0000 7f68c5200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c0111660 con 0x7f68c0074070 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50800/0 (socket says 172.21.15.46:50800) 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bf400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40742/0 (socket says 172.21.15.46:40742) 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 -- 172.21.15.46:0/2371488835 learned_addr learned my addr 172.21.15.46:0/2371488835 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bfe00640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0074070 0x7f68c01a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 -- 172.21.15.46:0/2371488835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0074070 msgr2=0x7f68c01a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0074070 0x7f68c01a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 -- 172.21.15.46:0/2371488835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 msgr2=0x7f68c00735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bea00640 1 -- 172.21.15.46:0/2371488835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c01a6270 con 0x7f68c01063f0 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.432+0000 7f68bf400640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68bea00640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e9a0 tx=0x7f68ac00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00cd80 con 0x7f68c01063f0 2024-03-08T23:13:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c01a6560 con 0x7f68c01063f0 2024-03-08T23:13:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68ac0035f0 con 0x7f68c01063f0 2024-03-08T23:13:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac010670 con 0x7f68c01063f0 2024-03-08T23:13:40.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.433+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c01a6b00 con 0x7f68c01063f0 2024-03-08T23:13:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.434+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c0068450 con 0x7f68c01063f0 2024-03-08T23:13:40.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.437+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f68ac0037a0 con 0x7f68c01063f0 2024-03-08T23:13:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.438+0000 7f68b7e00640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 0x7f68a40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.439+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f68ac0cf4c0 con 0x7f68c01063f0 2024-03-08T23:13:40.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.439+0000 7f68bf400640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 0x7f68a40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.439+0000 7f68bf400640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 0x7f68a40a4e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f68b00095f0 tx=0x7f68b0005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.440+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f68ac09bef0 con 0x7f68c01063f0 2024-03-08T23:13:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.683+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68c0074b20 con 0x7f68c01063f0 2024-03-08T23:13:40.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.684+0000 7f68b7e00640 1 -- 172.21.15.46:0/2371488835 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f68c0074b20 con 0x7f68c01063f0 2024-03-08T23:13:40.688 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:13:40.688 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:13:40.688 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:13:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 msgr2=0x7f68a40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 0x7f68a40a4e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f68b00095f0 tx=0x7f68b0005f90 comp rx=0 tx=0).stop 2024-03-08T23:13:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 msgr2=0x7f68c0073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f68ac00e9a0 tx=0x7f68ac00ee60 comp rx=0 tx=0).stop 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 shutdown_connections 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f68a40a29a0 0x7f68a40a4e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f68c0074070 0x7f68c01a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f68c01063f0 0x7f68c0073b30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 --2- 172.21.15.46:0/2371488835 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f68c0101320 0x7f68c00735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.687+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 >> 172.21.15.46:0/2371488835 conn(0x7f68c00fcbc0 msgr2=0x7f68c00fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.688+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 shutdown_connections 2024-03-08T23:13:40.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.688+0000 7f68c5200640 1 -- 172.21.15.46:0/2371488835 wait complete. 2024-03-08T23:13:40.713 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:40.713 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 -- 172.21.15.46:0/1408927545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f001065c0 msgr2=0x7f2f0010e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 --2- 172.21.15.46:0/1408927545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f001065c0 0x7f2f0010e190 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009ca0 tx=0x7f2ef802c100 comp rx=0 tx=0).stop 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 -- 172.21.15.46:0/1408927545 shutdown_connections 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 --2- 172.21.15.46:0/1408927545 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f001065c0 0x7f2f0010e190 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 --2- 172.21.15.46:0/1408927545 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f00101ea0 0x7f2f00105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 --2- 172.21.15.46:0/1408927545 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 0x7f2f001018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 -- 172.21.15.46:0/1408927545 >> 172.21.15.46:0/1408927545 conn(0x7f2f000786f0 msgr2=0x7f2f00078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 -- 172.21.15.46:0/1408927545 shutdown_connections 2024-03-08T23:13:40.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.943+0000 7f2f08000640 1 -- 172.21.15.46:0/1408927545 wait complete. 2024-03-08T23:13:40.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.944+0000 7f2f08000640 1 Processor -- start 2024-03-08T23:13:40.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.944+0000 7f2f08000640 1 -- start start 2024-03-08T23:13:40.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 0x7f2f000735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 0x7f2f001a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00111680 con 0x7f2f001065c0 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00111500 con 0x7f2f00074070 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f08000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f00111800 con 0x7f2f001014f0 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50818/0 (socket says 172.21.15.46:50818) 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f04c00640 1 -- 172.21.15.46:0/2266776892 learned_addr learned my addr 172.21.15.46:0/2266776892 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f06000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 0x7f2f001a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.945+0000 7f2f05600640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 0x7f2f000735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 -- 172.21.15.46:0/2266776892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 msgr2=0x7f2f000735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 0x7f2f000735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 -- 172.21.15.46:0/2266776892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 msgr2=0x7f2f001a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 0x7f2f001a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 -- 172.21.15.46:0/2266776892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f001a63d0 con 0x7f2f001065c0 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f06000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 0x7f2f001a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:40.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f04c00640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e9a0 tx=0x7f2eec00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2efe000640 1 -- 172.21.15.46:0/2266776892 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec00cd80 con 0x7f2f001065c0 2024-03-08T23:13:40.952 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2efe000640 1 -- 172.21.15.46:0/2266776892 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2eec0035f0 con 0x7f2f001065c0 2024-03-08T23:13:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001a66c0 con 0x7f2f001065c0 2024-03-08T23:13:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.946+0000 7f2efe000640 1 -- 172.21.15.46:0/2266776892 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec010670 con 0x7f2f001065c0 2024-03-08T23:13:40.953 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.947+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f001a6c10 con 0x7f2f001065c0 2024-03-08T23:13:40.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.950+0000 7f2f0ada8740 1 Processor -- start 2024-03-08T23:13:40.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.950+0000 7f2f0ada8740 1 -- start start 2024-03-08T23:13:40.954 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.950+0000 7f2efe000640 1 -- 172.21.15.46:0/2266776892 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2eec0037a0 con 0x7f2f001065c0 2024-03-08T23:13:40.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15a3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15ac850 0x55b5c15aec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c1507dd0 con 0x55b5c15a3350 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c14b0c70 con 0x55b5c15a3c70 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c1503e70 con 0x55b5c15ac850 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f06000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f05600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f06000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15a3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50826/0 (socket says 172.21.15.46:50826) 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f06000640 1 -- 172.21.15.46:0/886832953 learned_addr learned my addr 172.21.15.46:0/886832953 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.951+0000 7f2f04c00640 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15ac850 0x55b5c15aec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 -- 172.21.15.46:0/886832953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15ac850 msgr2=0x55b5c15aec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15ac850 0x55b5c15aec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 -- 172.21.15.46:0/886832953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 msgr2=0x55b5c15a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 -- 172.21.15.46:0/886832953 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5c156a2b0 con 0x55b5c15a3c70 2024-03-08T23:13:40.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.952+0000 7f2f05600640 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4004630 tx=0x7f2ef400bb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ed34f2eeac0b8c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2efe000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 0x7f2ee80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2efe000640 1 -- 172.21.15.46:0/2266776892 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2eec0cf3d0 con 0x7f2f001065c0 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2f06000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 0x7f2ee80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2ef2600640 1 -- 172.21.15.46:0/886832953 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4003bd0 con 0x55b5c15a3c70 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2ef2600640 1 -- 172.21.15.46:0/886832953 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ef4015070 con 0x55b5c15a3c70 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.953+0000 7f2ef2600640 1 -- 172.21.15.46:0/886832953 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4004d90 con 0x55b5c15a3c70 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 -- 172.21.15.46:0/886832953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 msgr2=0x55b5c15ac310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac310 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4004630 tx=0x7f2ef400bb20 comp rx=0 tx=0).stop 2024-03-08T23:13:40.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f06000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 0x7f2ee80a4e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009a60 tx=0x7f2ef8005d80 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 -- 172.21.15.46:0/886832953 shutdown_connections 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15ac850 0x55b5c15aec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac310 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/886832953 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.954+0000 7f2f0ada8740 1 -- 172.21.15.46:0/886832953 >> 172.21.15.46:0/886832953 conn(0x55b5c159da80 msgr2=0x55b5c159de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.955+0000 7f2f0ada8740 1 -- 172.21.15.46:0/886832953 shutdown_connections 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.955+0000 7f2f0ada8740 1 -- 172.21.15.46:0/886832953 wait complete. 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.955+0000 7f2f0ada8740 1 -- start start 2024-03-08T23:13:40.961 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.955+0000 7f2f0ada8740 1 Processor -- start 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3c70 0x55b5c15a9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15a9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c15af640 con 0x55b5c15a3c70 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c14c40c0 con 0x55b5c15ac850 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b5c15af7c0 con 0x55b5c15a3350 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f0ada8740 10 client.0 _ll_register_callbacks cb 0x55b5c14feec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:40.962 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3c70 0x55b5c15a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f05600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f05600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43386/0 (socket says 172.21.15.46:43386) 2024-03-08T23:13:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f05600640 1 -- 172.21.15.46:0/4118705689 learned_addr learned my addr 172.21.15.46:0/4118705689 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.956+0000 7f2f06000640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15a9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 -- 172.21.15.46:0/4118705689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 msgr2=0x55b5c15a9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15a9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 -- 172.21.15.46:0/4118705689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3c70 msgr2=0x55b5c15a9530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3c70 0x55b5c15a9530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 -- 172.21.15.46:0/4118705689 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5c121c3f0 con 0x55b5c15a3350 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2f05600640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400b5f0 tx=0x7f2ef400bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acce26cff1766692 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.957+0000 7f2ee6a00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4003de0 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2ef4011440 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2ef4015070 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ef4030050 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2f0ada8740 10 client.26791 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b5c12e67d0 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.958+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ef4015070 con 0x55b5c15a3350 2024-03-08T23:13:40.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2ef40257d0 con 0x55b5c15a3350 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 10 client.26791 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 10 client.26791 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 20 client.26791 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c15bc870 0x55b5c15bec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 4 client.26791 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:40.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5c14dc4f0 con 0x55b5c15bc870 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.959+0000 7f2f04c00640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c15bc870 0x55b5c15bec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.960+0000 7f2f04c00640 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c15bc870 0x55b5c15bec60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c7ffddc4be7505f0 server_cookie=940231c0e81a8827 in_seq=0 out_seq=0 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.960+0000 7f2ee6a00640 10 client.26791 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.960+0000 7f2ee6a00640 1 -- 172.21.15.46:0/4118705689 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b5c14dc4f0 con 0x55b5c15bc870 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.960+0000 7f2ee6a00640 10 client.26791 handle_command_reply: tid=0 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.960+0000 7f2f0ada8740 1 client.26791 shutdown 2024-03-08T23:13:40.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 msgr2=0x55b5c15ac1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400b5f0 tx=0x7f2ef400bfa0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 shutdown_connections 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c15bc870 0x55b5c15bec60 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15a9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3c70 0x55b5c15a9530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/4118705689 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15ac1a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2ee6a00640 0 client.26791 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 >> 172.21.15.46:0/4118705689 conn(0x55b5c159da80 msgr2=0x55b5c159f7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 shutdown_connections 2024-03-08T23:13:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 1 -- 172.21.15.46:0/4118705689 wait complete. 2024-03-08T23:13:40.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.961+0000 7f2f0ada8740 20 client.26791 trim_cache size 0 max 16384 2024-03-08T23:13:40.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:40 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2371488835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.974+0000 7f2f0ada8740 1 Processor -- start 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.974+0000 7f2f0ada8740 1 -- start start 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.974+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15abc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.974+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 0x55b5c15a6790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001b2c00 con 0x55b5c15ac850 2024-03-08T23:13:40.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001b07a0 con 0x55b5c15a3c70 2024-03-08T23:13:40.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001ae840 con 0x55b5c15a3350 2024-03-08T23:13:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f06000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 0x55b5c15a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f05600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15abc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f06000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 0x55b5c15a6790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50852/0 (socket says 172.21.15.46:50852) 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f05600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15abc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43394/0 (socket says 172.21.15.46:43394) 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f06000640 1 -- 172.21.15.46:0/302666418 learned_addr learned my addr 172.21.15.46:0/302666418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 -- 172.21.15.46:0/302666418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 msgr2=0x55b5c15abc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15abc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 -- 172.21.15.46:0/302666418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 msgr2=0x55b5c15a6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.981 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 0x55b5c15a6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.975+0000 7f2f04c00640 1 -- 172.21.15.46:0/302666418 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b5c1599340 con 0x55b5c15a3c70 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.976+0000 7f2f04c00640 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0cc760 tx=0x7f2eec086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b468ac25b67fe21 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.976+0000 7f2ef1c00640 1 -- 172.21.15.46:0/302666418 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0bb040 con 0x55b5c15a3c70 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.976+0000 7f2ef1c00640 1 -- 172.21.15.46:0/302666418 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2eec09bd70 con 0x55b5c15a3c70 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.976+0000 7f2ef1c00640 1 -- 172.21.15.46:0/302666418 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec099710 con 0x55b5c15a3c70 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- 172.21.15.46:0/302666418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 msgr2=0x55b5c15ac1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac1a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2eec0cc760 tx=0x7f2eec086040 comp rx=0 tx=0).stop 2024-03-08T23:13:40.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- 172.21.15.46:0/302666418 shutdown_connections 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15ac850 0x55b5c15a6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15a3c70 0x55b5c15ac1a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/302666418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3350 0x55b5c15abc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- 172.21.15.46:0/302666418 >> 172.21.15.46:0/302666418 conn(0x55b5c159da80 msgr2=0x55b5c15c3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- 172.21.15.46:0/302666418 shutdown_connections 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- 172.21.15.46:0/302666418 wait complete. 2024-03-08T23:13:40.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.977+0000 7f2f0ada8740 1 -- start start 2024-03-08T23:13:40.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 Processor -- start 2024-03-08T23:13:40.987 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3c70 0x55b5c15b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001aee40 con 0x55b5c15a3350 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001b0140 con 0x55b5c15ac850 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001af480 con 0x55b5c15a3c70 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f0ada8740 10 client.0 _ll_register_callbacks cb 0x7f2ef4004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f05600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f05600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50868/0 (socket says 172.21.15.46:50868) 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f05600640 1 -- 172.21.15.46:0/1100542415 learned_addr learned my addr 172.21.15.46:0/1100542415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f04c00640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3c70 0x55b5c15b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.978+0000 7f2f06000640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 -- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3c70 msgr2=0x55b5c15b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3c70 0x55b5c15b2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 -- 172.21.15.46:0/1100542415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 msgr2=0x55b5c15b2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15b2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 -- 172.21.15.46:0/1100542415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b5c1612c50 con 0x55b5c15a3350 2024-03-08T23:13:40.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f05600640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400bb60 tx=0x7f2ef4005ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321482e978ec0aaa server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2ee6a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4025430 con 0x55b5c15a3350 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2eec09bd70 con 0x55b5c15a3350 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.979+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7f2ef40255d0 con 0x55b5c15a3350 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ef4045050 con 0x55b5c15a3350 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 10 client.16656 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:40.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b5c1613f50 con 0x55b5c15a3350 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2eec09bd70 con 0x55b5c15a3350 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2ef4011c20 con 0x55b5c15a3350 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 10 client.16656 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 10 client.16656 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 20 client.16656 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c1614560 0x55b5c1616950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 4 client.16656 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.980+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b5c1515310 con 0x55b5c1614560 2024-03-08T23:13:40.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2f04c00640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c1614560 0x55b5c1616950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2f04c00640 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c1614560 0x55b5c1616950 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=686bea646824e1e8 server_cookie=2299bde4ac9d7857 in_seq=0 out_seq=0 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2ee6a00640 10 client.16656 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2ee6a00640 1 -- 172.21.15.46:0/1100542415 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b5c1515310 con 0x55b5c1614560 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2ee6a00640 10 client.16656 handle_command_reply: tid=0 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2f0ada8740 1 client.16656 shutdown 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 msgr2=0x55b5c15b2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.981+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400bb60 tx=0x7f2ef4005ae0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 shutdown_connections 2024-03-08T23:13:40.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55b5c1614560 0x55b5c1616950 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b5c15ac850 0x55b5c15b2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b5c15a3c70 0x55b5c15b2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2ee6a00640 0 client.16656 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 --2- 172.21.15.46:0/1100542415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b5c15a3350 0x55b5c15b2560 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 >> 172.21.15.46:0/1100542415 conn(0x55b5c159da80 msgr2=0x55b5c159c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 shutdown_connections 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 1 -- 172.21.15.46:0/1100542415 wait complete. 2024-03-08T23:13:40.993 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f0ada8740 20 client.16656 trim_cache size 0 max 16384 2024-03-08T23:13:40.994 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:40.994 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 msgr2=0x7f2ee80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 0x7f2ee80a4e60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2ef8009a60 tx=0x7f2ef8005d80 comp rx=0 tx=0).stop 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 msgr2=0x7f2f00073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.982+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2eec00e9a0 tx=0x7f2eec00ee60 comp rx=0 tx=0).stop 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 shutdown_connections 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2ee80a29a0 0x7f2ee80a4e60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2f00074070 0x7f2f001a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2f001065c0 0x7f2f00073b30 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 --2- 172.21.15.46:0/2266776892 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2f001014f0 0x7f2f000735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 >> 172.21.15.46:0/2266776892 conn(0x7f2f000786f0 msgr2=0x7f2f000ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 shutdown_connections 2024-03-08T23:13:40.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:40.983+0000 7f2f08000640 1 -- 172.21.15.46:0/2266776892 wait complete. 2024-03-08T23:13:41.003 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:41.003 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:13:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 -- 172.21.15.46:0/1062951397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 msgr2=0x7ffb8c107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 --2- 172.21.15.46:0/1062951397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c107d10 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ffb74009900 tx=0x7ffb7402bc40 comp rx=0 tx=0).stop 2024-03-08T23:13:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 -- 172.21.15.46:0/1062951397 shutdown_connections 2024-03-08T23:13:41.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 --2- 172.21.15.46:0/1062951397 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 --2- 172.21.15.46:0/1062951397 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c107d10 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 --2- 172.21.15.46:0/1062951397 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 0x7ffb8c10dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 -- 172.21.15.46:0/1062951397 >> 172.21.15.46:0/1062951397 conn(0x7ffb8c075830 msgr2=0x7ffb8c077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 -- 172.21.15.46:0/1062951397 shutdown_connections 2024-03-08T23:13:41.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.211+0000 7ffb92000640 1 -- 172.21.15.46:0/1062951397 wait complete. 2024-03-08T23:13:41.218 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.212+0000 7ffb92000640 1 Processor -- start 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 -- start start 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 0x7ffb8c1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c117d00 con 0x7ffb8c108250 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c117b80 con 0x7ffb8c107890 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb92000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c117e80 con 0x7ffb8c10d8a0 2024-03-08T23:13:41.219 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb83e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb83e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50896/0 (socket says 172.21.15.46:50896) 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.213+0000 7ffb83e00640 1 -- 172.21.15.46:0/2688203833 learned_addr learned my addr 172.21.15.46:0/2688203833 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb8b400640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb8be00640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 0x7ffb8c1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 -- 172.21.15.46:0/2688203833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 msgr2=0x7ffb8c1a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 0x7ffb8c1a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 -- 172.21.15.46:0/2688203833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 msgr2=0x7ffb8c19e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c19e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.220 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 -- 172.21.15.46:0/2688203833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c1a31b0 con 0x7ffb8c108250 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.214+0000 7ffb83e00640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffb74005ab0 tx=0x7ffb74004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.215+0000 7ffb88c00640 1 -- 172.21.15.46:0/2688203833 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74041070 con 0x7ffb8c108250 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.215+0000 7ffb88c00640 1 -- 172.21.15.46:0/2688203833 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb7402ccd0 con 0x7ffb8c108250 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.215+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c1a3440 con 0x7ffb8c108250 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.215+0000 7ffb88c00640 1 -- 172.21.15.46:0/2688203833 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb74003e20 con 0x7ffb8c108250 2024-03-08T23:13:41.221 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.215+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c1aad30 con 0x7ffb8c108250 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.218+0000 7ffb88c00640 1 -- 172.21.15.46:0/2688203833 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ffb74044070 con 0x7ffb8c108250 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.218+0000 7ffb94d84740 1 Processor -- start 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.218+0000 7ffb94d84740 1 -- start start 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69440310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69440850 0x555c69442c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c6939bdd0 con 0x555c69437350 2024-03-08T23:13:41.224 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c69344c70 con 0x555c69437c70 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb94d84740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c69397e70 con 0x555c69440850 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50900/0 (socket says 172.21.15.46:50900) 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb8be00640 1 -- 172.21.15.46:0/1896401542 learned_addr learned my addr 172.21.15.46:0/1896401542 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb8b400640 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69440310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.219+0000 7ffb83e00640 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69440850 0x555c69442c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 -- 172.21.15.46:0/1896401542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69440850 msgr2=0x555c69442c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.225 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69440850 0x555c69442c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 -- 172.21.15.46:0/1896401542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 msgr2=0x555c69440310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69440310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 -- 172.21.15.46:0/1896401542 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c693fe2b0 con 0x555c69437350 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb840039f0 tx=0x7ffb84003eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6501fe513b3f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb88c00640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 0x7ffb700a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb7be00640 1 -- 172.21.15.46:0/1896401542 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8401a470 con 0x555c69437350 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb7be00640 1 -- 172.21.15.46:0/1896401542 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84015070 con 0x555c69437350 2024-03-08T23:13:41.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb88c00640 1 -- 172.21.15.46:0/2688203833 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ffb740f0bc0 con 0x7ffb8c108250 2024-03-08T23:13:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.220+0000 7ffb7be00640 1 -- 172.21.15.46:0/1896401542 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb840108a0 con 0x555c69437350 2024-03-08T23:13:41.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb8b400640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 0x7ffb700a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 -- 172.21.15.46:0/1896401542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 msgr2=0x555c69437730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ffb840039f0 tx=0x7ffb84003eb0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 -- 172.21.15.46:0/1896401542 shutdown_connections 2024-03-08T23:13:41.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69440850 0x555c69442c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69440310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1896401542 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c69437730 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 -- 172.21.15.46:0/1896401542 >> 172.21.15.46:0/1896401542 conn(0x555c69431a80 msgr2=0x555c69431e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb8b400640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 0x7ffb700a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x555c69436e20 tx=0x7ffb7c00b390 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 -- 172.21.15.46:0/1896401542 shutdown_connections 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.221+0000 7ffb94d84740 1 -- 172.21.15.46:0/1896401542 wait complete. 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.222+0000 7ffb94d84740 1 -- start start 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.222+0000 7ffb94d84740 1 Processor -- start 2024-03-08T23:13:41.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437c70 0x555c6943d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 0x555c6943daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c69443640 con 0x555c69437350 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c693580c0 con 0x555c69440850 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c694437c0 con 0x555c69437c70 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb94d84740 10 client.0 _ll_register_callbacks cb 0x555c69392ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 0x555c6943daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50904/0 (socket says 172.21.15.46:50904) 2024-03-08T23:13:41.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 0x555c6943daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40830/0 (socket says 172.21.15.46:40830) 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 -- 172.21.15.46:0/1236077270 learned_addr learned my addr 172.21.15.46:0/1236077270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb83e00640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437c70 0x555c6943d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 -- 172.21.15.46:0/1236077270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437c70 msgr2=0x555c6943d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437c70 0x555c6943d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 -- 172.21.15.46:0/1236077270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 msgr2=0x555c6943daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 0x555c6943daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.223+0000 7ffb8b400640 1 -- 172.21.15.46:0/1236077270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c690b03f0 con 0x555c69437350 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb8b400640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c003cb0 tx=0x7ffb7c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147a9beaa93a67f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb78c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb7c02c060 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ffb7c017030 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb84015070 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb7c003ce0 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb94d84740 10 client.16668 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.224+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555c6917a7d0 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84015070 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb7c011510 con 0x555c69437350 2024-03-08T23:13:41.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 10 client.16668 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 10 client.16668 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 20 client.16668 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c69450950 0x555c69452d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 4 client.16668 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c693704f0 con 0x555c69450950 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.225+0000 7ffb83e00640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c69450950 0x555c69452d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.226+0000 7ffb83e00640 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c69450950 0x555c69452d40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c1d93426ae7c1df5 server_cookie=1ec881c3005cac98 in_seq=0 out_seq=0 2024-03-08T23:13:41.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.226+0000 7ffb78c00640 10 client.16668 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.226+0000 7ffb78c00640 1 -- 172.21.15.46:0/1236077270 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c693704f0 con 0x555c69450950 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.226+0000 7ffb78c00640 10 client.16668 handle_command_reply: tid=0 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.226+0000 7ffb94d84740 1 client.16668 shutdown 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 msgr2=0x555c694401e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ffb7c003cb0 tx=0x7ffb7c004730 comp rx=0 tx=0).stop 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 shutdown_connections 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69440850 0x555c6943daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c69450950 0x555c69452d40 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437c70 0x555c6943d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1236077270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69437350 0x555c694401e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 >> 172.21.15.46:0/1236077270 conn(0x555c69431a80 msgr2=0x555c694337d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb78c00640 0 client.16668 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 shutdown_connections 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 1 -- 172.21.15.46:0/1236077270 wait complete. 2024-03-08T23:13:41.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.227+0000 7ffb94d84740 20 client.16668 trim_cache size 0 max 16384 2024-03-08T23:13:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 Processor -- start 2024-03-08T23:13:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 -- start start 2024-03-08T23:13:41.244 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c6943d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69431e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b6fb0 con 0x555c69440850 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b4b50 con 0x555c69437c70 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.240+0000 7ffb94d84740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b2bf0 con 0x555c69437350 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50912/0 (socket says 172.21.15.46:50912) 2024-03-08T23:13:41.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 -- 172.21.15.46:0/1121474478 learned_addr learned my addr 172.21.15.46:0/1121474478 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb83e00640 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69431e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8b400640 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c6943d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 -- 172.21.15.46:0/1121474478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 msgr2=0x555c6943d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c6943d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 -- 172.21.15.46:0/1121474478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 msgr2=0x555c69431e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69431e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.241+0000 7ffb8be00640 1 -- 172.21.15.46:0/1121474478 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c694530b0 con 0x555c69440850 2024-03-08T23:13:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb8be00640 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffb840039f0 tx=0x7ffb84009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d214b9d9552259e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb7b400640 1 -- 172.21.15.46:0/1121474478 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84004a00 con 0x555c69440850 2024-03-08T23:13:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb7b400640 1 -- 172.21.15.46:0/1121474478 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84015040 con 0x555c69440850 2024-03-08T23:13:41.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb7b400640 1 -- 172.21.15.46:0/1121474478 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb840116a0 con 0x555c69440850 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb94d84740 1 -- 172.21.15.46:0/1121474478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 msgr2=0x555c6943fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.242+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ffb840039f0 tx=0x7ffb84009940 comp rx=0 tx=0).stop 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 -- 172.21.15.46:0/1121474478 shutdown_connections 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c6943fca0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c69431e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 --2- 172.21.15.46:0/1121474478 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c6943d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 -- 172.21.15.46:0/1121474478 >> 172.21.15.46:0/1121474478 conn(0x555c69431a80 msgr2=0x555c6943ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 -- 172.21.15.46:0/1121474478 shutdown_connections 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 -- 172.21.15.46:0/1121474478 wait complete. 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 -- start start 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.243+0000 7ffb94d84740 1 Processor -- start 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c69447d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c694460e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b31f0 con 0x555c69440850 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b44f0 con 0x555c69437c70 2024-03-08T23:13:41.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c1b3830 con 0x555c69437350 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb94d84740 10 client.0 _ll_register_callbacks cb 0x7ffb84009ae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb8be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50922/0 (socket says 172.21.15.46:50922) 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb8be00640 1 -- 172.21.15.46:0/4280921571 learned_addr learned my addr 172.21.15.46:0/4280921571 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb83e00640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c694460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.244+0000 7ffb8b400640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c69447d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 -- 172.21.15.46:0/4280921571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 msgr2=0x555c69447d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c69447d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 -- 172.21.15.46:0/4280921571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 msgr2=0x555c694460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c694460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 -- 172.21.15.46:0/4280921571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c69442fb0 con 0x555c69440850 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb83e00640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c694460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8b400640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c69447d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb8be00640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb84005280 tx=0x7ffb840052b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8200c00d76807ba6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.254 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb78c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84004a00 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7ffb84004ba0 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.245+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb84015040 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb8403b050 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 10 client.16674 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555c694a7b80 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb84015040 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb84025070 con 0x555c69440850 2024-03-08T23:13:41.255 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 10 client.16674 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 10 client.16674 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 20 client.16674 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c694a7f50 0x555c694aa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 4 client.16674 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.246+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555c693a9310 con 0x555c694a7f50 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb8b400640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c694a7f50 0x555c694aa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb8b400640 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c694a7f50 0x555c694aa340 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9eb13c91913ca687 server_cookie=b928663fb203199e in_seq=0 out_seq=0 2024-03-08T23:13:41.256 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb78c00640 10 client.16674 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:41.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb78c00640 1 -- 172.21.15.46:0/4280921571 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555c693a9310 con 0x555c694a7f50 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb78c00640 10 client.16674 handle_command_reply: tid=0 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb94d84740 1 client.16674 shutdown 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.247+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 msgr2=0x555c69446620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ffb84005280 tx=0x7ffb840052b0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 shutdown_connections 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555c69440850 0x555c69446620 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555c69437c70 0x555c694460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x555c694a7f50 0x555c694aa340 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 --2- 172.21.15.46:0/4280921571 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555c69437350 0x555c69447d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 >> 172.21.15.46:0/4280921571 conn(0x555c69431a80 msgr2=0x555c6943ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb78c00640 0 client.16674 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 shutdown_connections 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 1 -- 172.21.15.46:0/4280921571 wait complete. 2024-03-08T23:13:41.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.248+0000 7ffb94d84740 20 client.16674 trim_cache size 0 max 16384 2024-03-08T23:13:41.260 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:41.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 msgr2=0x7ffb700a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 0x7ffb700a4e30 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x555c69436e20 tx=0x7ffb7c00b390 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 msgr2=0x7ffb8c19e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7ffb74005ab0 tx=0x7ffb74004430 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 shutdown_connections 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffb700a2970 0x7ffb700a4e30 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffb8c10d8a0 0x7ffb8c1a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffb8c108250 0x7ffb8c19e700 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 --2- 172.21.15.46:0/2688203833 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffb8c107890 0x7ffb8c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 >> 172.21.15.46:0/2688203833 conn(0x7ffb8c075830 msgr2=0x7ffb8c075f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 shutdown_connections 2024-03-08T23:13:41.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.249+0000 7ffb92000640 1 -- 172.21.15.46:0/2688203833 wait complete. 2024-03-08T23:13:41.272 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:13:41.272 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:13:41.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:40 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2371488835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:41.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:40 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2371488835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:13:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 -- 172.21.15.46:0/2983202624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c0736c0 msgr2=0x7fa56c073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 --2- 172.21.15.46:0/2983202624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c0736c0 0x7fa56c073aa0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009860 tx=0x7fa55c02bc10 comp rx=0 tx=0).stop 2024-03-08T23:13:41.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 -- 172.21.15.46:0/2983202624 shutdown_connections 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 --2- 172.21.15.46:0/2983202624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c10ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 --2- 172.21.15.46:0/2983202624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c073fe0 0x7fa56c10d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 --2- 172.21.15.46:0/2983202624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c0736c0 0x7fa56c073aa0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.489+0000 7fa570e00640 1 -- 172.21.15.46:0/2983202624 >> 172.21.15.46:0/2983202624 conn(0x7fa56c0786f0 msgr2=0x7fa56c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.490+0000 7fa570e00640 1 -- 172.21.15.46:0/2983202624 shutdown_connections 2024-03-08T23:13:41.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.490+0000 7fa570e00640 1 -- 172.21.15.46:0/2983202624 wait complete. 2024-03-08T23:13:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 Processor -- start 2024-03-08T23:13:41.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 -- start start 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 0x7fa56c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 0x7fa56c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1138b0 con 0x7fa56c10db20 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c113730 con 0x7fa56c073fe0 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.491+0000 7fa570e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c113a30 con 0x7fa56c0736c0 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa569e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 0x7fa56c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50934/0 (socket says 172.21.15.46:50934) 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 -- 172.21.15.46:0/3357976625 learned_addr learned my addr 172.21.15.46:0/3357976625 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa563e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 0x7fa56c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 -- 172.21.15.46:0/3357976625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 msgr2=0x7fa56c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 0x7fa56c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 -- 172.21.15.46:0/3357976625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 msgr2=0x7fa56c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 0x7fa56c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa56a800640 1 -- 172.21.15.46:0/3357976625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c19eda0 con 0x7fa56c10db20 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa569e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 0x7fa56c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.492+0000 7fa563e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 0x7fa56c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa56a800640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ef30 tx=0x7fa56400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa562a00640 1 -- 172.21.15.46:0/3357976625 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564004b20 con 0x7fa56c10db20 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa562a00640 1 -- 172.21.15.46:0/3357976625 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa564003e50 con 0x7fa56c10db20 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c19f090 con 0x7fa56c10db20 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa562a00640 1 -- 172.21.15.46:0/3357976625 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56400b570 con 0x7fa56c10db20 2024-03-08T23:13:41.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.493+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c1a6920 con 0x7fa56c10db20 2024-03-08T23:13:41.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.496+0000 7fa573a7c740 1 Processor -- start 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa562a00640 1 -- 172.21.15.46:0/3357976625 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa56401d020 con 0x7fa56c10db20 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 -- start start 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5ed730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b551dd0 con 0x562c2b5f6850 2024-03-08T23:13:41.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b4fac70 con 0x562c2b5ed350 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.497+0000 7fa573a7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b54de70 con 0x562c2b5edc70 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa563e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa563e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50936/0 (socket says 172.21.15.46:50936) 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43464/0 (socket says 172.21.15.46:43464) 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa563e00640 1 -- 172.21.15.46:0/303990750 learned_addr learned my addr 172.21.15.46:0/303990750 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa563e00640 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5ed730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 -- 172.21.15.46:0/303990750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 msgr2=0x562c2b5ed730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5ed730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 -- 172.21.15.46:0/303990750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 msgr2=0x562c2b5f8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 -- 172.21.15.46:0/303990750 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c2b5b42b0 con 0x562c2b5edc70 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.498+0000 7fa569e00640 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa55c037040 tx=0x7fa55c0056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e02b4f2ef61fddc server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa562a00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 0x7fa5500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa55b400640 1 -- 172.21.15.46:0/303990750 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c044020 con 0x562c2b5edc70 2024-03-08T23:13:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa55b400640 1 -- 172.21.15.46:0/303990750 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa55c034920 con 0x562c2b5edc70 2024-03-08T23:13:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa55b400640 1 -- 172.21.15.46:0/303990750 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c039040 con 0x562c2b5edc70 2024-03-08T23:13:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa562a00640 1 -- 172.21.15.46:0/3357976625 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa5640cefb0 con 0x7fa56c10db20 2024-03-08T23:13:41.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.499+0000 7fa563e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 0x7fa5500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- 172.21.15.46:0/303990750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 msgr2=0x562c2b5f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa563e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 0x7fa5500a4eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x562c2b5e8c10 tx=0x7fa55400bb90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fa55c037040 tx=0x7fa55c0056d0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- 172.21.15.46:0/303990750 shutdown_connections 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f6310 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 --2- 172.21.15.46:0/303990750 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5ed730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- 172.21.15.46:0/303990750 >> 172.21.15.46:0/303990750 conn(0x562c2b5e7a80 msgr2=0x562c2b5e7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- 172.21.15.46:0/303990750 shutdown_connections 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- 172.21.15.46:0/303990750 wait complete. 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.500+0000 7fa573a7c740 1 -- start start 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 Processor -- start 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b5f95e0 con 0x562c2b5f6850 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b50e0c0 con 0x562c2b5ed350 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562c2b5f9760 con 0x562c2b5edc70 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa573a7c740 10 client.0 _ll_register_callbacks cb 0x562c2b548ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50940/0 (socket says 172.21.15.46:50940) 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa56a800640 1 -- 172.21.15.46:0/1664709157 learned_addr learned my addr 172.21.15.46:0/1664709157 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa563e00640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.501+0000 7fa569e00640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 -- 172.21.15.46:0/1664709157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 msgr2=0x562c2b5f3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 -- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 msgr2=0x562c2b5f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 -- 172.21.15.46:0/1664709157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c2b2663f0 con 0x562c2b5f6850 2024-03-08T23:13:41.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa56a800640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5640a34c0 tx=0x7fa56409ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f012424ebf1305 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa54fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56401a040 con 0x562c2b5f6850 2024-03-08T23:13:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa5640978c0 con 0x562c2b5f6850 2024-03-08T23:13:41.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.502+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa55c034920 con 0x562c2b5f6850 2024-03-08T23:13:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa56409f160 con 0x562c2b5f6850 2024-03-08T23:13:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 10 client.16686 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562c2b3307d0 con 0x562c2b5f6850 2024-03-08T23:13:41.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa55c034920 con 0x562c2b5f6850 2024-03-08T23:13:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa56400b040 con 0x562c2b5f6850 2024-03-08T23:13:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 10 client.16686 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 10 client.16686 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:41.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 20 client.16686 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b606870 0x562c2b608ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 4 client.16686 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.503+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562c2b5264f0 con 0x562c2b606870 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.504+0000 7fa569e00640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b606870 0x562c2b608ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.504+0000 7fa569e00640 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b606870 0x562c2b608ca0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db08c66dce1716f6 server_cookie=4f25291364c431eb in_seq=0 out_seq=0 2024-03-08T23:13:41.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.504+0000 7fa54fe00640 10 client.16686 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.505+0000 7fa54fe00640 1 -- 172.21.15.46:0/1664709157 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562c2b5264f0 con 0x562c2b606870 2024-03-08T23:13:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.505+0000 7fa54fe00640 10 client.16686 handle_command_reply: tid=0 2024-03-08T23:13:41.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.505+0000 7fa573a7c740 1 client.16686 shutdown 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 msgr2=0x562c2b5f3ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fa5640a34c0 tx=0x7fa56409ba90 comp rx=0 tx=0).stop 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 shutdown_connections 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b606870 0x562c2b608ca0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f3ae0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1664709157 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa54fe00640 0 client.16686 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 >> 172.21.15.46:0/1664709157 conn(0x562c2b5e7a80 msgr2=0x562c2b5e97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 shutdown_connections 2024-03-08T23:13:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 1 -- 172.21.15.46:0/1664709157 wait complete. 2024-03-08T23:13:41.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.506+0000 7fa573a7c740 20 client.16686 trim_cache size 0 max 16384 2024-03-08T23:13:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.519+0000 7fa573a7c740 1 Processor -- start 2024-03-08T23:13:41.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.519+0000 7fa573a7c740 1 -- start start 2024-03-08T23:13:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.519+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.519+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa573a7c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1b2ba0 con 0x562c2b5f6850 2024-03-08T23:13:41.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa573a7c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1b0740 con 0x562c2b5ed350 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa573a7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1ae7e0 con 0x562c2b5edc70 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40876/0 (socket says 172.21.15.46:40876) 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 -- 172.21.15.46:0/1297287601 learned_addr learned my addr 172.21.15.46:0/1297287601 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa56a800640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa563e00640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 -- 172.21.15.46:0/1297287601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 msgr2=0x562c2b5f61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 -- 172.21.15.46:0/1297287601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 msgr2=0x562c2b5f07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa569e00640 1 -- 172.21.15.46:0/1297287601 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562c2b5f3db0 con 0x562c2b5ed350 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.520+0000 7fa563e00640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa569e00640 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02c850 tx=0x7fa55c02bc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d3fc53b965d1700 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa55aa00640 1 -- 172.21.15.46:0/1297287601 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c044040 con 0x562c2b5ed350 2024-03-08T23:13:41.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa55aa00640 1 -- 172.21.15.46:0/1297287601 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa55c051be0 con 0x562c2b5ed350 2024-03-08T23:13:41.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa55aa00640 1 -- 172.21.15.46:0/1297287601 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c039040 con 0x562c2b5ed350 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 -- 172.21.15.46:0/1297287601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 msgr2=0x562c2b5f5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fa55c02c850 tx=0x7fa55c02bc10 comp rx=0 tx=0).stop 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 -- 172.21.15.46:0/1297287601 shutdown_connections 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b5f07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5edc70 0x562c2b5f61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.521+0000 7fa573a7c740 1 --2- 172.21.15.46:0/1297287601 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5ed350 0x562c2b5f5c90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.522+0000 7fa573a7c740 1 -- 172.21.15.46:0/1297287601 >> 172.21.15.46:0/1297287601 conn(0x562c2b5e7a80 msgr2=0x562c2b5f1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.522+0000 7fa573a7c740 1 -- 172.21.15.46:0/1297287601 shutdown_connections 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.522+0000 7fa573a7c740 1 -- 172.21.15.46:0/1297287601 wait complete. 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.522+0000 7fa573a7c740 1 -- start start 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.522+0000 7fa573a7c740 1 Processor -- start 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5ed350 0x562c2b5fcae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5edc70 0x562c2b5fd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1aede0 con 0x562c2b5f6850 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1b00e0 con 0x562c2b5edc70 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1af420 con 0x562c2b5ed350 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa573a7c740 10 client.0 _ll_register_callbacks cb 0x7fa564009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa563e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5edc70 0x562c2b5fd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50964/0 (socket says 172.21.15.46:50964) 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 -- 172.21.15.46:0/4265242009 learned_addr learned my addr 172.21.15.46:0/4265242009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa569e00640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5ed350 0x562c2b5fcae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 -- 172.21.15.46:0/4265242009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5ed350 msgr2=0x562c2b5fcae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5ed350 0x562c2b5fcae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 -- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5edc70 msgr2=0x562c2b5fd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5edc70 0x562c2b5fd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.523+0000 7fa56a800640 1 -- 172.21.15.46:0/4265242009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562c2b65d310 con 0x562c2b5f6850 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa56a800640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ef00 tx=0x7fa564004350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d583bf6a2367cd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa54fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56401a040 con 0x562c2b5f6850 2024-03-08T23:13:41.534 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fa5640978c0 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa55c051be0 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa56409b020 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa573a7c740 10 client.16692 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.524+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562c2b65e4f0 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa55c051be0 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa564096fa0 con 0x562c2b5f6850 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 10 client.16692 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:41.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 10 client.16692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 20 client.16692 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b65ebb0 0x562c2b660fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 4 client.16692 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562c2b55f310 con 0x562c2b65ebb0 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.525+0000 7fa569e00640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b65ebb0 0x562c2b660fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa569e00640 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b65ebb0 0x562c2b660fd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b9a20617920f74d server_cookie=f846360331f33851 in_seq=0 out_seq=0 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa54fe00640 10 client.16692 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa54fe00640 1 -- 172.21.15.46:0/4265242009 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562c2b55f310 con 0x562c2b65ebb0 2024-03-08T23:13:41.536 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa54fe00640 10 client.16692 handle_command_reply: tid=0 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 client.16692 shutdown 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 msgr2=0x562c2b65cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ef00 tx=0x7fa564004350 comp rx=0 tx=0).stop 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 shutdown_connections 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562c2b65ebb0 0x562c2b660fd0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562c2b5f6850 0x562c2b65cdd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562c2b5edc70 0x562c2b5fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.537 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 --2- 172.21.15.46:0/4265242009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562c2b5ed350 0x562c2b5fcae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa54fe00640 0 client.16692 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 >> 172.21.15.46:0/4265242009 conn(0x562c2b5e7a80 msgr2=0x562c2b5e6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 shutdown_connections 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 1 -- 172.21.15.46:0/4265242009 wait complete. 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.526+0000 7fa573a7c740 20 client.16692 trim_cache size 0 max 16384 2024-03-08T23:13:41.538 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:13:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 msgr2=0x7fa5500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 0x7fa5500a4eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x562c2b5e8c10 tx=0x7fa55400bb90 comp rx=0 tx=0).stop 2024-03-08T23:13:41.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 msgr2=0x7fa56c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa56400ef30 tx=0x7fa56400c800 comp rx=0 tx=0).stop 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 shutdown_connections 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa5500a29f0 0x7fa5500a4eb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa56c10db20 0x7fa56c19e670 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa56c073fe0 0x7fa56c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 --2- 172.21.15.46:0/3357976625 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa56c0736c0 0x7fa56c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 >> 172.21.15.46:0/3357976625 conn(0x7fa56c0786f0 msgr2=0x7fa56c0fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:41.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 shutdown_connections 2024-03-08T23:13:41.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:41.527+0000 7fa570e00640 1 -- 172.21.15.46:0/3357976625 wait complete. 2024-03-08T23:13:42.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:41 smithi046 ceph-mon[31859]: pgmap v180: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:42.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:41 smithi192 ceph-mon[36700]: pgmap v180: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:42.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:41 smithi062 ceph-mon[35761]: pgmap v180: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:42.548 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:13:42.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:13:42.549 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:13:42.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.785+0000 7fd668600640 1 -- 172.21.15.46:0/2872159851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 msgr2=0x7fd6601018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.785+0000 7fd668600640 1 --2- 172.21.15.46:0/2872159851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd6601018d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fd654009860 tx=0x7fd65402bb70 comp rx=0 tx=0).stop 2024-03-08T23:13:42.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 -- 172.21.15.46:0/2872159851 shutdown_connections 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 --2- 172.21.15.46:0/2872159851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601065c0 0x7fd66010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 --2- 172.21.15.46:0/2872159851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd660101ea0 0x7fd660105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 --2- 172.21.15.46:0/2872159851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd6601018d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 -- 172.21.15.46:0/2872159851 >> 172.21.15.46:0/2872159851 conn(0x7fd6600786f0 msgr2=0x7fd660078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 -- 172.21.15.46:0/2872159851 shutdown_connections 2024-03-08T23:13:42.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.786+0000 7fd668600640 1 -- 172.21.15.46:0/2872159851 wait complete. 2024-03-08T23:13:42.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.787+0000 7fd668600640 1 Processor -- start 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.787+0000 7fd668600640 1 -- start start 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd66019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601979e0 0x7fd660197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660111680 con 0x7fd6601014f0 2024-03-08T23:13:42.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660111500 con 0x7fd6601065c0 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd668600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd660111800 con 0x7fd6601979e0 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd66019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40910/0 (socket says 172.21.15.46:40910) 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 -- 172.21.15.46:0/3410359384 learned_addr learned my addr 172.21.15.46:0/3410359384 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd666600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601979e0 0x7fd660197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 -- 172.21.15.46:0/3410359384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601979e0 msgr2=0x7fd660197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601979e0 0x7fd660197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 -- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 msgr2=0x7fd66019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd66019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665200640 1 -- 172.21.15.46:0/3410359384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd660198500 con 0x7fd6601065c0 2024-03-08T23:13:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.788+0000 7fd665c00640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd66019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd665200640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e9a0 tx=0x7fd64c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd65e000640 1 -- 172.21.15.46:0/3410359384 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c00cd80 con 0x7fd6601065c0 2024-03-08T23:13:42.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6601987f0 con 0x7fd6601065c0 2024-03-08T23:13:42.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd65e000640 1 -- 172.21.15.46:0/3410359384 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64c0035f0 con 0x7fd6601065c0 2024-03-08T23:13:42.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd65e000640 1 -- 172.21.15.46:0/3410359384 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c010670 con 0x7fd6601065c0 2024-03-08T23:13:42.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.789+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6601a4a40 con 0x7fd6601065c0 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.793+0000 7fd66b3a0740 1 Processor -- start 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 -- start start 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466655a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564546665ae0 0x564546667ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645465c0a50 con 0x56454665c5e0 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564546569b00 con 0x56454665cf00 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645465bcaf0 con 0x564546665ae0 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd65e000640 1 -- 172.21.15.46:0/3410359384 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd64c0037a0 con 0x7fd6601065c0 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd666600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd666600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50988/0 (socket says 172.21.15.46:50988) 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd666600640 1 -- 172.21.15.46:0/3403575009 learned_addr learned my addr 172.21.15.46:0/3403575009 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.794+0000 7fd665c00640 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466655a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 -- 172.21.15.46:0/3403575009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564546665ae0 msgr2=0x564546667ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564546665ae0 0x564546667ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 -- 172.21.15.46:0/3403575009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 msgr2=0x5645466655a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466655a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 -- 172.21.15.46:0/3403575009 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645465871e0 con 0x56454665c5e0 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd666600640 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd65800f400 tx=0x7fd65800f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e4b7f16040d710 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.795+0000 7fd652a00640 1 -- 172.21.15.46:0/3403575009 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65801d070 con 0x56454665c5e0 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd652a00640 1 -- 172.21.15.46:0/3403575009 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd65800cb20 con 0x56454665c5e0 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd652a00640 1 -- 172.21.15.46:0/3403575009 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd658018c80 con 0x56454665c5e0 2024-03-08T23:13:42.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd65e000640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 0x7fd6480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd65e000640 1 -- 172.21.15.46:0/3410359384 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd64c0cf4d0 con 0x7fd6601065c0 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3403575009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 msgr2=0x56454665c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd65800f400 tx=0x7fd65800f8c0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd665c00640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 0x7fd6480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3403575009 shutdown_connections 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564546665ae0 0x564546667ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466655a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3403575009 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665c5e0 0x56454665c9c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3403575009 >> 172.21.15.46:0/3403575009 conn(0x564546656d10 msgr2=0x5645466570f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.796+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3403575009 shutdown_connections 2024-03-08T23:13:42.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.797+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3403575009 wait complete. 2024-03-08T23:13:42.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.797+0000 7fd66b3a0740 1 -- start start 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.797+0000 7fd665c00640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 0x7fd6480a4eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd654009830 tx=0x7fd65402e040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.797+0000 7fd66b3a0740 1 Processor -- start 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x564546662b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564546665ae0 0x564546663600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645466688d0 con 0x564546665ae0 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56454657cf60 con 0x56454665cf00 2024-03-08T23:13:42.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564546668a50 con 0x56454665c5e0 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd66b3a0740 10 client.0 _ll_register_callbacks cb 0x564546568b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd665200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd666600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564546665ae0 0x564546663600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd665200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:40918/0 (socket says 172.21.15.46:40918) 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd665200640 1 -- 172.21.15.46:0/50663761 learned_addr learned my addr 172.21.15.46:0/50663761 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.798+0000 7fd665c00640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x564546662b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 -- 172.21.15.46:0/50663761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 msgr2=0x564546662b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x564546662b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 -- 172.21.15.46:0/50663761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564546665ae0 msgr2=0x564546663600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564546665ae0 0x564546663600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 -- 172.21.15.46:0/50663761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56454639d7e0 con 0x56454665cf00 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd665200640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e970 tx=0x7fd64c09b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfc022e37f67f643 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd647400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c09f8c0 con 0x56454665cf00 2024-03-08T23:13:42.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd64c0cf170 con 0x56454665cf00 2024-03-08T23:13:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd65800cb20 con 0x56454665cf00 2024-03-08T23:13:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd64c0a3f10 con 0x56454665cf00 2024-03-08T23:13:42.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd66b3a0740 10 client.26858 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.799+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5645463fdf70 con 0x56454665cf00 2024-03-08T23:13:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd65800cb20 con 0x56454665cf00 2024-03-08T23:13:42.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd64c09abc0 con 0x56454665cf00 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 10 client.26858 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 10 client.26858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 20 client.26858 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564546675ed0 0x5645466782c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 4 client.26858 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645465cdf90 con 0x564546675ed0 2024-03-08T23:13:42.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.800+0000 7fd665c00640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564546675ed0 0x5645466782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.801+0000 7fd665c00640 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564546675ed0 0x5645466782c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=716ae2dafafd60e3 server_cookie=577da89e179cf803 in_seq=0 out_seq=0 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.801+0000 7fd647400640 10 client.26858 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd647400640 1 -- 172.21.15.46:0/50663761 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5645465cdf90 con 0x564546675ed0 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd647400640 10 client.26858 handle_command_reply: tid=0 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 client.26858 shutdown 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 msgr2=0x5645466630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e970 tx=0x7fd64c09b3a0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 shutdown_connections 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564546675ed0 0x5645466782c0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564546665ae0 0x564546663600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56454665cf00 0x5645466630c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/50663761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x564546662b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 >> 172.21.15.46:0/50663761 conn(0x564546656d10 msgr2=0x564546658a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd647400640 0 client.26858 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 shutdown_connections 2024-03-08T23:13:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 1 -- 172.21.15.46:0/50663761 wait complete. 2024-03-08T23:13:42.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.802+0000 7fd66b3a0740 20 client.26858 trim_cache size 0 max 16384 2024-03-08T23:13:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 Processor -- start 2024-03-08T23:13:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 -- start start 2024-03-08T23:13:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.824 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x564546661b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601b0a60 con 0x56454665cf00 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601ae600 con 0x564546665ae0 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.819+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601ac6a0 con 0x56454665c5e0 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:50998/0 (socket says 172.21.15.46:50998) 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 -- 172.21.15.46:0/1692811920 learned_addr learned my addr 172.21.15.46:0/1692811920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd666600640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x564546661b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665c00640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 -- 172.21.15.46:0/1692811920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 msgr2=0x5645466652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 -- 172.21.15.46:0/1692811920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 msgr2=0x564546661b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x564546661b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 -- 172.21.15.46:0/1692811920 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564546295170 con 0x56454665cf00 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665c00640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:13:42.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd666600640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x564546661b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.820+0000 7fd665200640 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd64c09b700 tx=0x7fd64c09b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceeb780ece83c613 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd652000640 1 -- 172.21.15.46:0/1692811920 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c09be70 con 0x56454665cf00 2024-03-08T23:13:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd652000640 1 -- 172.21.15.46:0/1692811920 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64c09f8c0 con 0x56454665cf00 2024-03-08T23:13:42.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd652000640 1 -- 172.21.15.46:0/1692811920 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c0ccae0 con 0x56454665cf00 2024-03-08T23:13:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 -- 172.21.15.46:0/1692811920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 msgr2=0x56454666d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd64c09b700 tx=0x7fd64c09b8f0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 -- 172.21.15.46:0/1692811920 shutdown_connections 2024-03-08T23:13:42.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x564546661b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x56454666d0d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/1692811920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 -- 172.21.15.46:0/1692811920 >> 172.21.15.46:0/1692811920 conn(0x564546656d10 msgr2=0x564546661450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.821+0000 7fd66b3a0740 1 -- 172.21.15.46:0/1692811920 shutdown_connections 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 -- 172.21.15.46:0/1692811920 wait complete. 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 -- start start 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 Processor -- start 2024-03-08T23:13:42.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x5645466c4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.822+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601acca0 con 0x56454665cf00 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601adfa0 con 0x564546665ae0 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd66b3a0740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6601ad2e0 con 0x56454665c5e0 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd66b3a0740 10 client.0 _ll_register_callbacks cb 0x7fd658009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:51008/0 (socket says 172.21.15.46:51008) 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 -- 172.21.15.46:0/3636894295 learned_addr learned my addr 172.21.15.46:0/3636894295 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665c00640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd666600640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x5645466c4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 -- 172.21.15.46:0/3636894295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 msgr2=0x5645466ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 -- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 msgr2=0x5645466c4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x5645466c4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 -- 172.21.15.46:0/3636894295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56454658c050 con 0x56454665cf00 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.823+0000 7fd665200640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd64c099a20 tx=0x7fd64c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fd6e95b2db22dc server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd64c09be70 con 0x56454665cf00 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fd64c09abc0 con 0x56454665cf00 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd64c09f8c0 con 0x56454665cf00 2024-03-08T23:13:42.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd64c09adc0 con 0x56454665cf00 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd66b3a0740 10 client.16710 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56454658c050 con 0x56454665cf00 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd64c09f8c0 con 0x56454665cf00 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.824+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd64c09a630 con 0x56454665cf00 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 10 client.16710 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:42.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 20 client.16710 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5645466d2130 0x5645466d4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 4 client.16710 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5645465c82e0 con 0x5645466d2130 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd665c00640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5645466d2130 0x5645466d4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd665c00640 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5645466d2130 0x5645466d4520 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b02923c179331c52 server_cookie=634b39e1b25ecdc7 in_seq=0 out_seq=0 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.825+0000 7fd647400640 10 client.16710 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:42.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd647400640 1 -- 172.21.15.46:0/3636894295 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5645465c82e0 con 0x5645466d2130 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd647400640 10 client.16710 handle_command_reply: tid=0 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 client.16710 shutdown 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 msgr2=0x5645466cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd64c099a20 tx=0x7fd64c000bc0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 shutdown_connections 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5645466d2130 0x5645466d4520 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564546665ae0 0x5645466c4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56454665cf00 0x5645466cae30 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 --2- 172.21.15.46:0/3636894295 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56454665c5e0 0x5645466ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd647400640 0 client.16710 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 >> 172.21.15.46:0/3636894295 conn(0x564546656d10 msgr2=0x564546661ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 shutdown_connections 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 1 -- 172.21.15.46:0/3636894295 wait complete. 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.827+0000 7fd66b3a0740 20 client.16710 trim_cache size 0 max 16384 2024-03-08T23:13:42.839 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:13:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "808a34e3-3bd9-433b-a5ab-2e697b968611", 2024-03-08T23:13:42.840 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:13:42.840 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 msgr2=0x7fd6480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 0x7fd6480a4eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd654009830 tx=0x7fd65402e040 comp rx=0 tx=0).stop 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 msgr2=0x7fd66019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd64c00e9a0 tx=0x7fd64c00ee60 comp rx=0 tx=0).stop 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 shutdown_connections 2024-03-08T23:13:42.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6480a29f0 0x7fd6480a4eb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6601979e0 0x7fd660197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd6601065c0 0x7fd66019dd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 --2- 172.21.15.46:0/3410359384 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd6601014f0 0x7fd66019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 >> 172.21.15.46:0/3410359384 conn(0x7fd6600786f0 msgr2=0x7fd6600ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 shutdown_connections 2024-03-08T23:13:42.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:42.828+0000 7fd668600640 1 -- 172.21.15.46:0/3410359384 wait complete. 2024-03-08T23:13:42.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:13:42.860 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:13:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 -- 172.21.15.46:0/195757232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c41073d0 msgr2=0x7fc2c41077b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 --2- 172.21.15.46:0/195757232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c41073d0 0x7fc2c41077b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fc2b4009900 tx=0x7fc2b402bc30 comp rx=0 tx=0).stop 2024-03-08T23:13:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 -- 172.21.15.46:0/195757232 shutdown_connections 2024-03-08T23:13:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 --2- 172.21.15.46:0/195757232 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 --2- 172.21.15.46:0/195757232 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c40ff840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 --2- 172.21.15.46:0/195757232 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c41073d0 0x7fc2c41077b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 -- 172.21.15.46:0/195757232 >> 172.21.15.46:0/195757232 conn(0x7fc2c4076340 msgr2=0x7fc2c40fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 -- 172.21.15.46:0/195757232 shutdown_connections 2024-03-08T23:13:43.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.145+0000 7fc2cb200640 1 -- 172.21.15.46:0/195757232 wait complete. 2024-03-08T23:13:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 Processor -- start 2024-03-08T23:13:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 -- start start 2024-03-08T23:13:43.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c410ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 0x7fc2c410b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41025e0 con 0x7fc2c41073d0 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4102460 con 0x7fc2c40ff3c0 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.147+0000 7fc2cb200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4102760 con 0x7fc2c40ffd80 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.148+0000 7fc2c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c410ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.148+0000 7fc2c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c410ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39456/0 (socket says 172.21.15.46:39456) 2024-03-08T23:13:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.148+0000 7fc2c3e00640 1 -- 172.21.15.46:0/3439758890 learned_addr learned my addr 172.21.15.46:0/3439758890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.148+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 0x7fc2c410b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.148+0000 7fc2c3400640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 -- 172.21.15.46:0/3439758890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 msgr2=0x7fc2c410ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c410ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 -- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 msgr2=0x7fc2c410b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 0x7fc2c410b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 -- 172.21.15.46:0/3439758890 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c410bac0 con 0x7fc2c40ffd80 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 0x7fc2c410b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:43.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.149+0000 7fc2c3400640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410ae50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000e9f0 tx=0x7fc2b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.150+0000 7fc2c0c00640 1 -- 172.21.15.46:0/3439758890 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b000ccc0 con 0x7fc2c40ffd80 2024-03-08T23:13:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.150+0000 7fc2c0c00640 1 -- 172.21.15.46:0/3439758890 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2b0010910 con 0x7fc2c40ffd80 2024-03-08T23:13:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.150+0000 7fc2c0c00640 1 -- 172.21.15.46:0/3439758890 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0018960 con 0x7fc2c40ffd80 2024-03-08T23:13:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.150+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c41aa780 con 0x7fc2c40ffd80 2024-03-08T23:13:43.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.150+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c41aab30 con 0x7fc2c40ffd80 2024-03-08T23:13:43.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.153+0000 7fc2c0c00640 1 -- 172.21.15.46:0/3439758890 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc2b0010430 con 0x7fc2c40ffd80 2024-03-08T23:13:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.155+0000 7fc2ce015740 1 Processor -- start 2024-03-08T23:13:43.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.155+0000 7fc2ce015740 1 -- start start 2024-03-08T23:13:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.155+0000 7fc2c0c00640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 0x7fc2a80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c0c00640 1 -- 172.21.15.46:0/3439758890 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc2b00cf4f0 con 0x7fc2c40ffd80 2024-03-08T23:13:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f292a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2933540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f28ec410 con 0x55d7f292a580 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f288ed50 con 0x55d7f2933a80 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2ce015740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f2837b80 con 0x55d7f292aea0 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39460/0 (socket says 172.21.15.46:39460) 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c9200640 1 -- 172.21.15.46:0/410105113 learned_addr learned my addr 172.21.15.46:0/410105113 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f292a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 0x7fc2a80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c3400640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2933540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.156+0000 7fc2c9200640 1 -- 172.21.15.46:0/410105113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 msgr2=0x55d7f2933540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c9200640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2933540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c9200640 1 -- 172.21.15.46:0/410105113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 msgr2=0x55d7f292a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c9200640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f292a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c9200640 1 -- 172.21.15.46:0/410105113 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7f266d7d0 con 0x55d7f2933a80 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c3400640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2933540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2c9200640 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc007310 tx=0x7fc2bc014840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc8b277df5725722 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2b9600640 1 -- 172.21.15.46:0/410105113 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc023070 con 0x55d7f2933a80 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2b9600640 1 -- 172.21.15.46:0/410105113 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc01d8f0 con 0x55d7f2933a80 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.157+0000 7fc2b9600640 1 -- 172.21.15.46:0/410105113 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc00e7e0 con 0x55d7f2933a80 2024-03-08T23:13:43.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.158+0000 7fc2ce015740 1 -- 172.21.15.46:0/410105113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 msgr2=0x55d7f2935e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.158+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 0x7fc2a80a4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc020000 tx=0x7fc2bc01b000 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc007310 tx=0x7fc2bc014840 comp rx=0 tx=0).stop 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 -- 172.21.15.46:0/410105113 shutdown_connections 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2935e70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2933540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 --2- 172.21.15.46:0/410105113 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f292a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 -- 172.21.15.46:0/410105113 >> 172.21.15.46:0/410105113 conn(0x55d7f2924cb0 msgr2=0x55d7f2925090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 -- 172.21.15.46:0/410105113 shutdown_connections 2024-03-08T23:13:43.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.159+0000 7fc2ce015740 1 -- 172.21.15.46:0/410105113 wait complete. 2024-03-08T23:13:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.160+0000 7fc2ce015740 1 -- start start 2024-03-08T23:13:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.160+0000 7fc2ce015740 1 Processor -- start 2024-03-08T23:13:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f2930a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2931500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f284aee0 con 0x55d7f292a580 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f288adf0 con 0x55d7f2933a80 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d7f2936870 con 0x55d7f292aea0 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2ce015740 10 client.0 _ll_register_callbacks cb 0x55d7f28518d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:49318/0 (socket says 172.21.15.46:49318) 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.161+0000 7fc2c3400640 1 -- 172.21.15.46:0/517921247 learned_addr learned my addr 172.21.15.46:0/517921247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c9200640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2931500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f2930a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 -- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 msgr2=0x55d7f2931500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2931500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 -- 172.21.15.46:0/517921247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 msgr2=0x55d7f2930a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f2930a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 -- 172.21.15.46:0/517921247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7f25a33f0 con 0x55d7f292aea0 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c9200640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2931500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.162+0000 7fc2c3400640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000c6a0 tx=0x7fc2b009b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c5e0bf0b34de08b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.163+0000 7fc2ae000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.163+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0086070 con 0x55d7f292aea0 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.163+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2bc01d8f0 con 0x55d7f292aea0 2024-03-08T23:13:43.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.163+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc2b00a3aa0 con 0x55d7f292aea0 2024-03-08T23:13:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2b0086210 con 0x55d7f292aea0 2024-03-08T23:13:43.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 10 client.26854 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d7f25630f0 con 0x55d7f292aea0 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc01d8f0 con 0x55d7f292aea0 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc2b009ac90 con 0x55d7f292aea0 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 10 client.26854 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 10 client.26854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 20 client.26854 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:43.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f2943b50 0x55d7f2945f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 4 client.26854 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.164+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d7f289c290 con 0x55d7f2943b50 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.165+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f2943b50 0x55d7f2945f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.165+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f2943b50 0x55d7f2945f40 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d32cfc6f1055150 server_cookie=4c1fe9ad55cea956 in_seq=0 out_seq=0 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.165+0000 7fc2ae000640 10 client.26854 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.166+0000 7fc2ae000640 1 -- 172.21.15.46:0/517921247 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d7f289c290 con 0x55d7f2943b50 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.166+0000 7fc2ae000640 10 client.26854 handle_command_reply: tid=0 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.166+0000 7fc2ce015740 1 client.26854 shutdown 2024-03-08T23:13:43.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 msgr2=0x55d7f2930fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000c6a0 tx=0x7fc2b009b980 comp rx=0 tx=0).stop 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 shutdown_connections 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f2943b50 0x55d7f2945f40 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f2931500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f2930fc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 --2- 172.21.15.46:0/517921247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f2930a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 >> 172.21.15.46:0/517921247 conn(0x55d7f2924cb0 msgr2=0x55d7f2923f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ae000640 0 client.26854 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 shutdown_connections 2024-03-08T23:13:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.167+0000 7fc2ce015740 1 -- 172.21.15.46:0/517921247 wait complete. 2024-03-08T23:13:43.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.168+0000 7fc2ce015740 20 client.26854 trim_cache size 0 max 16384 2024-03-08T23:13:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.193+0000 7fc2ce015740 1 Processor -- start 2024-03-08T23:13:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.193+0000 7fc2ce015740 1 -- start start 2024-03-08T23:13:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f29331b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f293aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.197 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b7010 con 0x55d7f292a580 2024-03-08T23:13:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b4bb0 con 0x55d7f2933a80 2024-03-08T23:13:43.198 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2ce015740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b2c50 con 0x55d7f292aea0 2024-03-08T23:13:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f293aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f29331b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.199 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39472/0 (socket says 172.21.15.46:39472) 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 -- 172.21.15.46:0/3067655702 learned_addr learned my addr 172.21.15.46:0/3067655702 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 -- 172.21.15.46:0/3067655702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 msgr2=0x55d7f293aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f293aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 -- 172.21.15.46:0/3067655702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 msgr2=0x55d7f29331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f29331b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c9200640 1 -- 172.21.15.46:0/3067655702 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d7f292acd0 con 0x55d7f2933a80 2024-03-08T23:13:43.200 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.194+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f29331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:13:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.195+0000 7fc2c9200640 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc011d70 tx=0x7fc2bc028000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee093e9109b40f37 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.195+0000 7fc2b8c00640 1 -- 172.21.15.46:0/3067655702 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc023050 con 0x55d7f2933a80 2024-03-08T23:13:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.195+0000 7fc2b8c00640 1 -- 172.21.15.46:0/3067655702 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc02d070 con 0x55d7f2933a80 2024-03-08T23:13:43.201 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.195+0000 7fc2b8c00640 1 -- 172.21.15.46:0/3067655702 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc003c60 con 0x55d7f2933a80 2024-03-08T23:13:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 -- 172.21.15.46:0/3067655702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 msgr2=0x55d7f293aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc011d70 tx=0x7fc2bc028000 comp rx=0 tx=0).stop 2024-03-08T23:13:43.204 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 -- 172.21.15.46:0/3067655702 shutdown_connections 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f293aff0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292aea0 0x55d7f293aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 --2- 172.21.15.46:0/3067655702 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292a580 0x55d7f29331b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.196+0000 7fc2ce015740 1 -- 172.21.15.46:0/3067655702 >> 172.21.15.46:0/3067655702 conn(0x55d7f2924cb0 msgr2=0x55d7f292b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.197+0000 7fc2ce015740 1 -- 172.21.15.46:0/3067655702 shutdown_connections 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.197+0000 7fc2ce015740 1 -- 172.21.15.46:0/3067655702 wait complete. 2024-03-08T23:13:43.205 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.197+0000 7fc2ce015740 1 -- start start 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.197+0000 7fc2ce015740 1 Processor -- start 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292a580 0x55d7f2993920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 0x55d7f2993e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b3250 con 0x55d7f292aea0 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b4550 con 0x55d7f2933a80 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41b3890 con 0x55d7f292a580 2024-03-08T23:13:43.206 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2ce015740 10 client.0 _ll_register_callbacks cb 0x7fc2bc011760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 0x55d7f2993e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2c3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 0x55d7f2993e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:34476/0 (socket says 172.21.15.46:34476) 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2c9200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:39480/0 (socket says 172.21.15.46:39480) 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.198+0000 7fc2c3400640 1 -- 172.21.15.46:0/629259505 learned_addr learned my addr 172.21.15.46:0/629259505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 -- 172.21.15.46:0/629259505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292a580 msgr2=0x55d7f2993920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292a580 0x55d7f2993920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.207 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292a580 0x55d7f2993920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 -- 172.21.15.46:0/629259505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 msgr2=0x55d7f2993e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 0x55d7f2993e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 -- 172.21.15.46:0/629259505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d7f298e6a0 con 0x55d7f2933a80 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2c9200640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc014d40 tx=0x7fc2bc003c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82ad38826a3cfa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2ae000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.199+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2bc004450 con 0x55d7f2933a80 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9387+0+0 (secure 0 0 0) 0x7fc2bc0045f0 con 0x55d7f2933a80 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc2bc02d070 con 0x55d7f2933a80 2024-03-08T23:13:43.208 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2bc023030 con 0x55d7f2933a80 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ce015740 10 client.26882 fetch_fsmap learned FSMap version 31 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d7f299b160 con 0x55d7f2933a80 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.200+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc2bc02d070 con 0x55d7f2933a80 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc2bc03b950 con 0x55d7f2933a80 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 10 client.26882 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 10 client.26882 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 20 client.26882 populate_metadata read hostname 'smithi046' 2024-03-08T23:13:43.209 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f299b580 0x55d7f299d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:13:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 4 client.26882 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:13:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d7f28965e0 con 0x55d7f299b580 2024-03-08T23:13:43.210 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.201+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f299b580 0x55d7f299d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:13:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2c3e00640 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f299b580 0x55d7f299d970 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4375f64add56ab8 server_cookie=b96725195fcf913d in_seq=0 out_seq=0 2024-03-08T23:13:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ae000640 10 client.26882 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ae000640 1 -- 172.21.15.46:0/629259505 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55d7f28965e0 con 0x55d7f299b580 2024-03-08T23:13:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ae000640 10 client.26882 handle_command_reply: tid=0 2024-03-08T23:13:43.211 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 client.26882 shutdown 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 msgr2=0x55d7f298dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc014d40 tx=0x7fc2bc003c20 comp rx=0 tx=0).stop 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 shutdown_connections 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d7f299b580 0x55d7f299d970 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d7f2933a80 0x55d7f298dfb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d7f292aea0 0x55d7f2993e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 --2- 172.21.15.46:0/629259505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d7f292a580 0x55d7f2993920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 >> 172.21.15.46:0/629259505 conn(0x55d7f2924cb0 msgr2=0x55d7f292ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.212 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ae000640 0 client.26882 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 shutdown_connections 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.202+0000 7fc2ce015740 1 -- 172.21.15.46:0/629259505 wait complete. 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2ce015740 20 client.26882 trim_cache size 0 max 16384 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:13:43.213 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: "808a34e3-3bd9-433b-a5ab-2e697b968611": { 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "808a34e3-3bd9-433b-a5ab-2e697b968611", 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:13:43.214 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:13:43.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 msgr2=0x7fc2a80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 0x7fc2a80a4eb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc020000 tx=0x7fc2bc01b000 comp rx=0 tx=0).stop 2024-03-08T23:13:43.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 msgr2=0x7fc2c410ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:13:43.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410ae50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000e9f0 tx=0x7fc2b000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.215 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 shutdown_connections 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc2a80a29f0 0x7fc2a80a4eb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc2c41073d0 0x7fc2c410b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2c40ffd80 0x7fc2c410ae50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 --2- 172.21.15.46:0/3439758890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc2c40ff3c0 0x7fc2c410ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 >> 172.21.15.46:0/3439758890 conn(0x7fc2c4076340 msgr2=0x7fc2c40fd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 shutdown_connections 2024-03-08T23:13:43.216 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:13:43.203+0000 7fc2cb200640 1 -- 172.21.15.46:0/3439758890 wait complete. 2024-03-08T23:13:43.222 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:42 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:13:43.222 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:42 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:43.222 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:42 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:43.222 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:42 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:43.223 INFO:tasks.cephfs.filesystem:scrub status for tag:808a34e3-3bd9-433b-a5ab-2e697b968611 - {'path': '/', 'tag': '808a34e3-3bd9-433b-a5ab-2e697b968611', 'options': 'recursive,force'} 2024-03-08T23:13:43.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:42 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:13:43.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:42 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:43.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:42 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:43.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:42 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:43.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:42 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:13:43.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:42 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:43.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:42 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:43.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:42 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:43 smithi046 ceph-mon[31859]: pgmap v181: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:43 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:13:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:43 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:43 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:43 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:43 smithi192 ceph-mon[36700]: pgmap v181: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:43 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:13:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:43 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:43 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:43 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:43 smithi062 ceph-mon[35761]: pgmap v181: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:13:44.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:43 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:13:44.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:43 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:43 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:44.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:43 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:13:46.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:45 smithi192 ceph-mon[36700]: pgmap v182: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:46.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:45 smithi062 ceph-mon[35761]: pgmap v182: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:46.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:45 smithi046 ceph-mon[31859]: pgmap v182: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:48.277 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:47 smithi046 ceph-mon[31859]: pgmap v183: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:48.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:47 smithi192 ceph-mon[36700]: pgmap v183: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:48.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:47 smithi062 ceph-mon[35761]: pgmap v183: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:13:50.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:49 smithi192 ceph-mon[36700]: pgmap v184: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:50.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:49 smithi062 ceph-mon[35761]: pgmap v184: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:50.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:49 smithi046 ceph-mon[31859]: pgmap v184: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:52.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:51 smithi192 ceph-mon[36700]: pgmap v185: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:52.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:51 smithi062 ceph-mon[35761]: pgmap v185: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:52.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:51 smithi046 ceph-mon[31859]: pgmap v185: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:54.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:53 smithi192 ceph-mon[36700]: pgmap v186: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:54.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:53 smithi062 ceph-mon[35761]: pgmap v186: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:54.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:53 smithi046 ceph-mon[31859]: pgmap v186: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:13:56.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:56 smithi192 ceph-mon[36700]: pgmap v187: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:56.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:56 smithi062 ceph-mon[35761]: pgmap v187: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:56.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:56 smithi046 ceph-mon[31859]: pgmap v187: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:57.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:57 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:57.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:57 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:57.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:57 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:57.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:57 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:57.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:57 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:13:57.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:57 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:13:58.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:58 smithi192 ceph-mon[36700]: pgmap v188: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:13:58 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:13:58.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:58 smithi062 ceph-mon[35761]: pgmap v188: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:58.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:13:58 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:13:58.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:58 smithi046 ceph-mon[31859]: pgmap v188: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:13:58.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:13:58 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:14:00.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:00 smithi192 ceph-mon[36700]: pgmap v189: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:00.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:00 smithi062 ceph-mon[35761]: pgmap v189: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:00.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:00 smithi046 ceph-mon[31859]: pgmap v189: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:02.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:02 smithi192 ceph-mon[36700]: pgmap v190: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:02.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:02 smithi062 ceph-mon[35761]: pgmap v190: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:02.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:02 smithi046 ceph-mon[31859]: pgmap v190: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:04.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:04 smithi192 ceph-mon[36700]: pgmap v191: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-08T23:14:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:04 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:14:04.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:04 smithi062 ceph-mon[35761]: pgmap v191: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-08T23:14:04.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:04 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:14:04.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:04 smithi046 ceph-mon[31859]: pgmap v191: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 24 op/s 2024-03-08T23:14:04.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:04 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:14:05.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:05 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:14:05.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:05 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:14:05.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:05 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:14:05.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:05 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:14:05.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:05 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:14:05.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:05 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:14:05.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:05 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:14:05.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:05 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:14:05.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:05 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:14:06.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:06 smithi192 ceph-mon[36700]: pgmap v192: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:06.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:06 smithi062 ceph-mon[35761]: pgmap v192: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:06.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:06 smithi046 ceph-mon[31859]: pgmap v192: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:08.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:08 smithi192 ceph-mon[36700]: pgmap v193: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:08.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:08 smithi192 ceph-mon[36700]: osdmap e95: 12 total, 12 up, 12 in 2024-03-08T23:14:08.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:08 smithi062 ceph-mon[35761]: pgmap v193: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:08.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:08 smithi062 ceph-mon[35761]: osdmap e95: 12 total, 12 up, 12 in 2024-03-08T23:14:08.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:08 smithi046 ceph-mon[31859]: pgmap v193: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-03-08T23:14:08.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:08 smithi046 ceph-mon[31859]: osdmap e95: 12 total, 12 up, 12 in 2024-03-08T23:14:10.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:10 smithi192 ceph-mon[36700]: pgmap v195: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:10.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:10 smithi062 ceph-mon[35761]: pgmap v195: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:10.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:10 smithi046 ceph-mon[31859]: pgmap v195: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:12.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:12 smithi192 ceph-mon[36700]: pgmap v196: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:12.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:12 smithi062 ceph-mon[35761]: pgmap v196: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:12.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:12 smithi046 ceph-mon[31859]: pgmap v196: 129 pgs: 129 active+clean; 78 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-03-08T23:14:13.225 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:14:13.225 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:14:13.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 -- 172.21.15.46:0/4220561281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107890 msgr2=0x7f7780107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 --2- 172.21.15.46:0/4220561281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107890 0x7f7780107d10 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f776c009930 tx=0x7f776c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 -- 172.21.15.46:0/4220561281 shutdown_connections 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 --2- 172.21.15.46:0/4220561281 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780108250 0x7f7780114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 --2- 172.21.15.46:0/4220561281 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780107890 0x7f7780107d10 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 --2- 172.21.15.46:0/4220561281 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f778010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 -- 172.21.15.46:0/4220561281 >> 172.21.15.46:0/4220561281 conn(0x7f7780075830 msgr2=0x7f7780077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 -- 172.21.15.46:0/4220561281 shutdown_connections 2024-03-08T23:14:13.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.459+0000 7f7785e00640 1 -- 172.21.15.46:0/4220561281 wait complete. 2024-03-08T23:14:13.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.460+0000 7f7785e00640 1 Processor -- start 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.460+0000 7f7785e00640 1 -- start start 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.460+0000 7f7785e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780107890 0x7f778019e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.460+0000 7f7785e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780108250 0x7f778019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.460+0000 7f7785e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f7785e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780117d00 con 0x7f7780108250 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f7785e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780117b80 con 0x7f7780107890 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f7785e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780117e80 con 0x7f778010d8a0 2024-03-08T23:14:13.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57838/0 (socket says 172.21.15.46:57838) 2024-03-08T23:14:13.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 -- 172.21.15.46:0/563131693 learned_addr learned my addr 172.21.15.46:0/563131693 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777e000640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780108250 0x7f778019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777ea00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780107890 0x7f778019e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 -- 172.21.15.46:0/563131693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780107890 msgr2=0x7f778019e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780107890 0x7f778019e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 -- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780108250 msgr2=0x7f778019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780108250 0x7f778019e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.461+0000 7f777f400640 1 -- 172.21.15.46:0/563131693 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77801a31f0 con 0x7f778010d8a0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f777f400640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f777800eb10 tx=0x7f777800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f7784a00640 1 -- 172.21.15.46:0/563131693 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778004b20 con 0x7f778010d8a0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77801a34e0 con 0x7f778010d8a0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f7784a00640 1 -- 172.21.15.46:0/563131693 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7778004cc0 con 0x7f778010d8a0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f7784a00640 1 -- 172.21.15.46:0/563131693 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7778010760 con 0x7f778010d8a0 2024-03-08T23:14:13.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.462+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77801aad70 con 0x7f778010d8a0 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.465+0000 7f7784a00640 1 -- 172.21.15.46:0/563131693 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7778005020 con 0x7f778010d8a0 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.465+0000 7f7788be7740 1 Processor -- start 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.465+0000 7f7788be7740 1 -- start start 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48ae80960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae8be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ae42410 con 0x55f48ae80ea0 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ade4d50 con 0x55f48ae89a80 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f7788be7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ad8db80 con 0x55f48ae80580 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777e000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48ae80960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36318/0 (socket says 172.21.15.46:36318) 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 -- 172.21.15.46:0/1595387244 learned_addr learned my addr 172.21.15.46:0/1595387244 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777e000640 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae8be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 -- 172.21.15.46:0/1595387244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 msgr2=0x55f48ae80960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48ae80960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 -- 172.21.15.46:0/1595387244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 msgr2=0x55f48ae8be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae8be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777ea00640 1 -- 172.21.15.46:0/1595387244 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f48abc37d0 con 0x55f48ae80ea0 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.466+0000 7f777e000640 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae8be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:13.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f777ea00640 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7770009ca0 tx=0x7f7770011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a548c358041c53 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f7784a00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 0x7f77640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f776be00640 1 -- 172.21.15.46:0/1595387244 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777001e070 con 0x55f48ae80ea0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f776be00640 1 -- 172.21.15.46:0/1595387244 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7770019b10 con 0x55f48ae80ea0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f7784a00640 1 -- 172.21.15.46:0/563131693 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f77780cf850 con 0x7f778010d8a0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f776be00640 1 -- 172.21.15.46:0/1595387244 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777000ea70 con 0x55f48ae80ea0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.467+0000 7f777e000640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 0x7f77640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 -- 172.21.15.46:0/1595387244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 msgr2=0x55f48ae89540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f7770009ca0 tx=0x7f7770011810 comp rx=0 tx=0).stop 2024-03-08T23:14:13.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 -- 172.21.15.46:0/1595387244 shutdown_connections 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae8be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48ae89540 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 --2- 172.21.15.46:0/1595387244 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48ae80960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 -- 172.21.15.46:0/1595387244 >> 172.21.15.46:0/1595387244 conn(0x55f48ae7acb0 msgr2=0x55f48ae7b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f777e000640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 0x7f77640a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f776c009900 tx=0x7f776c02df90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 -- 172.21.15.46:0/1595387244 shutdown_connections 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.468+0000 7f7788be7740 1 -- 172.21.15.46:0/1595387244 wait complete. 2024-03-08T23:14:13.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 -- start start 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 Processor -- start 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae87030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae87570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ada0ee0 con 0x55f48ae80580 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ade0df0 con 0x55f48ae89a80 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.469+0000 7f7788be7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f48ae8c870 con 0x55f48ae80ea0 2024-03-08T23:14:13.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f7788be7740 10 client.0 _ll_register_callbacks cb 0x55f48ada78d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36326/0 (socket says 172.21.15.46:36326) 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 -- 172.21.15.46:0/2040091389 learned_addr learned my addr 172.21.15.46:0/2040091389 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777e000640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae87030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777f400640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae87570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 -- 172.21.15.46:0/2040091389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 msgr2=0x55f48ae87030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae87030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 -- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 msgr2=0x55f48ae87570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae87570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 -- 172.21.15.46:0/2040091389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f48aaf93f0 con 0x55f48ae80580 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f777ea00640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7770009c70 tx=0x7f7770019920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a04e55776c634b05 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.470+0000 7f7768c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770027070 con 0x55f48ae80580 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f7770004020 con 0x55f48ae80580 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7770019b10 con 0x55f48ae80580 2024-03-08T23:14:13.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7770032050 con 0x55f48ae80580 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7788be7740 10 client.16737 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f48aab90f0 con 0x55f48ae80580 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.471+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7770019b10 con 0x55f48ae80580 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f777000e800 con 0x55f48ae80580 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 10 client.16737 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 20 client.16737 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:13.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48ae99bc0 0x55f48ae9bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 4 client.16737 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:13.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f48adf2290 con 0x55f48ae99bc0 2024-03-08T23:14:13.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.472+0000 7f777e000640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48ae99bc0 0x55f48ae9bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.473+0000 7f777e000640 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48ae99bc0 0x55f48ae9bfb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b306c8e438d3e0 server_cookie=998c2fa79eeb279c in_seq=0 out_seq=0 2024-03-08T23:14:13.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.473+0000 7f7768c00640 10 client.16737 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.473+0000 7f7768c00640 1 -- 172.21.15.46:0/2040091389 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f48adf2290 con 0x55f48ae99bc0 2024-03-08T23:14:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.473+0000 7f7768c00640 10 client.16737 handle_command_reply: tid=0 2024-03-08T23:14:13.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.473+0000 7f7788be7740 1 client.16737 shutdown 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 msgr2=0x55f48ae86af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7770009c70 tx=0x7f7770019920 comp rx=0 tx=0).stop 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 shutdown_connections 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48ae99bc0 0x55f48ae9bfb0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae87570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae87030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 --2- 172.21.15.46:0/2040091389 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae86af0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7768c00640 0 client.16737 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 >> 172.21.15.46:0/2040091389 conn(0x55f48ae7acb0 msgr2=0x55f48ae7ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 shutdown_connections 2024-03-08T23:14:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 1 -- 172.21.15.46:0/2040091389 wait complete. 2024-03-08T23:14:13.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.474+0000 7f7788be7740 20 client.16737 trim_cache size 0 max 16384 2024-03-08T23:14:13.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 Processor -- start 2024-03-08T23:14:13.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 -- start start 2024-03-08T23:14:13.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae79680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae79bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b7220 con 0x55f48ae80580 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b4dc0 con 0x55f48ae89a80 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f7788be7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b2e80 con 0x55f48ae80ea0 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.491+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36334/0 (socket says 172.21.15.46:36334) 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 -- 172.21.15.46:0/2255165802 learned_addr learned my addr 172.21.15.46:0/2255165802 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.496 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777e000640 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae79680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777f400640 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae79bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 -- 172.21.15.46:0/2255165802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 msgr2=0x55f48ae79680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae79680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 -- 172.21.15.46:0/2255165802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 msgr2=0x55f48ae79bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae79bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 -- 172.21.15.46:0/2255165802 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f48ae85040 con 0x55f48ae80580 2024-03-08T23:14:13.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f777ea00640 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f777000e5a0 tx=0x7f7770009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f93e766d02fda3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.492+0000 7f776b400640 1 -- 172.21.15.46:0/2255165802 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770027070 con 0x55f48ae80580 2024-03-08T23:14:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f776b400640 1 -- 172.21.15.46:0/2255165802 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f777000eda0 con 0x55f48ae80580 2024-03-08T23:14:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f776b400640 1 -- 172.21.15.46:0/2255165802 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777001e040 con 0x55f48ae80580 2024-03-08T23:14:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 -- 172.21.15.46:0/2255165802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 msgr2=0x55f48ae89220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f777000e5a0 tx=0x7f7770009fd0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 -- 172.21.15.46:0/2255165802 shutdown_connections 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48ae79bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80ea0 0x55f48ae79680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 --2- 172.21.15.46:0/2255165802 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80580 0x55f48ae89220 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 -- 172.21.15.46:0/2255165802 >> 172.21.15.46:0/2255165802 conn(0x55f48ae7acb0 msgr2=0x55f48ae812a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 -- 172.21.15.46:0/2255165802 shutdown_connections 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.493+0000 7f7788be7740 1 -- 172.21.15.46:0/2255165802 wait complete. 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 -- start start 2024-03-08T23:14:13.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 Processor -- start 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48aee8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48aee8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b3480 con 0x55f48ae80ea0 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b4760 con 0x55f48ae89a80 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.494+0000 7f7788be7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801b3aa0 con 0x55f48ae80580 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f7788be7740 10 client.0 _ll_register_callbacks cb 0x7f7770009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57880/0 (socket says 172.21.15.46:57880) 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 -- 172.21.15.46:0/3962012846 learned_addr learned my addr 172.21.15.46:0/3962012846 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48aee8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777e000640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48aee8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 -- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 msgr2=0x55f48aee8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48aee8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 -- 172.21.15.46:0/3962012846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 msgr2=0x55f48aee8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48aee8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 -- 172.21.15.46:0/3962012846 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f48aee97d0 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f777ea00640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f777000ed20 tx=0x7f777000ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46e3315abfeee2f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.495+0000 7f7768c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777001e040 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f777000eda0 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f777000e7d0 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7770043050 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7788be7740 10 client.26875 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.496+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f48aef0b10 con 0x55f48ae80580 2024-03-08T23:14:13.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f777000eda0 con 0x55f48ae80580 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f777003e030 con 0x55f48ae80580 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 10 client.26875 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 10 client.26875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 20 client.26875 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48aef10c0 0x55f48aef34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 4 client.26875 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f48adec5e0 con 0x55f48aef10c0 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.497+0000 7f777e000640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48aef10c0 0x55f48aef34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f777e000640 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48aef10c0 0x55f48aef34e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34f1c58405f65dc3 server_cookie=6ee56ab3f3ecae07 in_seq=0 out_seq=0 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7768c00640 10 client.26875 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7768c00640 1 -- 172.21.15.46:0/3962012846 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f48adec5e0 con 0x55f48aef10c0 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7768c00640 10 client.26875 handle_command_reply: tid=0 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 client.26875 shutdown 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 msgr2=0x55f48aee8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f777000ed20 tx=0x7f777000ef50 comp rx=0 tx=0).stop 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 shutdown_connections 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f48aef10c0 0x55f48aef34e0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f48ae89a80 0x55f48aee8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f48ae80ea0 0x55f48aee8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 --2- 172.21.15.46:0/3962012846 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f48ae80580 0x55f48aee8230 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7768c00640 0 client.26875 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 >> 172.21.15.46:0/3962012846 conn(0x55f48ae7acb0 msgr2=0x55f48ae79680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 shutdown_connections 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.498+0000 7f7788be7740 1 -- 172.21.15.46:0/3962012846 wait complete. 2024-03-08T23:14:13.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7788be7740 20 client.26875 trim_cache size 0 max 16384 2024-03-08T23:14:13.510 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:14:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:14:13.510 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:13.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 msgr2=0x7f77640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 0x7f77640a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f776c009900 tx=0x7f776c02df90 comp rx=0 tx=0).stop 2024-03-08T23:14:13.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 msgr2=0x7f77801a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f777800eb10 tx=0x7f777800c2d0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 shutdown_connections 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f77640a2970 0x7f77640a4e30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f778010d8a0 0x7f77801a2ac0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7780108250 0x7f778019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 --2- 172.21.15.46:0/563131693 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7780107890 0x7f778019e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 >> 172.21.15.46:0/563131693 conn(0x7f7780075830 msgr2=0x7f7780075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.499+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 shutdown_connections 2024-03-08T23:14:13.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.500+0000 7f7785e00640 1 -- 172.21.15.46:0/563131693 wait complete. 2024-03-08T23:14:13.520 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:14:13.520 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:14:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 -- 172.21.15.46:0/2642497364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a81078a0 msgr2=0x7fc7a8107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2642497364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a81078a0 0x7fc7a8107d20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc790009890 tx=0x7fc79002bb70 comp rx=0 tx=0).stop 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 -- 172.21.15.46:0/2642497364 shutdown_connections 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2642497364 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a8108260 0x7fc7a8114770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2642497364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a81078a0 0x7fc7a8107d20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2642497364 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a810d8b0 0x7fc7a810dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 -- 172.21.15.46:0/2642497364 >> 172.21.15.46:0/2642497364 conn(0x7fc7a8075840 msgr2=0x7fc7a8077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 -- 172.21.15.46:0/2642497364 shutdown_connections 2024-03-08T23:14:13.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.737+0000 7fc7ae800640 1 -- 172.21.15.46:0/2642497364 wait complete. 2024-03-08T23:14:13.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.738+0000 7fc7ae800640 1 Processor -- start 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 -- start start 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a8108260 0x7fc7a819e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 0x7fc7a81a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8117c80 con 0x7fc7a81078a0 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8117b00 con 0x7fc7a8108260 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7ae800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8117e00 con 0x7fc7a810d8b0 2024-03-08T23:14:13.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7a7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7a7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 0x7fc7a81a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7a7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36366/0 (socket says 172.21.15.46:36366) 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc7a7400640 1 -- 172.21.15.46:0/2208639738 learned_addr learned my addr 172.21.15.46:0/2208639738 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.739+0000 7fc79fe00640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a8108260 0x7fc7a819e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 -- 172.21.15.46:0/2208639738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a8108260 msgr2=0x7fc7a819e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a8108260 0x7fc7a819e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 -- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 msgr2=0x7fc7a819e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:13.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 -- 172.21.15.46:0/2208639738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a81a31b0 con 0x7fc7a810d8b0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7400640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a7e00640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 0x7fc7a81a2a80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009fd0 tx=0x7fc7a000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.740+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a000ce20 con 0x7fc7a810d8b0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.741+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a81a34a0 con 0x7fc7a810d8b0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.741+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7a0005020 con 0x7fc7a810d8b0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.741+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0010720 con 0x7fc7a810d8b0 2024-03-08T23:14:13.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.741+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a81aad30 con 0x7fc7a810d8b0 2024-03-08T23:14:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.742+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8068530 con 0x7fc7a810d8b0 2024-03-08T23:14:13.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.744+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc7a0005540 con 0x7fc7a810d8b0 2024-03-08T23:14:13.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.745+0000 7fc7a4c00640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 0x7fc78c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:13.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.745+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fc7a00cf9c0 con 0x7fc7a810d8b0 2024-03-08T23:14:13.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.746+0000 7fc7a7400640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 0x7fc78c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:13.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.746+0000 7fc7a7400640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 0x7fc78c0a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc7980096c0 tx=0x7fc798009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:13.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.750+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc7a009c2b0 con 0x7fc7a810d8b0 2024-03-08T23:14:13.995 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.992+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a810fe90 con 0x7fc7a810d8b0 2024-03-08T23:14:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:14:13.997 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:14:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.993+0000 7fc7a4c00640 1 -- 172.21.15.46:0/2208639738 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fc7a009bbb0 con 0x7fc7a810d8b0 2024-03-08T23:14:13.997 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:14:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 msgr2=0x7fc78c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 0x7fc78c0a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc7980096c0 tx=0x7fc798009260 comp rx=0 tx=0).stop 2024-03-08T23:14:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 msgr2=0x7fc7a81a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:13.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 0x7fc7a81a2a80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fc7a0009fd0 tx=0x7fc7a000ef10 comp rx=0 tx=0).stop 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 shutdown_connections 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc78c0a2970 0x7fc78c0a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7a810d8b0 0x7fc7a81a2a80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc7a8108260 0x7fc7a819e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 --2- 172.21.15.46:0/2208639738 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc7a81078a0 0x7fc7a819e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 >> 172.21.15.46:0/2208639738 conn(0x7fc7a8075840 msgr2=0x7fc7a8075fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 shutdown_connections 2024-03-08T23:14:14.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:13.996+0000 7fc7ae800640 1 -- 172.21.15.46:0/2208639738 wait complete. 2024-03-08T23:14:14.020 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:14.020 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 -- 172.21.15.46:0/3516825247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c8101d80 msgr2=0x7f56c8105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 --2- 172.21.15.46:0/3516825247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c8101d80 0x7f56c8105dc0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f56b0009900 tx=0x7f56b002bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:14.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 -- 172.21.15.46:0/3516825247 shutdown_connections 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 --2- 172.21.15.46:0/3516825247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81064a0 0x7f56c810e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 --2- 172.21.15.46:0/3516825247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c8101d80 0x7f56c8105dc0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 --2- 172.21.15.46:0/3516825247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c81013d0 0x7f56c81017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.238+0000 7f56c7200640 1 -- 172.21.15.46:0/3516825247 >> 172.21.15.46:0/3516825247 conn(0x7f56c80fcc90 msgr2=0x7f56c80ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.239+0000 7f56c7200640 1 -- 172.21.15.46:0/3516825247 shutdown_connections 2024-03-08T23:14:14.243 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.239+0000 7f56c7200640 1 -- 172.21.15.46:0/3516825247 wait complete. 2024-03-08T23:14:14.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.239+0000 7f56c7200640 1 Processor -- start 2024-03-08T23:14:14.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 -- start start 2024-03-08T23:14:14.245 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 0x7f56c819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81064a0 0x7f56c819dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 0x7f56c8197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8111580 con 0x7f56c81064a0 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8111400 con 0x7f56c81013d0 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c7200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c8111700 con 0x7f56c8197900 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c6800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 0x7f56c8197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 0x7f56c819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.246 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c6800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 0x7f56c8197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57894/0 (socket says 172.21.15.46:57894) 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56c6800640 1 -- 172.21.15.46:0/2806780658 learned_addr learned my addr 172.21.15.46:0/2806780658 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.240+0000 7f56bd400640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81064a0 0x7f56c819dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 -- 172.21.15.46:0/2806780658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 msgr2=0x7f56c8197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 0x7f56c8197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 -- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81064a0 msgr2=0x7f56c819dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81064a0 0x7f56c819dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 -- 172.21.15.46:0/2806780658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c8198420 con 0x7f56c81013d0 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 0x7f56c819d700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e9f0 tx=0x7f56b800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.247 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56bf400640 1 -- 172.21.15.46:0/2806780658 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b800ccc0 con 0x7f56c81013d0 2024-03-08T23:14:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c8198710 con 0x7f56c81013d0 2024-03-08T23:14:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56bf400640 1 -- 172.21.15.46:0/2806780658 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56b8005020 con 0x7f56c81013d0 2024-03-08T23:14:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.241+0000 7f56bf400640 1 -- 172.21.15.46:0/2806780658 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b80049e0 con 0x7f56c81013d0 2024-03-08T23:14:14.248 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.242+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c81a48c0 con 0x7f56c81013d0 2024-03-08T23:14:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 Processor -- start 2024-03-08T23:14:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56bf400640 1 -- 172.21.15.46:0/2806780658 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f56b8005560 con 0x7f56c81013d0 2024-03-08T23:14:14.250 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 -- start start 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c58357a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 0x55c58358a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5834dedd0 con 0x55c583579f10 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c583487c70 con 0x55c58357a8c0 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.245+0000 7f56cfe43740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5834dae70 con 0x55c583587e40 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53116/0 (socket says 172.21.15.46:53116) 2024-03-08T23:14:14.251 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c5e00640 1 -- 172.21.15.46:0/2674544500 learned_addr learned my addr 172.21.15.46:0/2674544500 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c6800640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c58357a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 0x55c58358a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 -- 172.21.15.46:0/2674544500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 msgr2=0x55c583587900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 -- 172.21.15.46:0/2674544500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 msgr2=0x55c58357a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c58357a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 -- 172.21.15.46:0/2674544500 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5835412b0 con 0x55c583587e40 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.252 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56c6800640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c58357a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.246+0000 7f56bd400640 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 0x55c58358a230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036040 tx=0x7f56b0009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ce8fc3ab46b2b672 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56bf400640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 0x7f56ac0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56b7400640 1 -- 172.21.15.46:0/2674544500 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b0045070 con 0x55c583587e40 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56bf400640 1 -- 172.21.15.46:0/2806780658 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f56b80cf230 con 0x7f56c81013d0 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56c6800640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 0x7f56ac0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56b7400640 1 -- 172.21.15.46:0/2674544500 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56b002c650 con 0x55c583587e40 2024-03-08T23:14:14.253 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.247+0000 7f56b7400640 1 -- 172.21.15.46:0/2674544500 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b00049b0 con 0x55c583587e40 2024-03-08T23:14:14.257 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- 172.21.15.46:0/2674544500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 msgr2=0x55c58358a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56c6800640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 0x7f56ac0a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56c00049d0 tx=0x7f56c0009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 0x55c58358a230 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f56b0036040 tx=0x7f56b0009560 comp rx=0 tx=0).stop 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- 172.21.15.46:0/2674544500 shutdown_connections 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583587e40 0x55c58358a230 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c58357a8c0 0x55c583587900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2674544500 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c58357a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- 172.21.15.46:0/2674544500 >> 172.21.15.46:0/2674544500 conn(0x55c583574c60 msgr2=0x55c583575040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.258 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- 172.21.15.46:0/2674544500 shutdown_connections 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- 172.21.15.46:0/2674544500 wait complete. 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.248+0000 7f56cfe43740 1 -- start start 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 Processor -- start 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c583584b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c583585120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58358aba0 con 0x55c58357a8c0 2024-03-08T23:14:14.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58349b0c0 con 0x55c583587e40 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c58358ad20 con 0x55c583579f10 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.249+0000 7f56cfe43740 10 client.0 _ll_register_callbacks cb 0x55c5834d5ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56bd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c583584b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57916/0 (socket says 172.21.15.46:57916) 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c583585120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 -- 172.21.15.46:0/769264711 learned_addr learned my addr 172.21.15.46:0/769264711 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 -- 172.21.15.46:0/769264711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 msgr2=0x55c583585120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c583585120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 -- 172.21.15.46:0/769264711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 msgr2=0x55c583584b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c583584b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 -- 172.21.15.46:0/769264711 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5831f33f0 con 0x55c583579f10 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56bd400640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c583584b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56c5e00640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cc620 tx=0x7f56b80d28a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=728a530fe47ef05c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56abe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b801c070 con 0x55c583579f10 2024-03-08T23:14:14.261 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f56b80a3990 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.250+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56b002c650 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56b809b250 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56cfe43740 10 client.26893 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c5832bd7d0 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56b002c650 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f56b80b40f0 con 0x55c583579f10 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56cfe43740 10 client.26893 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.252+0000 7f56cfe43740 10 client.26893 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56cfe43740 20 client.26893 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c583597e80 0x55c58359a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56cfe43740 4 client.26893 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5834b34f0 con 0x55c583597e80 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56bd400640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c583597e80 0x55c58359a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56bd400640 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c583597e80 0x55c58359a290 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=de52b17894bbac84 server_cookie=91b9e7c44158ed6 in_seq=0 out_seq=0 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56abe00640 10 client.26893 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56abe00640 1 -- 172.21.15.46:0/769264711 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c5834b34f0 con 0x55c583597e80 2024-03-08T23:14:14.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56abe00640 10 client.26893 handle_command_reply: tid=0 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.253+0000 7f56cfe43740 1 client.26893 shutdown 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 msgr2=0x55c5835877f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f56b80cc620 tx=0x7f56b80d28a0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 shutdown_connections 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c583597e80 0x55c58359a290 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c583585120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c583584b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 --2- 172.21.15.46:0/769264711 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c5835877f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56abe00640 0 client.26893 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 >> 172.21.15.46:0/769264711 conn(0x55c583574c60 msgr2=0x55c5835769b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 shutdown_connections 2024-03-08T23:14:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 1 -- 172.21.15.46:0/769264711 wait complete. 2024-03-08T23:14:14.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.254+0000 7f56cfe43740 20 client.26893 trim_cache size 0 max 16384 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.269+0000 7f56cfe43740 1 Processor -- start 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.269+0000 7f56cfe43740 1 -- start start 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c58357a8c0 0x55c5835872b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c5835877f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81ae750 con 0x55c583579f10 2024-03-08T23:14:14.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81ac2f0 con 0x55c583587e40 2024-03-08T23:14:14.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56cfe43740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81aa3b0 con 0x55c58357a8c0 2024-03-08T23:14:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c5835877f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36442/0 (socket says 172.21.15.46:36442) 2024-03-08T23:14:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c5e00640 1 -- 172.21.15.46:0/2116463115 learned_addr learned my addr 172.21.15.46:0/2116463115 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56bd400640 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c58357a8c0 0x55c5835872b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 -- 172.21.15.46:0/2116463115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c58357a8c0 msgr2=0x55c5835872b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c58357a8c0 0x55c5835872b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 -- 172.21.15.46:0/2116463115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 msgr2=0x55c5835847d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c6800640 1 -- 172.21.15.46:0/2116463115 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c583585490 con 0x55c583587e40 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.270+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.271+0000 7f56c6800640 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c5835877f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f56c000f790 tx=0x7f56c001a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=702ab9acc95eee4f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.271+0000 7f56b6a00640 1 -- 172.21.15.46:0/2116463115 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c0004b20 con 0x55c583587e40 2024-03-08T23:14:14.277 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.271+0000 7f56b6a00640 1 -- 172.21.15.46:0/2116463115 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56c0004cc0 con 0x55c583587e40 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.271+0000 7f56b6a00640 1 -- 172.21.15.46:0/2116463115 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c00053a0 con 0x55c583587e40 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 -- 172.21.15.46:0/2116463115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 msgr2=0x55c5835877f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c5835877f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f56c000f790 tx=0x7f56c001a000 comp rx=0 tx=0).stop 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 -- 172.21.15.46:0/2116463115 shutdown_connections 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c5835877f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c58357a8c0 0x55c5835872b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2116463115 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c583579f10 0x55c5835847d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 -- 172.21.15.46:0/2116463115 >> 172.21.15.46:0/2116463115 conn(0x55c583574c60 msgr2=0x55c5835840c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 -- 172.21.15.46:0/2116463115 shutdown_connections 2024-03-08T23:14:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.272+0000 7f56cfe43740 1 -- 172.21.15.46:0/2116463115 wait complete. 2024-03-08T23:14:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 -- start start 2024-03-08T23:14:14.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 Processor -- start 2024-03-08T23:14:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c58358de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c58358e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81aa9b0 con 0x55c58357a8c0 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81abc90 con 0x55c583587e40 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c81aafd0 con 0x55c583579f10 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56cfe43740 10 client.0 _ll_register_callbacks cb 0x7f56b8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.273+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57936/0 (socket says 172.21.15.46:57936) 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56bd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c58358de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 -- 172.21.15.46:0/2888374421 learned_addr learned my addr 172.21.15.46:0/2888374421 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c6800640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c58358e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 -- 172.21.15.46:0/2888374421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 msgr2=0x55c58358e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c58358e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 -- 172.21.15.46:0/2888374421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 msgr2=0x55c58358de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c58358de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 -- 172.21.15.46:0/2888374421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5835ee380 con 0x55c583579f10 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56c5e00640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56b809b180 tx=0x7f56b80d2bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ed2300937588994 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56abe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:14.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56b801c070 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f56b80a39a0 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.274+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56c0004cc0 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56b80cca00 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 10 client.26899 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c5835ef570 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56c0004cc0 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f56b809f120 con 0x55c583579f10 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 10 client.26899 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 10 client.26899 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 20 client.26899 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c5835efc60 0x55c5835f2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 4 client.26899 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c5834ec310 con 0x55c5835efc60 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.275+0000 7f56bd400640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c5835efc60 0x55c5835f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56bd400640 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c5835efc60 0x55c5835f2080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c3da8848f0ec197d server_cookie=88c6462a7f3a5fa0 in_seq=0 out_seq=0 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56abe00640 10 client.26899 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:14.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56abe00640 1 -- 172.21.15.46:0/2888374421 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c5834ec310 con 0x55c5835efc60 2024-03-08T23:14:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56abe00640 10 client.26899 handle_command_reply: tid=0 2024-03-08T23:14:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 client.26899 shutdown 2024-03-08T23:14:14.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 msgr2=0x55c58358d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f56b809b180 tx=0x7f56b80d2bc0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 shutdown_connections 2024-03-08T23:14:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55c5835efc60 0x55c5835f2080 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c583587e40 0x55c58358e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c58357a8c0 0x55c58358de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56abe00640 0 client.26899 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 --2- 172.21.15.46:0/2888374421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c583579f10 0x55c58358d8d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.276+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 >> 172.21.15.46:0/2888374421 conn(0x55c583574c60 msgr2=0x55c583584950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 shutdown_connections 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56cfe43740 1 -- 172.21.15.46:0/2888374421 wait complete. 2024-03-08T23:14:14.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56cfe43740 20 client.26899 trim_cache size 0 max 16384 2024-03-08T23:14:14.291 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 msgr2=0x7f56ac0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 0x7f56ac0a4e30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f56c00049d0 tx=0x7f56c0009500 comp rx=0 tx=0).stop 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 msgr2=0x7f56c819d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.277+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 0x7f56c819d700 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f56b800e9f0 tx=0x7f56b800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 shutdown_connections 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f56ac0a2970 0x7f56ac0a4e30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f56c8197900 0x7f56c8197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f56c81064a0 0x7f56c819dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 --2- 172.21.15.46:0/2806780658 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f56c81013d0 0x7f56c819d700 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 >> 172.21.15.46:0/2806780658 conn(0x7f56c80fcc90 msgr2=0x7f56c80ff000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 shutdown_connections 2024-03-08T23:14:14.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.278+0000 7f56c7200640 1 -- 172.21.15.46:0/2806780658 wait complete. 2024-03-08T23:14:14.299 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:14.299 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:14:14.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 -- 172.21.15.46:0/2702372468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c10d8a0 msgr2=0x7fc53c10dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 --2- 172.21.15.46:0/2702372468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c10d8a0 0x7fc53c10dc80 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fc52c009cd0 tx=0x7fc52c02c0f0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 -- 172.21.15.46:0/2702372468 shutdown_connections 2024-03-08T23:14:14.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 --2- 172.21.15.46:0/2702372468 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c108250 0x7fc53c114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 --2- 172.21.15.46:0/2702372468 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c107890 0x7fc53c107d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 --2- 172.21.15.46:0/2702372468 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c10d8a0 0x7fc53c10dc80 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 -- 172.21.15.46:0/2702372468 >> 172.21.15.46:0/2702372468 conn(0x7fc53c075830 msgr2=0x7fc53c077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 -- 172.21.15.46:0/2702372468 shutdown_connections 2024-03-08T23:14:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.519+0000 7fc542e00640 1 -- 172.21.15.46:0/2702372468 wait complete. 2024-03-08T23:14:14.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.520+0000 7fc542e00640 1 Processor -- start 2024-03-08T23:14:14.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 -- start start 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 0x7fc53c19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c117c70 con 0x7fc53c108250 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c117af0 con 0x7fc53c10d8a0 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc542e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c117df0 con 0x7fc53c107890 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc53b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc540e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc53be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 0x7fc53c19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc53b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36472/0 (socket says 172.21.15.46:36472) 2024-03-08T23:14:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc540e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53158/0 (socket says 172.21.15.46:53158) 2024-03-08T23:14:14.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc53be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 0x7fc53c19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57958/0 (socket says 172.21.15.46:57958) 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.521+0000 7fc53b400640 1 -- 172.21.15.46:0/3904283976 learned_addr learned my addr 172.21.15.46:0/3904283976 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 -- 172.21.15.46:0/3904283976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 msgr2=0x7fc53c19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 0x7fc53c19e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 -- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 msgr2=0x7fc53c19e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 -- 172.21.15.46:0/3904283976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53c1a3100 con 0x7fc53c10d8a0 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc53b400640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc540e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc53400c970 tx=0x7fc53400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc538c00640 1 -- 172.21.15.46:0/3904283976 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534007bb0 con 0x7fc53c10d8a0 2024-03-08T23:14:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c1a33f0 con 0x7fc53c10d8a0 2024-03-08T23:14:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc538c00640 1 -- 172.21.15.46:0/3904283976 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc534007d50 con 0x7fc53c10d8a0 2024-03-08T23:14:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.522+0000 7fc538c00640 1 -- 172.21.15.46:0/3904283976 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534018770 con 0x7fc53c10d8a0 2024-03-08T23:14:14.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.523+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53c1aac80 con 0x7fc53c10d8a0 2024-03-08T23:14:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc538c00640 1 -- 172.21.15.46:0/3904283976 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc53400f450 con 0x7fc53c10d8a0 2024-03-08T23:14:14.539 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 Processor -- start 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 -- start start 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68c1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f6825dd0 con 0x55f5f68c1c70 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f67cec70 con 0x55f5f68ca850 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc545a7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f6821e70 con 0x55f5f68c1350 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.540 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36476/0 (socket says 172.21.15.46:36476) 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 -- 172.21.15.46:0/1660130768 learned_addr learned my addr 172.21.15.46:0/1660130768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53b400640 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53b400640 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 -- 172.21.15.46:0/1660130768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 msgr2=0x55f5f68c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 -- 172.21.15.46:0/1660130768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 msgr2=0x55f5f68ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc53be00640 1 -- 172.21.15.46:0/1660130768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f68882b0 con 0x55f5f68c1c70 2024-03-08T23:14:14.541 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.527+0000 7fc538c00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 0x7fc5200a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc538c00640 1 -- 172.21.15.46:0/3904283976 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc5340cf100 con 0x7fc53c10d8a0 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc53be00640 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c700 tx=0x7fc52c032440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=684d36c5295a572d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc53b400640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 0x7fc5200a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc531600640 1 -- 172.21.15.46:0/1660130768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c043070 con 0x55f5f68c1c70 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc531600640 1 -- 172.21.15.46:0/1660130768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52c0098e0 con 0x55f5f68c1c70 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.528+0000 7fc531600640 1 -- 172.21.15.46:0/1660130768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c039040 con 0x55f5f68c1c70 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 -- 172.21.15.46:0/1660130768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 msgr2=0x55f5f68ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.542 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc53b400640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 0x7fc5200a4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55f5f68bcc10 tx=0x7fc528005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc52c02c700 tx=0x7fc52c032440 comp rx=0 tx=0).stop 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 -- 172.21.15.46:0/1660130768 shutdown_connections 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68ccc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68ca310 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1660130768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 -- 172.21.15.46:0/1660130768 >> 172.21.15.46:0/1660130768 conn(0x55f5f68bba80 msgr2=0x55f5f68bbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 -- 172.21.15.46:0/1660130768 shutdown_connections 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.529+0000 7fc545a7f740 1 -- 172.21.15.46:0/1660130768 wait complete. 2024-03-08T23:14:14.543 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 -- start start 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 Processor -- start 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68c7450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 0x55f5f68c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f68cd5e0 con 0x55f5f68c1c70 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f67e20c0 con 0x55f5f68c1350 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5f68cd760 con 0x55f5f68ca850 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.530+0000 7fc545a7f740 10 client.0 _ll_register_callbacks cb 0x55f5f681cec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.544 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53174/0 (socket says 172.21.15.46:53174) 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 -- 172.21.15.46:0/3283222070 learned_addr learned my addr 172.21.15.46:0/3283222070 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc540e00640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 0x55f5f68c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 -- 172.21.15.46:0/3283222070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 msgr2=0x55f5f68c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 0x55f5f68c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 -- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 msgr2=0x55f5f68c7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68c7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.545 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 -- 172.21.15.46:0/3283222070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f653a3f0 con 0x55f5f68c1350 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc540e00640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 0x55f5f68c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc53be00640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc52c037040 tx=0x7fc52c005c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fcd608409c37f24 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc526000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.531+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c04c030 con 0x55f5f68c1350 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc52c0327a0 con 0x55f5f68c1350 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc52c0098e0 con 0x55f5f68c1350 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc52c059050 con 0x55f5f68c1350 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 10 client.26936 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.546 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f5f66047d0 con 0x55f5f68c1350 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52c0098e0 con 0x55f5f68c1350 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc52c03f440 con 0x55f5f68c1350 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 10 client.26936 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 10 client.26936 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 20 client.26936 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f68da630 0x55f5f68dca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 4 client.26936 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.532+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5f67fa4f0 con 0x55f5f68da630 2024-03-08T23:14:14.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.533+0000 7fc53b400640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f68da630 0x55f5f68dca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.533+0000 7fc53b400640 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f68da630 0x55f5f68dca20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25d299cab393be21 server_cookie=3e785e20a6f6dbeb in_seq=0 out_seq=0 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.533+0000 7fc526000640 10 client.26936 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc526000640 1 -- 172.21.15.46:0/3283222070 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5f67fa4f0 con 0x55f5f68da630 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc526000640 10 client.26936 handle_command_reply: tid=0 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 client.26936 shutdown 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 msgr2=0x55f5f68ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fc52c037040 tx=0x7fc52c005c50 comp rx=0 tx=0).stop 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 shutdown_connections 2024-03-08T23:14:14.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f68da630 0x55f5f68dca20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68ca850 0x55f5f68c7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68c7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc526000640 0 client.26936 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3283222070 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68ca0c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.534+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 >> 172.21.15.46:0/3283222070 conn(0x55f5f68bba80 msgr2=0x55f5f68bd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.536+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 shutdown_connections 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.536+0000 7fc545a7f740 1 -- 172.21.15.46:0/3283222070 wait complete. 2024-03-08T23:14:14.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.536+0000 7fc545a7f740 20 client.26936 trim_cache size 0 max 16384 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 Processor -- start 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 -- start start 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1c70 0x55f5f68c28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 0x55f5f68c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b6fa0 con 0x55f5f68ca850 2024-03-08T23:14:14.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b4b40 con 0x55f5f68c1350 2024-03-08T23:14:14.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.548+0000 7fc545a7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b2c00 con 0x55f5f68c1c70 2024-03-08T23:14:14.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc540e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 0x55f5f68c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc540e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 0x55f5f68c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36498/0 (socket says 172.21.15.46:36498) 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53b400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1c70 0x55f5f68c28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53188/0 (socket says 172.21.15.46:53188) 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc540e00640 1 -- 172.21.15.46:0/1612667201 learned_addr learned my addr 172.21.15.46:0/1612667201 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 -- 172.21.15.46:0/1612667201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1c70 msgr2=0x55f5f68c28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1c70 0x55f5f68c28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 -- 172.21.15.46:0/1612667201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 msgr2=0x55f5f68c9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 0x55f5f68c9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 -- 172.21.15.46:0/1612667201 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5f68c59a0 con 0x55f5f68c1350 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.549+0000 7fc53be00640 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc52c000f80 tx=0x7fc52c0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fcc24a4a7c46f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc530c00640 1 -- 172.21.15.46:0/1612667201 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c04c070 con 0x55f5f68c1350 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc530c00640 1 -- 172.21.15.46:0/1612667201 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52c03f440 con 0x55f5f68c1350 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc530c00640 1 -- 172.21.15.46:0/1612667201 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52c039070 con 0x55f5f68c1350 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 -- 172.21.15.46:0/1612667201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 msgr2=0x55f5f68c65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fc52c000f80 tx=0x7fc52c0095c0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 -- 172.21.15.46:0/1612667201 shutdown_connections 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68ca850 0x55f5f68c9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1c70 0x55f5f68c28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 --2- 172.21.15.46:0/1612667201 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68c1350 0x55f5f68c65b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.550+0000 7fc545a7f740 1 -- 172.21.15.46:0/1612667201 >> 172.21.15.46:0/1612667201 conn(0x55f5f68bba80 msgr2=0x55f5f68c2190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.551+0000 7fc545a7f740 1 -- 172.21.15.46:0/1612667201 shutdown_connections 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.551+0000 7fc545a7f740 1 -- 172.21.15.46:0/1612667201 wait complete. 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.551+0000 7fc545a7f740 1 -- start start 2024-03-08T23:14:14.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.551+0000 7fc545a7f740 1 Processor -- start 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b3200 con 0x55f5f68c1c70 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b44e0 con 0x55f5f68ca850 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53c1b3820 con 0x55f5f68c1350 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc545a7f740 10 client.0 _ll_register_callbacks cb 0x7fc528005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36508/0 (socket says 172.21.15.46:36508) 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 -- 172.21.15.46:0/3111519541 learned_addr learned my addr 172.21.15.46:0/3111519541 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc540e00640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53be00640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 -- 172.21.15.46:0/3111519541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 msgr2=0x55f5f68d0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68d0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 -- 172.21.15.46:0/3111519541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 msgr2=0x55f5f68d0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68d0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53b400640 1 -- 172.21.15.46:0/3111519541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5f6930830 con 0x55f5f68c1c70 2024-03-08T23:14:14.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.552+0000 7fc53be00640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:14.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc53b400640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc528011c20 tx=0x7fc5280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9da70f600f65b88 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc526000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc528011d10 con 0x55f5f68c1c70 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc52c03f440 con 0x55f5f68c1c70 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc52802d310 con 0x55f5f68c1c70 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc52803d050 con 0x55f5f68c1c70 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc545a7f740 10 client.16785 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.553+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f5f6931ad0 con 0x55f5f68c1c70 2024-03-08T23:14:14.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc52c03f440 con 0x55f5f68c1c70 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc5280047a0 con 0x55f5f68c1c70 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 10 client.16785 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 10 client.16785 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 20 client.16785 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f6932180 0x55f5f69345a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 4 client.16785 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f5f6833310 con 0x55f5f6932180 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.554+0000 7fc53be00640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f6932180 0x55f5f69345a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc53be00640 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f6932180 0x55f5f69345a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=786ed84288b64e80 server_cookie=ccc88dd75291a957 in_seq=0 out_seq=0 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc526000640 10 client.16785 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc526000640 1 -- 172.21.15.46:0/3111519541 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f5f6833310 con 0x55f5f6932180 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc526000640 10 client.16785 handle_command_reply: tid=0 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 client.16785 shutdown 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 msgr2=0x55f5f68d0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fc528011c20 tx=0x7fc5280046c0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 shutdown_connections 2024-03-08T23:14:14.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f5f68ca850 0x55f5f68d0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f5f68c1c70 0x55f5f68d0840 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55f5f6932180 0x55f5f69345a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 --2- 172.21.15.46:0/3111519541 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f5f68c1350 0x55f5f68d0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 >> 172.21.15.46:0/3111519541 conn(0x55f5f68bba80 msgr2=0x55f5f68c2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc526000640 0 client.16785 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 shutdown_connections 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.555+0000 7fc545a7f740 1 -- 172.21.15.46:0/3111519541 wait complete. 2024-03-08T23:14:14.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc545a7f740 20 client.16785 trim_cache size 0 max 16384 2024-03-08T23:14:14.568 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:14.569 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:14 smithi192 ceph-mon[36700]: pgmap v197: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:14:14.569 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:14 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2208639738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:14.569 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:14 smithi192 ceph-mon[36700]: osdmap e96: 12 total, 12 up, 12 in 2024-03-08T23:14:14.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 msgr2=0x7fc5200a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 0x7fc5200a4eb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x55f5f68bcc10 tx=0x7fc528005f90 comp rx=0 tx=0).stop 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 msgr2=0x7fc53c1a29d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc53400c970 tx=0x7fc53400ce30 comp rx=0 tx=0).stop 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 shutdown_connections 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc5200a29f0 0x7fc5200a4eb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc53c10d8a0 0x7fc53c1a29d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc53c108250 0x7fc53c19e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 --2- 172.21.15.46:0/3904283976 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc53c107890 0x7fc53c19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 >> 172.21.15.46:0/3904283976 conn(0x7fc53c075830 msgr2=0x7fc53c075fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.556+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 shutdown_connections 2024-03-08T23:14:14.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.557+0000 7fc542e00640 1 -- 172.21.15.46:0/3904283976 wait complete. 2024-03-08T23:14:14.575 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:14 smithi046 ceph-mon[31859]: pgmap v197: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:14:14.575 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:14 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2208639738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:14.575 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:14 smithi046 ceph-mon[31859]: osdmap e96: 12 total, 12 up, 12 in 2024-03-08T23:14:14.577 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:14.578 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:14:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:14 smithi062 ceph-mon[35761]: pgmap v197: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:14:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:14 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2208639738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:14 smithi062 ceph-mon[35761]: osdmap e96: 12 total, 12 up, 12 in 2024-03-08T23:14:14.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 -- 172.21.15.46:0/3349424942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d410b740 msgr2=0x7ff8d410dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3349424942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d410b740 0x7ff8d410dbe0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc009ca0 tx=0x7ff8cc02c180 comp rx=0 tx=0).stop 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 -- 172.21.15.46:0/3349424942 shutdown_connections 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3349424942 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d410b740 0x7ff8d410dbe0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3349424942 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d410b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3349424942 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d4102320 0x7ff8d4102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 -- 172.21.15.46:0/3349424942 >> 172.21.15.46:0/3349424942 conn(0x7ff8d40fbb40 msgr2=0x7ff8d40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 -- 172.21.15.46:0/3349424942 shutdown_connections 2024-03-08T23:14:14.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.790+0000 7ff8d3200640 1 -- 172.21.15.46:0/3349424942 wait complete. 2024-03-08T23:14:14.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 Processor -- start 2024-03-08T23:14:14.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 -- start start 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d4102320 0x7ff8d40fff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d41004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d4111500 con 0x7ff8d410b740 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d4111380 con 0x7ff8d4102320 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.791+0000 7ff8d3200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d4111680 con 0x7ff8d4102c40 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36524/0 (socket says 172.21.15.46:36524) 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 -- 172.21.15.46:0/3104336213 learned_addr learned my addr 172.21.15.46:0/3104336213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d1400640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d41004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d4102320 0x7ff8d40fff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 -- 172.21.15.46:0/3104336213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 msgr2=0x7ff8d41004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d41004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 -- 172.21.15.46:0/3104336213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d4102320 msgr2=0x7ff8d40fff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d4102320 0x7ff8d40fff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d2800640 1 -- 172.21.15.46:0/3104336213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d41a5f20 con 0x7ff8d410b740 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.792+0000 7ff8d1400640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8d2800640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc02c790 tx=0x7ff8cc035e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8caa00640 1 -- 172.21.15.46:0/3104336213 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc039040 con 0x7ff8d410b740 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d41a61b0 con 0x7ff8d410b740 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8caa00640 1 -- 172.21.15.46:0/3104336213 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8cc044070 con 0x7ff8d410b740 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8caa00640 1 -- 172.21.15.46:0/3104336213 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8cc032940 con 0x7ff8d410b740 2024-03-08T23:14:14.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.793+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d41a66a0 con 0x7ff8d410b740 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.796+0000 7ff8caa00640 1 -- 172.21.15.46:0/3104336213 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff8cc032ae0 con 0x7ff8d410b740 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.796+0000 7ff8dbbd1740 1 Processor -- start 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.796+0000 7ff8dbbd1740 1 -- start start 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d19254f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1927e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d1880dd0 con 0x55e4d191c530 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d1829c70 con 0x55e4d1925a30 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d187ce70 con 0x55e4d191ce50 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8d1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8d1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36526/0 (socket says 172.21.15.46:36526) 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8d1400640 1 -- 172.21.15.46:0/26914650 learned_addr learned my addr 172.21.15.46:0/26914650 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d19254f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.797+0000 7ff8d1400640 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1927e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 -- 172.21.15.46:0/26914650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 msgr2=0x55e4d19254f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d19254f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 -- 172.21.15.46:0/26914650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 msgr2=0x55e4d1927e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1927e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 -- 172.21.15.46:0/26914650 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4d18e32b0 con 0x55e4d191c530 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8caa00640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 0x7ff8b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1400640 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0013990 tx=0x7ff8c0009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cdf5877e30a901 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8caa00640 1 -- 172.21.15.46:0/3104336213 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff8cc0f1550 con 0x7ff8d410b740 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 0x7ff8b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8bf400640 1 -- 172.21.15.46:0/26914650 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000f8c0 con 0x55e4d191c530 2024-03-08T23:14:14.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8bf400640 1 -- 172.21.15.46:0/26914650 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c0005020 con 0x55e4d191c530 2024-03-08T23:14:14.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.798+0000 7ff8bf400640 1 -- 172.21.15.46:0/26914650 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0004b40 con 0x55e4d191c530 2024-03-08T23:14:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/26914650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 msgr2=0x55e4d191c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 0x7ff8b80a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55e4d191c000 tx=0x7ff8c4005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0013990 tx=0x7ff8c0009d70 comp rx=0 tx=0).stop 2024-03-08T23:14:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/26914650 shutdown_connections 2024-03-08T23:14:14.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1927e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d19254f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/26914650 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d191c910 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/26914650 >> 172.21.15.46:0/26914650 conn(0x55e4d1916c60 msgr2=0x55e4d1917040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/26914650 shutdown_connections 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.799+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/26914650 wait complete. 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.800+0000 7ff8dbbd1740 1 -- start start 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.800+0000 7ff8dbbd1740 1 Processor -- start 2024-03-08T23:14:14.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 0x55e4d1925310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d1925a30 0x55e4d1922c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d1928820 con 0x55e4d191ce50 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d183d0c0 con 0x55e4d191c530 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4d19289a0 con 0x55e4d1925a30 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8dbbd1740 10 client.0 _ll_register_callbacks cb 0x55e4d1877ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8d1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8d1e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 0x55e4d1925310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8d1400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36536/0 (socket says 172.21.15.46:36536) 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8d1400640 1 -- 172.21.15.46:0/63160135 learned_addr learned my addr 172.21.15.46:0/63160135 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.801+0000 7ff8d2800640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d1925a30 0x55e4d1922c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 -- 172.21.15.46:0/63160135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d1925a30 msgr2=0x55e4d1922c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d1925a30 0x55e4d1922c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 -- 172.21.15.46:0/63160135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 msgr2=0x55e4d1922710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 -- 172.21.15.46:0/63160135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4d15953f0 con 0x55e4d191c530 2024-03-08T23:14:14.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1400640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 0x55e4d1925310 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff8c401e9d0 tx=0x7ff8c4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a7dd776ada29aa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c402e060 con 0x55e4d191c530 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff8c4018030 con 0x55e4d191c530 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.802+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8c0005020 con 0x55e4d191c530 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8c4035050 con 0x55e4d191c530 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 10 client.26954 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e4d165f7d0 con 0x55e4d191c530 2024-03-08T23:14:14.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c0005020 con 0x55e4d191c530 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff8c401c560 con 0x55e4d191c530 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 10 client.26954 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 10 client.26954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 20 client.26954 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d1935ac0 0x55e4d1937ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 4 client.26954 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.803+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4d18554f0 con 0x55e4d1935ac0 2024-03-08T23:14:14.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.804+0000 7ff8d1400640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d1935ac0 0x55e4d1937ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.804+0000 7ff8d1400640 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d1935ac0 0x55e4d1937ef0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a62c04ca0f2f7808 server_cookie=306ac479a167ac95 in_seq=0 out_seq=0 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.804+0000 7ff8b3e00640 10 client.26954 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8b3e00640 1 -- 172.21.15.46:0/63160135 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e4d18554f0 con 0x55e4d1935ac0 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8b3e00640 10 client.26954 handle_command_reply: tid=0 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 client.26954 shutdown 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 msgr2=0x55e4d1925310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 0x55e4d1925310 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff8c401e9d0 tx=0x7ff8c4004730 comp rx=0 tx=0).stop 2024-03-08T23:14:14.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 shutdown_connections 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d1935ac0 0x55e4d1937ef0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d1925a30 0x55e4d1922c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191ce50 0x55e4d1922710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/63160135 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d191c530 0x55e4d1925310 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 >> 172.21.15.46:0/63160135 conn(0x55e4d1916c60 msgr2=0x55e4d19189b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.805+0000 7ff8b3e00640 0 client.26954 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.806+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 shutdown_connections 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.806+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/63160135 wait complete. 2024-03-08T23:14:14.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.806+0000 7ff8dbbd1740 20 client.26954 trim_cache size 0 max 16384 2024-03-08T23:14:14.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 Processor -- start 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 -- start start 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d1921000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1921540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41b2ac0 con 0x55e4d191c530 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41b0660 con 0x55e4d1925a30 2024-03-08T23:14:14.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41ae720 con 0x55e4d191ce50 2024-03-08T23:14:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36544/0 (socket says 172.21.15.46:36544) 2024-03-08T23:14:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8d1e00640 1 -- 172.21.15.46:0/3533353962 learned_addr learned my addr 172.21.15.46:0/3533353962 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.819+0000 7ff8d1400640 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d1921000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d2800640 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1921540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 -- 172.21.15.46:0/3533353962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 msgr2=0x55e4d1921000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d1921000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 -- 172.21.15.46:0/3533353962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 msgr2=0x55e4d1921540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1921540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 -- 172.21.15.46:0/3533353962 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4d191cc80 con 0x55e4d191c530 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff8c401ced0 tx=0x7ff8c401cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e6e600c873107c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8bea00640 1 -- 172.21.15.46:0/3533353962 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c403d070 con 0x55e4d191c530 2024-03-08T23:14:14.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.820+0000 7ff8bea00640 1 -- 172.21.15.46:0/3533353962 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c401cb20 con 0x55e4d191c530 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8bea00640 1 -- 172.21.15.46:0/3533353962 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c4029db0 con 0x55e4d191c530 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/3533353962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 msgr2=0x55e4d1924e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff8c401ced0 tx=0x7ff8c401cf50 comp rx=0 tx=0).stop 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/3533353962 shutdown_connections 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d1921540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d1921000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/3533353962 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d1924e40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/3533353962 >> 172.21.15.46:0/3533353962 conn(0x55e4d1916c60 msgr2=0x55e4d19226d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/3533353962 shutdown_connections 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.821+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/3533353962 wait complete. 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 -- start start 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 Processor -- start 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d192b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d192c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41aed20 con 0x55e4d191c530 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41b0000 con 0x55e4d1925a30 2024-03-08T23:14:14.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d41af340 con 0x55e4d191ce50 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.822+0000 7ff8dbbd1740 10 client.0 _ll_register_callbacks cb 0x7ff8c0009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d192b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36554/0 (socket says 172.21.15.46:36554) 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1e00640 1 -- 172.21.15.46:0/2856269149 learned_addr learned my addr 172.21.15.46:0/2856269149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d2800640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d192c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 -- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 msgr2=0x55e4d192c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d192c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 -- 172.21.15.46:0/2856269149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 msgr2=0x55e4d192b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 -- 172.21.15.46:0/2856269149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4d192cc60 con 0x55e4d191ce50 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8d1400640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d192b7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55e4d1938800 tx=0x7ff8c0009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86d25c0d1deb9898 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8b3e00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.823+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0032060 con 0x55e4d191ce50 2024-03-08T23:14:14.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff8c401cb20 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff8c002d680 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8c0039050 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 10 client.26932 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e4d198ca80 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8c401cb20 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff8c002dce0 con 0x55e4d191ce50 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 10 client.26932 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:14.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 10 client.26932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:14.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 20 client.26932 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:14.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d198d020 0x55e4d198f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:14.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 4 client.26932 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:14.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.824+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e4d188e310 con 0x55e4d198d020 2024-03-08T23:14:14.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.825+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d198d020 0x55e4d198f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:14.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.825+0000 7ff8d1e00640 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d198d020 0x55e4d198f440 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd3cadb0806ef9f1 server_cookie=cea9e732ae6f1092 in_seq=0 out_seq=0 2024-03-08T23:14:14.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.825+0000 7ff8b3e00640 10 client.26932 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.825+0000 7ff8b3e00640 1 -- 172.21.15.46:0/2856269149 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e4d188e310 con 0x55e4d198d020 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8b3e00640 10 client.26932 handle_command_reply: tid=0 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 client.26932 shutdown 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 msgr2=0x55e4d192b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d192b7a0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x55e4d1938800 tx=0x7ff8c0009bf0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 shutdown_connections 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e4d198d020 0x55e4d198f440 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e4d1925a30 0x55e4d192c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e4d191ce50 0x55e4d192b7a0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 --2- 172.21.15.46:0/2856269149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e4d191c530 0x55e4d192b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 >> 172.21.15.46:0/2856269149 conn(0x55e4d1916c60 msgr2=0x55e4d1920f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8b3e00640 0 client.26932 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 shutdown_connections 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 1 -- 172.21.15.46:0/2856269149 wait complete. 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.826+0000 7ff8dbbd1740 20 client.26932 trim_cache size 0 max 16384 2024-03-08T23:14:14.836 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:14.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 msgr2=0x7ff8b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 0x7ff8b80a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x55e4d191c000 tx=0x7ff8c4005f90 comp rx=0 tx=0).stop 2024-03-08T23:14:14.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 msgr2=0x7ff8d41009f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:14.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff8cc02c790 tx=0x7ff8cc035e00 comp rx=0 tx=0).stop 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 shutdown_connections 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff8b80a2970 0x7ff8b80a4e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff8d410b740 0x7ff8d41009f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff8d4102c40 0x7ff8d41004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 --2- 172.21.15.46:0/3104336213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff8d4102320 0x7ff8d40fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 >> 172.21.15.46:0/3104336213 conn(0x7ff8d40fbb40 msgr2=0x7ff8d40fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 shutdown_connections 2024-03-08T23:14:14.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:14.827+0000 7ff8d3200640 1 -- 172.21.15.46:0/3104336213 wait complete. 2024-03-08T23:14:15.849 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:14:15.849 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:14:15.849 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:14:16.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 -- 172.21.15.46:0/3959123385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 msgr2=0x7f1188110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 --2- 172.21.15.46:0/3959123385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f1188110280 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1180009ca0 tx=0x7f118002c100 comp rx=0 tx=0).stop 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 -- 172.21.15.46:0/3959123385 shutdown_connections 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 --2- 172.21.15.46:0/3959123385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f1188110280 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 --2- 172.21.15.46:0/3959123385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1188103f90 0x7f1188107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 --2- 172.21.15.46:0/3959123385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11881035e0 0x7f11881039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 -- 172.21.15.46:0/3959123385 >> 172.21.15.46:0/3959123385 conn(0x7f11880fee20 msgr2=0x7f1188101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 -- 172.21.15.46:0/3959123385 shutdown_connections 2024-03-08T23:14:16.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.087+0000 7f118ec00640 1 -- 172.21.15.46:0/3959123385 wait complete. 2024-03-08T23:14:16.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.088+0000 7f118ec00640 1 Processor -- start 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.088+0000 7f118ec00640 1 -- start start 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f118819fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188199a30 0x7f1188199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1188113750 con 0x7f11881035e0 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881135d0 con 0x7f11881086b0 2024-03-08T23:14:16.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ec00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881138d0 con 0x7f1188199a30 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36574/0 (socket says 172.21.15.46:36574) 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 -- 172.21.15.46:0/878014770 learned_addr learned my addr 172.21.15.46:0/878014770 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ce00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f118819fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118e200640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188199a30 0x7f1188199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 -- 172.21.15.46:0/878014770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188199a30 msgr2=0x7f1188199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188199a30 0x7f1188199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 -- 172.21.15.46:0/878014770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 msgr2=0x7f118819fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f118819fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118d800640 1 -- 172.21.15.46:0/878014770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118819a550 con 0x7f11881035e0 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.089+0000 7f118ce00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f118819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f118d800640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e9f0 tx=0x7f117c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f1186000640 1 -- 172.21.15.46:0/878014770 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c00ccc0 con 0x7f11881035e0 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f1186000640 1 -- 172.21.15.46:0/878014770 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f117c00ce60 con 0x7f11881035e0 2024-03-08T23:14:16.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f1186000640 1 -- 172.21.15.46:0/878014770 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c010690 con 0x7f11881035e0 2024-03-08T23:14:16.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118819a840 con 0x7f11881035e0 2024-03-08T23:14:16.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.090+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11881a6a40 con 0x7f11881035e0 2024-03-08T23:14:16.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.093+0000 7f1193039740 1 Processor -- start 2024-03-08T23:14:16.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1186000640 1 -- 172.21.15.46:0/878014770 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f117c0046c0 con 0x7f11881035e0 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 -- start start 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 0x55df43c7d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43bd8a50 con 0x55df43c7dca0 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43b81b00 con 0x55df43c747a0 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f1193039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43bd4af0 con 0x55df43c750c0 2024-03-08T23:14:16.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118d800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 0x55df43c7d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118e200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53288/0 (socket says 172.21.15.46:53288) 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36576/0 (socket says 172.21.15.46:36576) 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118d800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 0x55df43c7d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:58096/0 (socket says 172.21.15.46:58096) 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.094+0000 7f118e200640 1 -- 172.21.15.46:0/2108999497 learned_addr learned my addr 172.21.15.46:0/2108999497 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 -- 172.21.15.46:0/2108999497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 msgr2=0x55df43c7d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 0x55df43c7d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 -- 172.21.15.46:0/2108999497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 msgr2=0x55df43c74b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 -- 172.21.15.46:0/2108999497 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df43b9f1e0 con 0x55df43c7dca0 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118e200640 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f118ce00640 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f118819b220 tx=0x7f1174011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d04c0ce721ac3e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f1186000640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 0x7f11700a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.095+0000 7f1186000640 1 -- 172.21.15.46:0/878014770 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f117c0cf640 con 0x7f11881035e0 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f117aa00640 1 -- 172.21.15.46:0/2108999497 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174004230 con 0x55df43c7dca0 2024-03-08T23:14:16.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f118e200640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 0x7f11700a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f117aa00640 1 -- 172.21.15.46:0/2108999497 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11740043d0 con 0x55df43c7dca0 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f117aa00640 1 -- 172.21.15.46:0/2108999497 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740035f0 con 0x55df43c7dca0 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 -- 172.21.15.46:0/2108999497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 msgr2=0x55df43c80090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f118819b220 tx=0x7f1174011810 comp rx=0 tx=0).stop 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f118e200640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 0x7f11700a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1180009a60 tx=0x7f1180005d80 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 -- 172.21.15.46:0/2108999497 shutdown_connections 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c7dca0 0x55df43c80090 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c750c0 0x55df43c7d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 --2- 172.21.15.46:0/2108999497 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43c74b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.096+0000 7f1193039740 1 -- 172.21.15.46:0/2108999497 >> 172.21.15.46:0/2108999497 conn(0x55df43c6eed0 msgr2=0x55df43c6f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.097+0000 7f1193039740 1 -- 172.21.15.46:0/2108999497 shutdown_connections 2024-03-08T23:14:16.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.097+0000 7f1193039740 1 -- 172.21.15.46:0/2108999497 wait complete. 2024-03-08T23:14:16.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.097+0000 7f1193039740 1 -- start start 2024-03-08T23:14:16.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.097+0000 7f1193039740 1 Processor -- start 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c747a0 0x55df43c7ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 0x55df43c7b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43c80a90 con 0x55df43c750c0 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43b94f60 con 0x55df43c7dca0 2024-03-08T23:14:16.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df43c80c10 con 0x55df43c747a0 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f1193039740 10 client.0 _ll_register_callbacks cb 0x55df43b80b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f118ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f118ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36592/0 (socket says 172.21.15.46:36592) 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f118ce00640 1 -- 172.21.15.46:0/371583493 learned_addr learned my addr 172.21.15.46:0/371583493 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f118e200640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 0x55df43c7b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.098+0000 7f118d800640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c747a0 0x55df43c7ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 -- 172.21.15.46:0/371583493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c747a0 msgr2=0x55df43c7ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c747a0 0x55df43c7ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 -- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 msgr2=0x55df43c7b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 0x55df43c7b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 -- 172.21.15.46:0/371583493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df439b57e0 con 0x55df43c750c0 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118e200640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 0x55df43c7b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f118ce00640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f117400fc40 tx=0x7f117400fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8077e8e97edb1f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f116f400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174028020 con 0x55df43c750c0 2024-03-08T23:14:16.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f1174004ee0 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11740043d0 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.099+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1174033050 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 10 client.16815 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55df43a15f70 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11740043d0 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1174019700 con 0x55df43c750c0 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 10 client.16815 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 10 client.16815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:16.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 20 client.16815 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43c8e010 0x55df43c90400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 4 client.16815 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df43be5f90 con 0x55df43c8e010 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.100+0000 7f118d800640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43c8e010 0x55df43c90400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.101+0000 7f118d800640 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43c8e010 0x55df43c90400 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33954851cd2995ea server_cookie=ca32c36aecaf9dd3 in_seq=0 out_seq=0 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.101+0000 7f116f400640 10 client.16815 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f116f400640 1 -- 172.21.15.46:0/371583493 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55df43be5f90 con 0x55df43c8e010 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f116f400640 10 client.16815 handle_command_reply: tid=0 2024-03-08T23:14:16.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 client.16815 shutdown 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 msgr2=0x55df43c7b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f117400fc40 tx=0x7f117400fc70 comp rx=0 tx=0).stop 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 shutdown_connections 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43c8e010 0x55df43c90400 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c7dca0 0x55df43c7b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43c7b260 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 --2- 172.21.15.46:0/371583493 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c747a0 0x55df43c7ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 >> 172.21.15.46:0/371583493 conn(0x55df43c6eed0 msgr2=0x55df43c70c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f116f400640 0 client.16815 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 shutdown_connections 2024-03-08T23:14:16.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 1 -- 172.21.15.46:0/371583493 wait complete. 2024-03-08T23:14:16.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.102+0000 7f1193039740 20 client.16815 trim_cache size 0 max 16384 2024-03-08T23:14:16.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.119+0000 7f1193039740 1 Processor -- start 2024-03-08T23:14:16.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.119+0000 7f1193039740 1 -- start start 2024-03-08T23:14:16.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c747a0 0x55df43c7d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c78540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881b2b70 con 0x55df43c747a0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881b0710 con 0x55df43c750c0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f1193039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881ae7d0 con 0x55df43c7dca0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c747a0 0x55df43c7d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118e200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c78540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53308/0 (socket says 172.21.15.46:53308) 2024-03-08T23:14:16.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 -- 172.21.15.46:0/2518127019 learned_addr learned my addr 172.21.15.46:0/2518127019 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 -- 172.21.15.46:0/2518127019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 msgr2=0x55df43c78540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c78540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 -- 172.21.15.46:0/2518127019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c747a0 msgr2=0x55df43c7d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c747a0 0x55df43c7d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.120+0000 7f118ce00640 1 -- 172.21.15.46:0/2518127019 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df438ad170 con 0x55df43c750c0 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.121+0000 7f118ce00640 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1174019ad0 tx=0x7f117400f920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37b6e74e333cc4a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.121+0000 7f117a000640 1 -- 172.21.15.46:0/2518127019 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117403c070 con 0x55df43c750c0 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.121+0000 7f117a000640 1 -- 172.21.15.46:0/2518127019 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f117401f040 con 0x55df43c750c0 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.121+0000 7f117a000640 1 -- 172.21.15.46:0/2518127019 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174028040 con 0x55df43c750c0 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- 172.21.15.46:0/2518127019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 msgr2=0x55df43c78000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f1174019ad0 tx=0x7f117400f920 comp rx=0 tx=0).stop 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- 172.21.15.46:0/2518127019 shutdown_connections 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c78540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c750c0 0x55df43c78000 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 --2- 172.21.15.46:0/2518127019 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c747a0 0x55df43c7d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- 172.21.15.46:0/2518127019 >> 172.21.15.46:0/2518127019 conn(0x55df43c6eed0 msgr2=0x55df43c79650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- 172.21.15.46:0/2518127019 shutdown_connections 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- 172.21.15.46:0/2518127019 wait complete. 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 -- start start 2024-03-08T23:14:16.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.122+0000 7f1193039740 1 Processor -- start 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43cdeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c84050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881aedd0 con 0x55df43c750c0 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881b00b0 con 0x55df43c747a0 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11881af3f0 con 0x55df43c7dca0 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f1193039740 10 client.0 _ll_register_callbacks cb 0x7f1174009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:16.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f118d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f118ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43cdeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f118d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53318/0 (socket says 172.21.15.46:53318) 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f118d800640 1 -- 172.21.15.46:0/803292876 learned_addr learned my addr 172.21.15.46:0/803292876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.123+0000 7f118e200640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c84050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 -- 172.21.15.46:0/803292876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 msgr2=0x55df43c84050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c84050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 -- 172.21.15.46:0/803292876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 msgr2=0x55df43cdeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43cdeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 -- 172.21.15.46:0/803292876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df43ba4050 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f118d800640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f117c09b580 tx=0x7f117c086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=942c77bf9623ff63 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f116f400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c09b880 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f117c0b4670 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f117401f040 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f117c09f9d0 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f1193039740 10 client.26978 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.124+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55df43ce5e00 con 0x55df43c747a0 2024-03-08T23:14:16.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f117401f040 con 0x55df43c747a0 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f117c0ccbf0 con 0x55df43c747a0 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 10 client.26978 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 10 client.26978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 20 client.26978 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43ce6480 0x55df43ce8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 4 client.26978 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55df43be02e0 con 0x55df43ce6480 2024-03-08T23:14:16.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.125+0000 7f118ce00640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43ce6480 0x55df43ce8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.126+0000 7f118ce00640 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43ce6480 0x55df43ce8870 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba8b61583f1d399f server_cookie=4e2f222138181755 in_seq=0 out_seq=0 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.126+0000 7f116f400640 10 client.26978 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.127+0000 7f116f400640 1 -- 172.21.15.46:0/803292876 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55df43be02e0 con 0x55df43ce6480 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.127+0000 7f116f400640 10 client.26978 handle_command_reply: tid=0 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.127+0000 7f1193039740 1 client.26978 shutdown 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 msgr2=0x55df43cde760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f117c09b580 tx=0x7f117c086040 comp rx=0 tx=0).stop 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 shutdown_connections 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55df43ce6480 0x55df43ce8870 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55df43c7dca0 0x55df43c84050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55df43c750c0 0x55df43cdeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f116f400640 0 client.26978 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 --2- 172.21.15.46:0/803292876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55df43c747a0 0x55df43cde760 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 >> 172.21.15.46:0/803292876 conn(0x55df43c6eed0 msgr2=0x55df43c6d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 shutdown_connections 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 1 -- 172.21.15.46:0/803292876 wait complete. 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.128+0000 7f1193039740 20 client.26978 trim_cache size 0 max 16384 2024-03-08T23:14:16.136 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:14:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "07ea3577-bb26-4c2b-a63f-a61a0de29a7c", 2024-03-08T23:14:16.137 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:14:16.137 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 msgr2=0x7f11700a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 0x7f11700a4eb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1180009a60 tx=0x7f1180005d80 comp rx=0 tx=0).stop 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 msgr2=0x7f118819f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f117c00e9f0 tx=0x7f117c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 shutdown_connections 2024-03-08T23:14:16.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11700a29f0 0x7f11700a4eb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1188199a30 0x7f1188199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f11881086b0 0x7f118819fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 --2- 172.21.15.46:0/878014770 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f11881035e0 0x7f118819f880 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 >> 172.21.15.46:0/878014770 conn(0x7f11880fee20 msgr2=0x7f11881010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 shutdown_connections 2024-03-08T23:14:16.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.129+0000 7f118ec00640 1 -- 172.21.15.46:0/878014770 wait complete. 2024-03-08T23:14:16.156 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:14:16.156 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:14:16.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 -- 172.21.15.46:0/3812638051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6108109570 msgr2=0x7f6108109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 --2- 172.21.15.46:0/3812638051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6108109570 0x7f6108109950 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f60f8009930 tx=0x7f60f802bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 -- 172.21.15.46:0/3812638051 shutdown_connections 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 --2- 172.21.15.46:0/3812638051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108103e70 0x7f6108110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 --2- 172.21.15.46:0/3812638051 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f61081034b0 0x7f6108103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 --2- 172.21.15.46:0/3812638051 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6108109570 0x7f6108109950 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 -- 172.21.15.46:0/3812638051 >> 172.21.15.46:0/3812638051 conn(0x7f61080fef60 msgr2=0x7f6108101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 -- 172.21.15.46:0/3812638051 shutdown_connections 2024-03-08T23:14:16.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.443+0000 7f610d400640 1 -- 172.21.15.46:0/3812638051 wait complete. 2024-03-08T23:14:16.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.444+0000 7f610d400640 1 Processor -- start 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.444+0000 7f610d400640 1 -- start start 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6108103e70 0x7f61081a3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108109570 0x7f610810f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61080774b0 con 0x7f6108103e70 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6108077330 con 0x7f61081034b0 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f610d400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6108077630 con 0x7f6108109570 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f6106800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.448 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f6106800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53324/0 (socket says 172.21.15.46:53324) 2024-03-08T23:14:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f6106800640 1 -- 172.21.15.46:0/1443286310 learned_addr learned my addr 172.21.15.46:0/1443286310 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f6105e00640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6108103e70 0x7f61081a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.445+0000 7f6107200640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108109570 0x7f610810f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 -- 172.21.15.46:0/1443286310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108109570 msgr2=0x7f610810f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108109570 0x7f610810f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 -- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6108103e70 msgr2=0x7f61081a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6108103e70 0x7f61081a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 -- 172.21.15.46:0/1443286310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6108110060 con 0x7f61081034b0 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.446+0000 7f6106800640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f60f8000c00 tx=0x7f60f8004360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.447+0000 7f60ff400640 1 -- 172.21.15.46:0/1443286310 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8043020 con 0x7f61081034b0 2024-03-08T23:14:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.447+0000 7f60ff400640 1 -- 172.21.15.46:0/1443286310 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60f80054e0 con 0x7f61081034b0 2024-03-08T23:14:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.447+0000 7f60ff400640 1 -- 172.21.15.46:0/1443286310 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f8003b20 con 0x7f61081034b0 2024-03-08T23:14:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.447+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61081102f0 con 0x7f61081034b0 2024-03-08T23:14:16.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.447+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61081aade0 con 0x7f61081034b0 2024-03-08T23:14:16.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 Processor -- start 2024-03-08T23:14:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 -- start start 2024-03-08T23:14:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f60ff400640 1 -- 172.21.15.46:0/1443286310 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f60f8046020 con 0x7f61081034b0 2024-03-08T23:14:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747ea0 0x55f372750540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f372752e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f372709410 con 0x55f372747580 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3726abd50 con 0x55f372747ea0 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.453+0000 7f611019c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f372654b80 con 0x55f372750a80 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6107200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6107200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:36630/0 (socket says 172.21.15.46:36630) 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6107200640 1 -- 172.21.15.46:0/1397047393 learned_addr learned my addr 172.21.15.46:0/1397047393 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6105e00640 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f372752e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6106800640 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747ea0 0x55f372750540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.454+0000 7f6107200640 1 -- 172.21.15.46:0/1397047393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 msgr2=0x55f372752e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f6107200640 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f372752e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f6107200640 1 -- 172.21.15.46:0/1397047393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747ea0 msgr2=0x55f372750540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f6107200640 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747ea0 0x55f372750540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f6107200640 1 -- 172.21.15.46:0/1397047393 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f37248a7d0 con 0x55f372747580 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f6107200640 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6100009f60 tx=0x7f6100009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d870371b7ca66ef3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.455+0000 7f60ff400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 0x7f60ec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.456+0000 7f60ff400640 1 -- 172.21.15.46:0/1443286310 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f60f80f0e60 con 0x7f61081034b0 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.456+0000 7f60f3400640 1 -- 172.21.15.46:0/1397047393 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610001e070 con 0x55f372747580 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.456+0000 7f60f3400640 1 -- 172.21.15.46:0/1397047393 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6100003e50 con 0x55f372747580 2024-03-08T23:14:16.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.456+0000 7f60f3400640 1 -- 172.21.15.46:0/1397047393 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610000f7a0 con 0x55f372747580 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.456+0000 7f6106800640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 0x7f60ec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 -- 172.21.15.46:0/1397047393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 msgr2=0x55f372747960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6100009f60 tx=0x7f6100009f90 comp rx=0 tx=0).stop 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 -- 172.21.15.46:0/1397047393 shutdown_connections 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f372752e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747ea0 0x55f372750540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 --2- 172.21.15.46:0/1397047393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f372747960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.457+0000 7f611019c740 1 -- 172.21.15.46:0/1397047393 >> 172.21.15.46:0/1397047393 conn(0x55f372741cb0 msgr2=0x55f372742090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.458+0000 7f6106800640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 0x7f60ec0a4eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f372747050 tx=0x7f60f80035f0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.458+0000 7f611019c740 1 -- 172.21.15.46:0/1397047393 shutdown_connections 2024-03-08T23:14:16.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.458+0000 7f611019c740 1 -- 172.21.15.46:0/1397047393 wait complete. 2024-03-08T23:14:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.458+0000 7f611019c740 1 -- start start 2024-03-08T23:14:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.459+0000 7f611019c740 1 Processor -- start 2024-03-08T23:14:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f37274dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f37274e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f372667ee0 con 0x55f372747ea0 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3726a7df0 con 0x55f372750a80 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f372753870 con 0x55f372747580 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f6107200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f6107200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:58170/0 (socket says 172.21.15.46:58170) 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f6107200640 1 -- 172.21.15.46:0/2047503223 learned_addr learned my addr 172.21.15.46:0/2047503223 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f6107200640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f37274e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.460+0000 7f611019c740 10 client.0 _ll_register_callbacks cb 0x55f37266e8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:16.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6105e00640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f37274dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 -- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 msgr2=0x55f37274e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f37274e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 -- 172.21.15.46:0/2047503223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 msgr2=0x55f37274dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f37274dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 -- 172.21.15.46:0/2047503223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3723c03f0 con 0x55f372747580 2024-03-08T23:14:16.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6105e00640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f37274dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:14:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.461+0000 7f6107200640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6100011d60 tx=0x7f6100004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ba831b3b00eef0f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.462+0000 7f60e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.462+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002d060 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.462+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f610000f540 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.462+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6100003e50 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.463+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6100034050 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.463+0000 7f611019c740 10 client.26962 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.463+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f3723800f0 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.463+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f610001f040 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6100019740 con 0x55f372747580 2024-03-08T23:14:16.467 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 10 client.26962 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 10 client.26962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 20 client.26962 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f372760b90 0x55f372762f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 4 client.26962 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:16.468 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.464+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6100003e50 con 0x55f372760b90 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.465+0000 7f6105e00640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f372760b90 0x55f372762f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.466+0000 7f6105e00640 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f372760b90 0x55f372762f80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a30dfe6d4b4acc3 server_cookie=32fbfe3b2ff065a6 in_seq=0 out_seq=0 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.466+0000 7f60e7e00640 10 client.26962 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f60e7e00640 1 -- 172.21.15.46:0/2047503223 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f6100003e50 con 0x55f372760b90 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f60e7e00640 10 client.26962 handle_command_reply: tid=0 2024-03-08T23:14:16.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 client.26962 shutdown 2024-03-08T23:14:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 msgr2=0x55f37274daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6100011d60 tx=0x7f6100004700 comp rx=0 tx=0).stop 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 shutdown_connections 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f372760b90 0x55f372762f80 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f37274e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f37274dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 --2- 172.21.15.46:0/2047503223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747580 0x55f37274daa0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 >> 172.21.15.46:0/2047503223 conn(0x55f372741cb0 msgr2=0x55f372740f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.467+0000 7f60e7e00640 0 client.26962 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.468+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 shutdown_connections 2024-03-08T23:14:16.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.468+0000 7f611019c740 1 -- 172.21.15.46:0/2047503223 wait complete. 2024-03-08T23:14:16.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.468+0000 7f611019c740 20 client.26962 trim_cache size 0 max 16384 2024-03-08T23:14:16.502 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.500+0000 7f611019c740 1 Processor -- start 2024-03-08T23:14:16.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.500+0000 7f611019c740 1 -- start start 2024-03-08T23:14:16.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.500+0000 7f611019c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f3727501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.503 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.500+0000 7f611019c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f611019c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f372757b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f611019c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b7190 con 0x55f372747580 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f611019c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b4d30 con 0x55f372750a80 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f611019c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b2df0 con 0x55f372747ea0 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6107200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f3727501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:58172/0 (socket says 172.21.15.46:58172) 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 -- 172.21.15.46:0/240348408 learned_addr learned my addr 172.21.15.46:0/240348408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.504 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6107200640 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f372757b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 -- 172.21.15.46:0/240348408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 msgr2=0x55f372757b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f372757b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 -- 172.21.15.46:0/240348408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 msgr2=0x55f3727501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f3727501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6105e00640 1 -- 172.21.15.46:0/240348408 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f372752e70 con 0x55f372747ea0 2024-03-08T23:14:16.505 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.501+0000 7f6107200640 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f372757b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:16.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.502+0000 7f6105e00640 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55f37274d300 tx=0x7f60f400bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41084bf90b47aa48 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.506 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.502+0000 7f60f2a00640 1 -- 172.21.15.46:0/240348408 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4003de0 con 0x55f372747ea0 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.502+0000 7f60f2a00640 1 -- 172.21.15.46:0/240348408 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60f401c890 con 0x55f372747ea0 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.502+0000 7f60f2a00640 1 -- 172.21.15.46:0/240348408 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4010430 con 0x55f372747ea0 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 -- 172.21.15.46:0/240348408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 msgr2=0x55f37274c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55f37274d300 tx=0x7f60f400bf80 comp rx=0 tx=0).stop 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 -- 172.21.15.46:0/240348408 shutdown_connections 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372750a80 0x55f372757b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372747ea0 0x55f37274c950 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 --2- 172.21.15.46:0/240348408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747580 0x55f3727501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.507 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.503+0000 7f611019c740 1 -- 172.21.15.46:0/240348408 >> 172.21.15.46:0/240348408 conn(0x55f372741cb0 msgr2=0x55f37274c240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.504+0000 7f611019c740 1 -- 172.21.15.46:0/240348408 shutdown_connections 2024-03-08T23:14:16.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.504+0000 7f611019c740 1 -- 172.21.15.46:0/240348408 wait complete. 2024-03-08T23:14:16.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.504+0000 7f611019c740 1 -- start start 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 Processor -- start 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f3727b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f3727aae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b33f0 con 0x55f372747ea0 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b46d0 con 0x55f372747580 2024-03-08T23:14:16.510 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61081b3a10 con 0x55f372750a80 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f611019c740 10 client.0 _ll_register_callbacks cb 0x7f6100009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f6107200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f6107200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53362/0 (socket says 172.21.15.46:53362) 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f6107200640 1 -- 172.21.15.46:0/3720024439 learned_addr learned my addr 172.21.15.46:0/3720024439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f6105e00640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f3727b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.505+0000 7f6107200640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f3727aae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 -- 172.21.15.46:0/3720024439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 msgr2=0x55f3727aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f3727aae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.511 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 -- 172.21.15.46:0/3720024439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 msgr2=0x55f3727b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f3727b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 -- 172.21.15.46:0/3720024439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3727ab530 con 0x55f372747580 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f6107200640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6100004870 tx=0x7f61000049a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6818875a245be109 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f60e7e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.506+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002d040 con 0x55f372747580 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f6100004310 con 0x55f372747580 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60f401c890 con 0x55f372747580 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f610001e070 con 0x55f372747580 2024-03-08T23:14:16.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f611019c740 10 client.26996 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f3727b80a0 con 0x55f372747580 2024-03-08T23:14:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.507+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60f401c890 con 0x55f372747580 2024-03-08T23:14:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6100028a40 con 0x55f372747580 2024-03-08T23:14:16.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 10 client.26996 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 10 client.26996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 20 client.26996 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f3727b8740 0x55f3727bab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:16.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 4 client.26996 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.508+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f3726b9290 con 0x55f3727b8740 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.509+0000 7f6105e00640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f3727b8740 0x55f3727bab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f6105e00640 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f3727b8740 0x55f3727bab60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694e663f0efd7f79 server_cookie=a9d4f8a292cdc1de in_seq=0 out_seq=0 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f60e7e00640 10 client.26996 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f60e7e00640 1 -- 172.21.15.46:0/3720024439 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f3726b9290 con 0x55f3727b8740 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f60e7e00640 10 client.26996 handle_command_reply: tid=0 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 client.26996 shutdown 2024-03-08T23:14:16.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 msgr2=0x55f3727b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6100004870 tx=0x7f61000049a0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 shutdown_connections 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55f3727b8740 0x55f3727bab60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55f372750a80 0x55f3727aae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55f372747ea0 0x55f3727b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f60e7e00640 0 client.26996 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 --2- 172.21.15.46:0/3720024439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55f372747580 0x55f3727b08d0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 >> 172.21.15.46:0/3720024439 conn(0x55f372741cb0 msgr2=0x55f372740f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 shutdown_connections 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.510+0000 7f611019c740 1 -- 172.21.15.46:0/3720024439 wait complete. 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f611019c740 20 client.26996 trim_cache size 0 max 16384 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:14:16.517 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: "07ea3577-bb26-4c2b-a63f-a61a0de29a7c": { 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "07ea3577-bb26-4c2b-a63f-a61a0de29a7c", 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:14:16.518 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:16.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 msgr2=0x7f60ec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 0x7f60ec0a4eb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f372747050 tx=0x7f60f80035f0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 msgr2=0x7f61081a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:16.519 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f60f8000c00 tx=0x7f60f8004360 comp rx=0 tx=0).stop 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 shutdown_connections 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f60ec0a29f0 0x7f60ec0a4eb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6108109570 0x7f610810f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6108103e70 0x7f61081a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 --2- 172.21.15.46:0/1443286310 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f61081034b0 0x7f61081a2d00 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 >> 172.21.15.46:0/1443286310 conn(0x7f61080fef60 msgr2=0x7f61081009f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.511+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 shutdown_connections 2024-03-08T23:14:16.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:16.512+0000 7f610d400640 1 -- 172.21.15.46:0/1443286310 wait complete. 2024-03-08T23:14:16.528 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:16 smithi046 ceph-mon[31859]: pgmap v199: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:14:16.530 INFO:tasks.cephfs.filesystem:scrub status for tag:07ea3577-bb26-4c2b-a63f-a61a0de29a7c - {'path': '/', 'tag': '07ea3577-bb26-4c2b-a63f-a61a0de29a7c', 'options': 'recursive,force'} 2024-03-08T23:14:16.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:16 smithi192 ceph-mon[36700]: pgmap v199: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:14:16.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:16 smithi062 ceph-mon[35761]: pgmap v199: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:14:17.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:17 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:17 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:14:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:17.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:17.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:17 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:18.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:18 smithi192 ceph-mon[36700]: pgmap v200: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2024-03-08T23:14:18.628 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:18 smithi046 ceph-mon[31859]: pgmap v200: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2024-03-08T23:14:18.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:18 smithi062 ceph-mon[35761]: pgmap v200: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 218 B/s wr, 6 op/s 2024-03-08T23:14:20.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:20 smithi062 ceph-mon[35761]: pgmap v201: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:20.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:20 smithi046 ceph-mon[31859]: pgmap v201: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:20.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:20 smithi192 ceph-mon[36700]: pgmap v201: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:22 smithi062 ceph-mon[35761]: pgmap v202: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:22.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:22 smithi046 ceph-mon[31859]: pgmap v202: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:22.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:22 smithi192 ceph-mon[36700]: pgmap v202: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:14:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:24 smithi062 ceph-mon[35761]: pgmap v203: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-08T23:14:24.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:24 smithi046 ceph-mon[31859]: pgmap v203: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-08T23:14:24.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:24 smithi192 ceph-mon[36700]: pgmap v203: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-08T23:14:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:26 smithi062 ceph-mon[35761]: pgmap v204: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:14:26.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:26 smithi046 ceph-mon[31859]: pgmap v204: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:14:26.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:26 smithi192 ceph-mon[36700]: pgmap v204: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:14:27.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:27 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:14:27.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:27 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:14:27.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:27 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:14:27.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:27 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:14:27.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:27 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:14:27.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:27 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:14:28.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:28 smithi062 ceph-mon[35761]: pgmap v205: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-08T23:14:28.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:28 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:14:28.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:28 smithi046 ceph-mon[31859]: pgmap v205: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-08T23:14:28.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:28 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:14:28.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:28 smithi192 ceph-mon[36700]: pgmap v205: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-08T23:14:28.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:28 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:14:29.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:29 smithi062 ceph-mon[35761]: pgmap v206: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:14:29.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:29 smithi046 ceph-mon[31859]: pgmap v206: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:14:29.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:29 smithi192 ceph-mon[36700]: pgmap v206: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 7 op/s 2024-03-08T23:14:31.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:31 smithi046 ceph-mon[31859]: pgmap v207: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:14:32.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:31 smithi192 ceph-mon[36700]: pgmap v207: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:14:32.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:31 smithi062 ceph-mon[35761]: pgmap v207: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:14:33.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:33 smithi046 ceph-mon[31859]: pgmap v208: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:34.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:33 smithi192 ceph-mon[36700]: pgmap v208: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:34.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:33 smithi062 ceph-mon[35761]: pgmap v208: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:35.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:35 smithi046 ceph-mon[31859]: pgmap v209: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:36.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:35 smithi192 ceph-mon[36700]: pgmap v209: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:36.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:35 smithi062 ceph-mon[35761]: pgmap v209: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:37.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:37 smithi046 ceph-mon[31859]: pgmap v210: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:37 smithi192 ceph-mon[36700]: pgmap v210: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:38.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:37 smithi062 ceph-mon[35761]: pgmap v210: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:40.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:39 smithi192 ceph-mon[36700]: pgmap v211: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:40.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:39 smithi062 ceph-mon[35761]: pgmap v211: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:40.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:39 smithi046 ceph-mon[31859]: pgmap v211: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:42.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:41 smithi192 ceph-mon[36700]: pgmap v212: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:42.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:41 smithi062 ceph-mon[35761]: pgmap v212: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:42.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:41 smithi046 ceph-mon[31859]: pgmap v212: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:44.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:43 smithi192 ceph-mon[36700]: pgmap v213: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:44.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:43 smithi062 ceph-mon[35761]: pgmap v213: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:44.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:43 smithi046 ceph-mon[31859]: pgmap v213: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:46.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:14:46.532 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:14:46.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:46 smithi192 ceph-mon[36700]: pgmap v214: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:46.625 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:46 smithi046 ceph-mon[31859]: pgmap v214: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:46.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:46 smithi062 ceph-mon[35761]: pgmap v214: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:46.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.771+0000 7fbaddc00640 1 -- 172.21.15.46:0/3259133876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad81034b0 msgr2=0x7fbad8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.771+0000 7fbaddc00640 1 --2- 172.21.15.46:0/3259133876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad81034b0 0x7fbad8103930 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009900 tx=0x7fbac402bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 -- 172.21.15.46:0/3259133876 shutdown_connections 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 --2- 172.21.15.46:0/3259133876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad8103e70 0x7fbad8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 --2- 172.21.15.46:0/3259133876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad81034b0 0x7fbad8103930 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 --2- 172.21.15.46:0/3259133876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 0x7fbad8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 -- 172.21.15.46:0/3259133876 >> 172.21.15.46:0/3259133876 conn(0x7fbad80fefa0 msgr2=0x7fbad81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 -- 172.21.15.46:0/3259133876 shutdown_connections 2024-03-08T23:14:46.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.772+0000 7fbaddc00640 1 -- 172.21.15.46:0/3259133876 wait complete. 2024-03-08T23:14:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.773+0000 7fbaddc00640 1 Processor -- start 2024-03-08T23:14:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.773+0000 7fbaddc00640 1 -- start start 2024-03-08T23:14:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.773+0000 7fbaddc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbaddc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad8103e70 0x7fbad819fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbaddc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 0x7fbad8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbaddc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81138a0 con 0x7fbad8103e70 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbaddc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8113720 con 0x7fbad81034b0 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbaddc00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad8113a20 con 0x7fbad8109570 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34170/0 (socket says 172.21.15.46:34170) 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 -- 172.21.15.46:0/2448279385 learned_addr learned my addr 172.21.15.46:0/2448279385 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6000640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad8103e70 0x7fbad819fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad7400640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 0x7fbad8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.779 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 -- 172.21.15.46:0/2448279385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 msgr2=0x7fbad8199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 0x7fbad8199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 -- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad8103e70 msgr2=0x7fbad819fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad8103e70 0x7fbad819fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.774+0000 7fbad6a00640 1 -- 172.21.15.46:0/2448279385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad819a280 con 0x7fbad81034b0 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbad6a00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbac800d770 tx=0x7fbac800dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbacf400640 1 -- 172.21.15.46:0/2448279385 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8017070 con 0x7fbad81034b0 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbacf400640 1 -- 172.21.15.46:0/2448279385 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbac8005500 con 0x7fbad81034b0 2024-03-08T23:14:46.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad819a570 con 0x7fbad81034b0 2024-03-08T23:14:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbacf400640 1 -- 172.21.15.46:0/2448279385 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8004e00 con 0x7fbad81034b0 2024-03-08T23:14:46.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.775+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad81a6810 con 0x7fbad81034b0 2024-03-08T23:14:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.778+0000 7fbae09bd740 1 Processor -- start 2024-03-08T23:14:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 -- start start 2024-03-08T23:14:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4965540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b4967e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b491e410 con 0x55a2b495cea0 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b48c0d50 con 0x55a2b495c580 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbae09bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b4869b80 con 0x55a2b4965a80 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbacf400640 1 -- 172.21.15.46:0/2448279385 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbac8004fa0 con 0x7fbad81034b0 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.779+0000 7fbad7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34180/0 (socket says 172.21.15.46:34180) 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 -- 172.21.15.46:0/2231618437 learned_addr learned my addr 172.21.15.46:0/2231618437 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:46.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad6a00640 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4965540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad6000640 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b4967e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 -- 172.21.15.46:0/2231618437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 msgr2=0x55a2b4967e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b4967e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 -- 172.21.15.46:0/2231618437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 msgr2=0x55a2b4965540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4965540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 -- 172.21.15.46:0/2231618437 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2b469f7d0 con 0x55a2b495c580 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.780+0000 7fbad7400640 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009fd0 tx=0x7fbad00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb73b2d183f6463d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.781+0000 7fbac3e00640 1 -- 172.21.15.46:0/2231618437 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0004d20 con 0x55a2b495c580 2024-03-08T23:14:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.781+0000 7fbac3e00640 1 -- 172.21.15.46:0/2231618437 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0004050 con 0x55a2b495c580 2024-03-08T23:14:46.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.781+0000 7fbac3e00640 1 -- 172.21.15.46:0/2231618437 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0005240 con 0x55a2b495c580 2024-03-08T23:14:46.789 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 -- 172.21.15.46:0/2231618437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 msgr2=0x55a2b495c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009fd0 tx=0x7fbad00118d0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbacf400640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 0x7fbabc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbacf400640 1 -- 172.21.15.46:0/2448279385 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fbac80cec20 con 0x7fbad81034b0 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 -- 172.21.15.46:0/2231618437 shutdown_connections 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b4967e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4965540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 --2- 172.21.15.46:0/2231618437 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b495c960 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 -- 172.21.15.46:0/2231618437 >> 172.21.15.46:0/2231618437 conn(0x55a2b4956cb0 msgr2=0x55a2b4957090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbad7400640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 0x7fbabc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 -- 172.21.15.46:0/2231618437 shutdown_connections 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.782+0000 7fbae09bd740 1 -- 172.21.15.46:0/2231618437 wait complete. 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbae09bd740 1 -- start start 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbad7400640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 0x7fbabc0a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009de0 tx=0x7fbad0009be0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbae09bd740 1 Processor -- start 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4962a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.783+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b49634e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbae09bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b487cee0 con 0x55a2b495cea0 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbae09bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b48bcdf0 con 0x55a2b495c580 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbae09bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a2b4968870 con 0x55a2b4965a80 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbae09bd740 10 client.0 _ll_register_callbacks cb 0x55a2b48838d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52618/0 (socket says 172.21.15.46:52618) 2024-03-08T23:14:46.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 -- 172.21.15.46:0/3537001012 learned_addr learned my addr 172.21.15.46:0/3537001012 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4962a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad7400640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b49634e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 -- 172.21.15.46:0/3537001012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 msgr2=0x55a2b49634e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b49634e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 -- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 msgr2=0x55a2b4962a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4962a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.784+0000 7fbad6000640 1 -- 172.21.15.46:0/3537001012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2b45d53f0 con 0x55a2b495cea0 2024-03-08T23:14:46.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbad6000640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009900 tx=0x7fbac402c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8712c298a5029fad server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbac0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac402ccc0 con 0x55a2b495cea0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fbac40046c0 con 0x55a2b495cea0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbad0004050 con 0x55a2b495cea0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbac40048c0 con 0x55a2b495cea0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbae09bd740 10 client.16848 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.785+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a2b45950f0 con 0x55a2b495cea0 2024-03-08T23:14:46.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0004050 con 0x55a2b495cea0 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbac4046070 con 0x55a2b495cea0 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 10 client.16848 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 20 client.16848 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b4975c70 0x55a2b49780a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 4 client.16848 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a2b48ce290 con 0x55a2b4975c70 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.786+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b4975c70 0x55a2b49780a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.787+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b4975c70 0x55a2b49780a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1253408fe650c9f server_cookie=e83ee5215171dee6 in_seq=0 out_seq=0 2024-03-08T23:14:46.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.787+0000 7fbac0c00640 10 client.16848 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbac0c00640 1 -- 172.21.15.46:0/3537001012 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a2b48ce290 con 0x55a2b4975c70 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbac0c00640 10 client.16848 handle_command_reply: tid=0 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 client.16848 shutdown 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 msgr2=0x55a2b4962fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009900 tx=0x7fbac402c7e0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 shutdown_connections 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b4975c70 0x55a2b49780a0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b4965a80 0x55a2b49634e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b495cea0 0x55a2b4962fa0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3537001012 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4962a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbac0c00640 0 client.16848 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 >> 172.21.15.46:0/3537001012 conn(0x55a2b4956cb0 msgr2=0x55a2b4958a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 shutdown_connections 2024-03-08T23:14:46.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 1 -- 172.21.15.46:0/3537001012 wait complete. 2024-03-08T23:14:46.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.788+0000 7fbae09bd740 20 client.16848 trim_cache size 0 max 16384 2024-03-08T23:14:46.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.805+0000 7fbae09bd740 1 Processor -- start 2024-03-08T23:14:46.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.805+0000 7fbae09bd740 1 -- start start 2024-03-08T23:14:46.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4965190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 0x55a2b496cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81b2c50 con 0x55a2b4965a80 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81b07f0 con 0x55a2b495c580 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbae09bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81ae8b0 con 0x55a2b495cea0 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52620/0 (socket says 172.21.15.46:52620) 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4965190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 -- 172.21.15.46:0/449156342 learned_addr learned my addr 172.21.15.46:0/449156342 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:46.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad6000640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 0x55a2b496cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 -- 172.21.15.46:0/449156342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 msgr2=0x55a2b496cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 0x55a2b496cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 -- 172.21.15.46:0/449156342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 msgr2=0x55a2b4965190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4965190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad7400640 1 -- 172.21.15.46:0/449156342 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a2b4963a30 con 0x55a2b4965a80 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad6000640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 0x55a2b496cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.806+0000 7fbad6a00640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4965190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.807+0000 7fbad7400640 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009c50 tx=0x7fbad0022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ecf4902ba4fd77 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.807+0000 7fbac3400640 1 -- 172.21.15.46:0/449156342 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad0035030 con 0x55a2b4965a80 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.807+0000 7fbac3400640 1 -- 172.21.15.46:0/449156342 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0029070 con 0x55a2b4965a80 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.807+0000 7fbac3400640 1 -- 172.21.15.46:0/449156342 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbad000f4e0 con 0x55a2b4965a80 2024-03-08T23:14:46.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- 172.21.15.46:0/449156342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 msgr2=0x55a2b496cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009c50 tx=0x7fbad0022000 comp rx=0 tx=0).stop 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- 172.21.15.46:0/449156342 shutdown_connections 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b496cff0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495cea0 0x55a2b496cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 --2- 172.21.15.46:0/449156342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495c580 0x55a2b4965190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- 172.21.15.46:0/449156342 >> 172.21.15.46:0/449156342 conn(0x55a2b4956cb0 msgr2=0x55a2b4957090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- 172.21.15.46:0/449156342 shutdown_connections 2024-03-08T23:14:46.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- 172.21.15.46:0/449156342 wait complete. 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.808+0000 7fbae09bd740 1 -- start start 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 Processor -- start 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495cea0 0x55a2b49c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b49c0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81aeeb0 con 0x55a2b4965a80 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81b0190 con 0x55a2b495cea0 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbad81af4d0 con 0x55a2b495c580 2024-03-08T23:14:46.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbae09bd740 10 client.0 _ll_register_callbacks cb 0x7fbad0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbad7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b49c0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbad6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39094/0 (socket says 172.21.15.46:39094) 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbad6a00640 1 -- 172.21.15.46:0/3512431229 learned_addr learned my addr 172.21.15.46:0/3512431229 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.809+0000 7fbad6000640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495cea0 0x55a2b49c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 -- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495cea0 msgr2=0x55a2b49c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495cea0 0x55a2b49c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 -- 172.21.15.46:0/3512431229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 msgr2=0x55a2b49c0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b49c0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 -- 172.21.15.46:0/3512431229 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a2b49c0840 con 0x55a2b495c580 2024-03-08T23:14:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad7400640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b49c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbad6a00640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009610 tx=0x7fbac80048a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bca78a78c44cc0ae server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:46.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbac0c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:46.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.810+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbac8020040 con 0x55a2b495c580 2024-03-08T23:14:46.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fbac8009cc0 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbad0029070 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbac809af10 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 10 client.26986 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a2b49ccf80 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbad0029070 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbac80cc260 con 0x55a2b495c580 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 10 client.26986 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:46.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 10 client.26986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.811+0000 7fbae09bd740 20 client.26986 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b49cd6a0 0x55a2b49cfac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbae09bd740 4 client.26986 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a2b48c85e0 con 0x55a2b49cd6a0 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbad6000640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b49cd6a0 0x55a2b49cfac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbad6000640 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b49cd6a0 0x55a2b49cfac0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86b2dfa078261b9b server_cookie=3da3bdc49b72732 in_seq=0 out_seq=0 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbac0c00640 10 client.26986 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbac0c00640 1 -- 172.21.15.46:0/3512431229 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a2b48c85e0 con 0x55a2b49cd6a0 2024-03-08T23:14:46.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.812+0000 7fbac0c00640 10 client.26986 handle_command_reply: tid=0 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 client.26986 shutdown 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 msgr2=0x55a2b49c5710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fbac8009610 tx=0x7fbac80048a0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 shutdown_connections 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55a2b49cd6a0 0x55a2b49cfac0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a2b4965a80 0x55a2b49c0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a2b495cea0 0x55a2b49c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 --2- 172.21.15.46:0/3512431229 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a2b495c580 0x55a2b49c5710 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbac0c00640 0 client.26986 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 >> 172.21.15.46:0/3512431229 conn(0x55a2b4956cb0 msgr2=0x55a2b4961350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 shutdown_connections 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 1 -- 172.21.15.46:0/3512431229 wait complete. 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.813+0000 7fbae09bd740 20 client.26986 trim_cache size 0 max 16384 2024-03-08T23:14:46.823 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:46.824 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:14:46.824 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:14:46.824 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 msgr2=0x7fbabc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 0x7fbabc0a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbad0009de0 tx=0x7fbad0009be0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 msgr2=0x7fbad819f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fbac800d770 tx=0x7fbac800dc30 comp rx=0 tx=0).stop 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 shutdown_connections 2024-03-08T23:14:46.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbabc0a2970 0x7fbabc0a4e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbad8109570 0x7fbad8199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbad8103e70 0x7fbad819fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 --2- 172.21.15.46:0/2448279385 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbad81034b0 0x7fbad819f950 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 >> 172.21.15.46:0/2448279385 conn(0x7fbad80fefa0 msgr2=0x7fbad81005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 shutdown_connections 2024-03-08T23:14:46.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:46.814+0000 7fbaddc00640 1 -- 172.21.15.46:0/2448279385 wait complete. 2024-03-08T23:14:46.833 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:14:46.834 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 -- 172.21.15.46:0/1733957235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff70108070 msgr2=0x7eff70114580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 --2- 172.21.15.46:0/1733957235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff70108070 0x7eff70114580 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7eff68009900 tx=0x7eff6802bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 -- 172.21.15.46:0/1733957235 shutdown_connections 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 --2- 172.21.15.46:0/1733957235 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff70108070 0x7eff70114580 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 --2- 172.21.15.46:0/1733957235 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff701076b0 0x7eff70107b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 --2- 172.21.15.46:0/1733957235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff7010d6c0 0x7eff7010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 -- 172.21.15.46:0/1733957235 >> 172.21.15.46:0/1733957235 conn(0x7eff70076e10 msgr2=0x7eff70079230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 -- 172.21.15.46:0/1733957235 shutdown_connections 2024-03-08T23:14:47.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.033+0000 7eff77000640 1 -- 172.21.15.46:0/1733957235 wait complete. 2024-03-08T23:14:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.034+0000 7eff77000640 1 Processor -- start 2024-03-08T23:14:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 -- start start 2024-03-08T23:14:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff70108070 0x7eff7019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff7010d6c0 0x7eff701a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70117b20 con 0x7eff70108070 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff701179a0 con 0x7eff7010d6c0 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff77000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff70117ca0 con 0x7eff701076b0 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff75200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff70108070 0x7eff7019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff75c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff75c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39122/0 (socket says 172.21.15.46:39122) 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff76600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff7010d6c0 0x7eff701a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.035+0000 7eff75c00640 1 -- 172.21.15.46:0/3010159524 learned_addr learned my addr 172.21.15.46:0/3010159524 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 -- 172.21.15.46:0/3010159524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff7010d6c0 msgr2=0x7eff701a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff7010d6c0 0x7eff701a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 -- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff70108070 msgr2=0x7eff7019e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff70108070 0x7eff7019e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 -- 172.21.15.46:0/3010159524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff701a2ef0 con 0x7eff701076b0 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff75c00640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7eff6400eb90 tx=0x7eff6400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.036+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64004d20 con 0x7eff701076b0 2024-03-08T23:14:47.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.037+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff6400ee90 con 0x7eff701076b0 2024-03-08T23:14:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.037+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff701a31e0 con 0x7eff701076b0 2024-03-08T23:14:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.037+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff64010830 con 0x7eff701076b0 2024-03-08T23:14:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.037+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff701aaa70 con 0x7eff701076b0 2024-03-08T23:14:47.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.038+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff70068450 con 0x7eff701076b0 2024-03-08T23:14:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.040+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7eff64005020 con 0x7eff701076b0 2024-03-08T23:14:47.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.041+0000 7eff6e000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 0x7eff580a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.042+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7eff640cf790 con 0x7eff701076b0 2024-03-08T23:14:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.042+0000 7eff75200640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 0x7eff580a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.042+0000 7eff75200640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 0x7eff580a4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff7010cc00 tx=0x7eff5c005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.044+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7eff6409c0d0 con 0x7eff701076b0 2024-03-08T23:14:47.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.296+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff7019f250 con 0x7eff701076b0 2024-03-08T23:14:47.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.297+0000 7eff6e000640 1 -- 172.21.15.46:0/3010159524 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7eff6409b9d0 con 0x7eff701076b0 2024-03-08T23:14:47.300 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:14:47.300 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:14:47.300 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:14:47.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 msgr2=0x7eff580a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 0x7eff580a4eb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7eff7010cc00 tx=0x7eff5c005f90 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 msgr2=0x7eff7019df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7eff6400eb90 tx=0x7eff6400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 shutdown_connections 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7eff580a29f0 0x7eff580a4eb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7eff7010d6c0 0x7eff701a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7eff70108070 0x7eff7019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 --2- 172.21.15.46:0/3010159524 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7eff701076b0 0x7eff7019df00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 >> 172.21.15.46:0/3010159524 conn(0x7eff70076e10 msgr2=0x7eff70077330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 shutdown_connections 2024-03-08T23:14:47.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.300+0000 7eff77000640 1 -- 172.21.15.46:0/3010159524 wait complete. 2024-03-08T23:14:47.320 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:47.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:14:47.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 -- 172.21.15.46:0/4120305175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda201013d0 msgr2=0x7fda201017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 --2- 172.21.15.46:0/4120305175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda201013d0 0x7fda201017b0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fda10009ca0 tx=0x7fda1002c0f0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.547 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 -- 172.21.15.46:0/4120305175 shutdown_connections 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 --2- 172.21.15.46:0/4120305175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 0x7fda2010e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 --2- 172.21.15.46:0/4120305175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda20101d80 0x7fda20105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 --2- 172.21.15.46:0/4120305175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda201013d0 0x7fda201017b0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 -- 172.21.15.46:0/4120305175 >> 172.21.15.46:0/4120305175 conn(0x7fda200fcc50 msgr2=0x7fda200ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 -- 172.21.15.46:0/4120305175 shutdown_connections 2024-03-08T23:14:47.548 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.544+0000 7fda25000640 1 -- 172.21.15.46:0/4120305175 wait complete. 2024-03-08T23:14:47.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.546+0000 7fda25000640 1 Processor -- start 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.546+0000 7fda25000640 1 -- start start 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.546+0000 7fda25000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda201013d0 0x7fda200735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.546+0000 7fda25000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 0x7fda20073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.546+0000 7fda25000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda25000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda20111550 con 0x7fda20074070 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda25000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201113d0 con 0x7fda201064a0 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda25000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201116d0 con 0x7fda201013d0 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52660/0 (socket says 172.21.15.46:52660) 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 -- 172.21.15.46:0/2117589008 learned_addr learned my addr 172.21.15.46:0/2117589008 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda16a00640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 0x7fda20073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1f400640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda201013d0 0x7fda200735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 -- 172.21.15.46:0/2117589008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda201013d0 msgr2=0x7fda200735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda201013d0 0x7fda200735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 -- 172.21.15.46:0/2117589008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 msgr2=0x7fda20073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 0x7fda20073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.547+0000 7fda1fe00640 1 -- 172.21.15.46:0/2117589008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda201a6340 con 0x7fda20074070 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda1fe00640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fda1800d950 tx=0x7fda1800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda1cc00640 1 -- 172.21.15.46:0/2117589008 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18017030 con 0x7fda20074070 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda1cc00640 1 -- 172.21.15.46:0/2117589008 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda18004020 con 0x7fda20074070 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda1cc00640 1 -- 172.21.15.46:0/2117589008 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda18003c00 con 0x7fda20074070 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda201a6630 con 0x7fda20074070 2024-03-08T23:14:47.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.548+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda201a6b80 con 0x7fda20074070 2024-03-08T23:14:47.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.551+0000 7fda1cc00640 1 -- 172.21.15.46:0/2117589008 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fda1801a070 con 0x7fda20074070 2024-03-08T23:14:47.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.551+0000 7fda29371740 1 Processor -- start 2024-03-08T23:14:47.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 -- start start 2024-03-08T23:14:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4df4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4e1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d43add0 con 0x561d5d4d6e50 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d3e3c70 con 0x561d5d4dfa30 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.552+0000 7fda29371740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d436e70 con 0x561d5d4d6530 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda1f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4df4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39146/0 (socket says 172.21.15.46:39146) 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 -- 172.21.15.46:0/2245244174 learned_addr learned my addr 172.21.15.46:0/2245244174 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4e1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 -- 172.21.15.46:0/2245244174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 msgr2=0x561d5d4e1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4e1e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 -- 172.21.15.46:0/2245244174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 msgr2=0x561d5d4df4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4df4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 -- 172.21.15.46:0/2245244174 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d5d49d2b0 con 0x561d5d4d6530 2024-03-08T23:14:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.553+0000 7fda16a00640 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda0800bdb0 tx=0x7fda08004120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22f8297598f3c3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda14c00640 1 -- 172.21.15.46:0/2245244174 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda0800bea0 con 0x561d5d4d6530 2024-03-08T23:14:47.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda1cc00640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 0x7fda040a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda14c00640 1 -- 172.21.15.46:0/2245244174 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda08003e80 con 0x561d5d4d6530 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda14c00640 1 -- 172.21.15.46:0/2245244174 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda08012450 con 0x561d5d4d6530 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda1cc00640 1 -- 172.21.15.46:0/2117589008 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fda180ce9d0 con 0x7fda20074070 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.554+0000 7fda1f400640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 0x7fda040a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- 172.21.15.46:0/2245244174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 msgr2=0x561d5d4d6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fda0800bdb0 tx=0x7fda08004120 comp rx=0 tx=0).stop 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- 172.21.15.46:0/2245244174 shutdown_connections 2024-03-08T23:14:47.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4e1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4df4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 --2- 172.21.15.46:0/2245244174 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6530 0x561d5d4d6910 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- 172.21.15.46:0/2245244174 >> 172.21.15.46:0/2245244174 conn(0x561d5d4d0c60 msgr2=0x561d5d4d1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda1f400640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 0x7fda040a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda10000f80 tx=0x7fda10009360 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- 172.21.15.46:0/2245244174 shutdown_connections 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- 172.21.15.46:0/2245244174 wait complete. 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.555+0000 7fda29371740 1 -- start start 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 Processor -- start 2024-03-08T23:14:47.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4dc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4dcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d4e2820 con 0x561d5d4d6530 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d3f70c0 con 0x561d5d4dfa30 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561d5d4e29a0 con 0x561d5d4d6e50 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda29371740 10 client.0 _ll_register_callbacks cb 0x561d5d431ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.556+0000 7fda1f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52682/0 (socket says 172.21.15.46:52682) 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 -- 172.21.15.46:0/2507641165 learned_addr learned my addr 172.21.15.46:0/2507641165 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda16a00640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4dc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1fe00640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4dcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 -- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 msgr2=0x561d5d4dc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4dc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 -- 172.21.15.46:0/2507641165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 msgr2=0x561d5d4dcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4dcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 -- 172.21.15.46:0/2507641165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d5d14f3f0 con 0x561d5d4d6530 2024-03-08T23:14:47.565 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda1f400640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fda10009ca0 tx=0x7fda100095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d24e9b4107154 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.557+0000 7fda0d600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10039040 con 0x561d5d4d6530 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fda10003b70 con 0x561d5d4d6530 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561d5d40f4f0 con 0x561d5d4d6530 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda1005b050 con 0x561d5d4d6530 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda29371740 10 client.16872 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:47.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x561d5d2197d0 con 0x561d5d4d6530 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.558+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561d5d40f4f0 con 0x561d5d4d6530 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fda1004d6c0 con 0x561d5d4d6530 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 10 client.16872 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 10 client.16872 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 20 client.16872 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d4efa60 0x561d5d4f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 4 client.16872 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:47.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561d5d448310 con 0x561d5d4efa60 2024-03-08T23:14:47.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.559+0000 7fda16a00640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d4efa60 0x561d5d4f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.560+0000 7fda16a00640 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d4efa60 0x561d5d4f1e50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=296c1ca2cbe81c42 server_cookie=39735dca24587a44 in_seq=0 out_seq=0 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.560+0000 7fda0d600640 10 client.16872 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda0d600640 1 -- 172.21.15.46:0/2507641165 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561d5d448310 con 0x561d5d4efa60 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda0d600640 10 client.16872 handle_command_reply: tid=0 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 client.16872 shutdown 2024-03-08T23:14:47.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 msgr2=0x561d5d4df360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fda10009ca0 tx=0x7fda100095c0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 shutdown_connections 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d4efa60 0x561d5d4f1e50 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4dcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4dc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 --2- 172.21.15.46:0/2507641165 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4df360 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda0d600640 0 client.16872 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:47.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 >> 172.21.15.46:0/2507641165 conn(0x561d5d4d0c60 msgr2=0x561d5d4d29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 shutdown_connections 2024-03-08T23:14:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 1 -- 172.21.15.46:0/2507641165 wait complete. 2024-03-08T23:14:47.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.561+0000 7fda29371740 20 client.16872 trim_cache size 0 max 16384 2024-03-08T23:14:47.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.574+0000 7fda29371740 1 Processor -- start 2024-03-08T23:14:47.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 -- start start 2024-03-08T23:14:47.578 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4df360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4d9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201b2a40 con 0x561d5d4d6530 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201b05e0 con 0x561d5d4dfa30 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda29371740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201ae6a0 con 0x561d5d4d6e50 2024-03-08T23:14:47.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda1f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda1f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52690/0 (socket says 172.21.15.46:52690) 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda1f400640 1 -- 172.21.15.46:0/349046656 learned_addr learned my addr 172.21.15.46:0/349046656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda1fe00640 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4d9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.575+0000 7fda16a00640 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4df360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 -- 172.21.15.46:0/349046656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 msgr2=0x561d5d4df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4df360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 -- 172.21.15.46:0/349046656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 msgr2=0x561d5d4d9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.583 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4d9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 -- 172.21.15.46:0/349046656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561d5d4e2870 con 0x561d5d4d6530 2024-03-08T23:14:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda1f400640 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fda1002c7e0 tx=0x7fda10005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fade556b49fe870a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda0fe00640 1 -- 172.21.15.46:0/349046656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10064070 con 0x561d5d4d6530 2024-03-08T23:14:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda0fe00640 1 -- 172.21.15.46:0/349046656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda10003e50 con 0x561d5d4d6530 2024-03-08T23:14:47.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.576+0000 7fda0fe00640 1 -- 172.21.15.46:0/349046656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10039070 con 0x561d5d4d6530 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- 172.21.15.46:0/349046656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 msgr2=0x561d5d4dee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fda1002c7e0 tx=0x7fda10005c70 comp rx=0 tx=0).stop 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- 172.21.15.46:0/349046656 shutdown_connections 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4dfa30 0x561d5d4d9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4d6e50 0x561d5d4df360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 --2- 172.21.15.46:0/349046656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6530 0x561d5d4dee20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- 172.21.15.46:0/349046656 >> 172.21.15.46:0/349046656 conn(0x561d5d4d0c60 msgr2=0x561d5d4dafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- 172.21.15.46:0/349046656 shutdown_connections 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- 172.21.15.46:0/349046656 wait complete. 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.577+0000 7fda29371740 1 -- start start 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 Processor -- start 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4e5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 0x561d5d4e5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201aeca0 con 0x561d5d4d6e50 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201aff80 con 0x561d5d4d6530 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda201af2c0 con 0x561d5d4dfa30 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda29371740 10 client.0 _ll_register_callbacks cb 0x7fda08004470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda1fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 0x561d5d4e5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda1f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda1f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34254/0 (socket says 172.21.15.46:34254) 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda1fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 0x561d5d4e5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39174/0 (socket says 172.21.15.46:39174) 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda1f400640 1 -- 172.21.15.46:0/3080236723 learned_addr learned my addr 172.21.15.46:0/3080236723 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.578+0000 7fda16a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 -- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 msgr2=0x561d5d4e5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 0x561d5d4e5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 -- 172.21.15.46:0/3080236723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 msgr2=0x561d5d4e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 -- 172.21.15.46:0/3080236723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561d5d4e5ab0 con 0x561d5d4d6530 2024-03-08T23:14:47.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda1f400640 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda10009bf0 tx=0x7fda10003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87fd05dd49261fea server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda0d600640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda10064070 con 0x561d5d4d6530 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda10003e50 con 0x561d5d4d6530 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.579+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fda100041f0 con 0x561d5d4d6530 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda10071050 con 0x561d5d4d6530 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 10 client.27038 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x561d5d546bd0 con 0x561d5d4d6530 2024-03-08T23:14:47.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda10003e50 con 0x561d5d4d6530 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fda10034700 con 0x561d5d4d6530 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 10 client.27038 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 10 client.27038 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 20 client.27038 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d547180 0x561d5d5495a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 4 client.27038 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.580+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561d5d442660 con 0x561d5d547180 2024-03-08T23:14:47.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda16a00640 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d547180 0x561d5d5495a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda16a00640 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d547180 0x561d5d5495a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=43c078db937c8e0b server_cookie=bdb0b2574059238f in_seq=0 out_seq=0 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda0d600640 10 client.27038 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda0d600640 1 -- 172.21.15.46:0/3080236723 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561d5d442660 con 0x561d5d547180 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda0d600640 10 client.27038 handle_command_reply: tid=0 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.581+0000 7fda29371740 1 client.27038 shutdown 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 msgr2=0x561d5d4e6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fda10009bf0 tx=0x7fda10003ba0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 shutdown_connections 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561d5d547180 0x561d5d5495a0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561d5d4dfa30 0x561d5d4e5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda0d600640 0 client.27038 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561d5d4d6e50 0x561d5d4e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 --2- 172.21.15.46:0/3080236723 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561d5d4d6530 0x561d5d4e6c40 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 >> 172.21.15.46:0/3080236723 conn(0x561d5d4d0c60 msgr2=0x561d5d4cfce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 shutdown_connections 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 1 -- 172.21.15.46:0/3080236723 wait complete. 2024-03-08T23:14:47.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.582+0000 7fda29371740 20 client.27038 trim_cache size 0 max 16384 2024-03-08T23:14:47.597 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 msgr2=0x7fda040a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 0x7fda040a4eb0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fda10000f80 tx=0x7fda10009360 comp rx=0 tx=0).stop 2024-03-08T23:14:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 msgr2=0x7fda201a5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fda1800d950 tx=0x7fda1800de10 comp rx=0 tx=0).stop 2024-03-08T23:14:47.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 shutdown_connections 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fda040a29f0 0x7fda040a4eb0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fda20074070 0x7fda201a5ca0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fda201064a0 0x7fda20073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 --2- 172.21.15.46:0/2117589008 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fda201013d0 0x7fda200735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 >> 172.21.15.46:0/2117589008 conn(0x7fda200fcc50 msgr2=0x7fda200fdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 shutdown_connections 2024-03-08T23:14:47.600 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.583+0000 7fda25000640 1 -- 172.21.15.46:0/2117589008 wait complete. 2024-03-08T23:14:47.603 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:47.603 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:14:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.807+0000 7f6b8fe00640 1 -- 172.21.15.46:0/199092745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90103f90 msgr2=0x7f6b90107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.807+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/199092745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90103f90 0x7f6b90107fd0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f6b78009900 tx=0x7f6b7802bc50 comp rx=0 tx=0).stop 2024-03-08T23:14:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 -- 172.21.15.46:0/199092745 shutdown_connections 2024-03-08T23:14:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/199092745 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 0x7f6b90110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/199092745 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90103f90 0x7f6b90107fd0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/199092745 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b901039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 -- 172.21.15.46:0/199092745 >> 172.21.15.46:0/199092745 conn(0x7f6b900fee20 msgr2=0x7f6b90101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 -- 172.21.15.46:0/199092745 shutdown_connections 2024-03-08T23:14:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.808+0000 7f6b8fe00640 1 -- 172.21.15.46:0/199092745 wait complete. 2024-03-08T23:14:47.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.809+0000 7f6b8fe00640 1 Processor -- start 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.809+0000 7f6b8fe00640 1 -- start start 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 0x7f6b9019fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 0x7f6b90199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90113790 con 0x7f6b90199ab0 2024-03-08T23:14:47.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90113610 con 0x7f6b901035e0 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8fe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b90113910 con 0x7f6b901086b0 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 0x7f6b90199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34268/0 (socket says 172.21.15.46:34268) 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3002909618 learned_addr learned my addr 172.21.15.46:0/3002909618 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b86000640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 0x7f6b9019fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3002909618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 msgr2=0x7f6b9019fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 0x7f6b9019fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.815 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 msgr2=0x7f6b90199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 0x7f6b90199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3002909618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9019a5d0 con 0x7f6b901035e0 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.810+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 0x7f6b90199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000ea60 tx=0x7f6b8000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b87e00640 1 -- 172.21.15.46:0/3002909618 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8000cde0 con 0x7f6b901035e0 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b87e00640 1 -- 172.21.15.46:0/3002909618 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b800035f0 con 0x7f6b901035e0 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9019a8c0 con 0x7f6b901035e0 2024-03-08T23:14:47.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b87e00640 1 -- 172.21.15.46:0/3002909618 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b80010670 con 0x7f6b901035e0 2024-03-08T23:14:47.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.811+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b901a6b60 con 0x7f6b901035e0 2024-03-08T23:14:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.814+0000 7f6b983d9740 1 Processor -- start 2024-03-08T23:14:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.814+0000 7f6b983d9740 1 -- start start 2024-03-08T23:14:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.814+0000 7f6b87e00640 1 -- 172.21.15.46:0/3002909618 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6b800037a0 con 0x7f6b901035e0 2024-03-08T23:14:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b172910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b17de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b0d6dd0 con 0x55606b172e50 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b07fc70 con 0x55606b17ba30 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b983d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b0d2e70 con 0x55606b172530 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52720/0 (socket says 172.21.15.46:52720) 2024-03-08T23:14:47.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b8ea00640 1 -- 172.21.15.46:0/1562796532 learned_addr learned my addr 172.21.15.46:0/1562796532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b86000640 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b17de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.815+0000 7f6b8f400640 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b172910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 -- 172.21.15.46:0/1562796532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 msgr2=0x55606b172910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b172910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 -- 172.21.15.46:0/1562796532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 msgr2=0x55606b17de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b17de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 -- 172.21.15.46:0/1562796532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55606b1392b0 con 0x55606b172e50 2024-03-08T23:14:47.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cc850 tx=0x7f6b800d5a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467b90631a9bb483 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b87e00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 0x7f6b740a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b7f400640 1 -- 172.21.15.46:0/1562796532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b800e2070 con 0x55606b172e50 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b87e00640 1 -- 172.21.15.46:0/3002909618 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f6b800cf510 con 0x7f6b901035e0 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 0x7f6b740a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b7f400640 1 -- 172.21.15.46:0/1562796532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b8009bef0 con 0x55606b172e50 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.816+0000 7f6b7f400640 1 -- 172.21.15.46:0/1562796532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8009b330 con 0x55606b172e50 2024-03-08T23:14:47.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 -- 172.21.15.46:0/1562796532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 msgr2=0x55606b17b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 0x7f6b740a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fd0 tx=0x7f6b88009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.823 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cc850 tx=0x7f6b800d5a30 comp rx=0 tx=0).stop 2024-03-08T23:14:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 -- 172.21.15.46:0/1562796532 shutdown_connections 2024-03-08T23:14:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b17de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b17b4f0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 --2- 172.21.15.46:0/1562796532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b172910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 -- 172.21.15.46:0/1562796532 >> 172.21.15.46:0/1562796532 conn(0x55606b16cc60 msgr2=0x55606b16d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 -- 172.21.15.46:0/1562796532 shutdown_connections 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.817+0000 7f6b983d9740 1 -- 172.21.15.46:0/1562796532 wait complete. 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.818+0000 7f6b983d9740 1 -- start start 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.818+0000 7f6b983d9740 1 Processor -- start 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 0x55606b17b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b178780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b17ba30 0x55606b178d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b17e820 con 0x55606b172e50 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b0930c0 con 0x55606b172530 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606b17e9a0 con 0x55606b17ba30 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b983d9740 10 client.0 _ll_register_callbacks cb 0x55606b0cdec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b86000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b178780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 0x55606b17b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b86000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b178780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52728/0 (socket says 172.21.15.46:52728) 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b86000640 1 -- 172.21.15.46:0/3287985025 learned_addr learned my addr 172.21.15.46:0/3287985025 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b17ba30 0x55606b178d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3287985025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b17ba30 msgr2=0x55606b178d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b17ba30 0x55606b178d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 msgr2=0x55606b178780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b178780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.819+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3287985025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55606adeb3f0 con 0x55606b172530 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 0x55606b17b3f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cc850 tx=0x7f6b80003e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5701334bc39a17d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b73e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8009b150 con 0x55606b172530 2024-03-08T23:14:47.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f6b800860d0 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b8009bef0 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b8009b330 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b983d9740 10 client.27050 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.820+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55606aeb57d0 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b8009bef0 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b800bb040 con 0x55606b172530 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 10 client.27050 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:47.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 10 client.27050 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 20 client.27050 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b18b970 0x55606b18dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 4 client.27050 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55606b0ab4f0 con 0x55606b18b970 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.821+0000 7f6b86000640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b18b970 0x55606b18dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.822+0000 7f6b86000640 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b18b970 0x55606b18dd60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e8fde4cd457ccb62 server_cookie=c8ac2b8c1c1098cc in_seq=0 out_seq=0 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.822+0000 7f6b73e00640 10 client.27050 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:47.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b73e00640 1 -- 172.21.15.46:0/3287985025 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55606b0ab4f0 con 0x55606b18b970 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b73e00640 10 client.27050 handle_command_reply: tid=0 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 client.27050 shutdown 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 msgr2=0x55606b17b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 0x55606b17b3f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6b800cc850 tx=0x7f6b80003e30 comp rx=0 tx=0).stop 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 shutdown_connections 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b18b970 0x55606b18dd60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b17ba30 0x55606b178d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172e50 0x55606b178780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3287985025 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172530 0x55606b17b3f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 >> 172.21.15.46:0/3287985025 conn(0x55606b16cc60 msgr2=0x55606b16e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b73e00640 0 client.27050 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 shutdown_connections 2024-03-08T23:14:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 1 -- 172.21.15.46:0/3287985025 wait complete. 2024-03-08T23:14:47.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.823+0000 7f6b983d9740 20 client.27050 trim_cache size 0 max 16384 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.836+0000 7f6b983d9740 1 Processor -- start 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.836+0000 7f6b983d9740 1 -- start start 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.836+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b1778e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172e50 0x55606b17aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b983d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901b2b90 con 0x55606b17ba30 2024-03-08T23:14:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b983d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901b0730 con 0x55606b172e50 2024-03-08T23:14:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b983d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901ae7f0 con 0x55606b172530 2024-03-08T23:14:47.840 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.841 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52736/0 (socket says 172.21.15.46:52736) 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 -- 172.21.15.46:0/3071596521 learned_addr learned my addr 172.21.15.46:0/3071596521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b86000640 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172e50 0x55606b17aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b1778e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 -- 172.21.15.46:0/3071596521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 msgr2=0x55606b1778e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b1778e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 -- 172.21.15.46:0/3071596521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172e50 msgr2=0x55606b17aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172e50 0x55606b17aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.837+0000 7f6b8f400640 1 -- 172.21.15.46:0/3071596521 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55606b179100 con 0x55606b17ba30 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.838+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6b88003cc0 tx=0x7f6b88017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a38dece2e74115ed server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.838+0000 7f6b7ea00640 1 -- 172.21.15.46:0/3071596521 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88003db0 con 0x55606b17ba30 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.838+0000 7f6b7ea00640 1 -- 172.21.15.46:0/3071596521 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88004040 con 0x55606b17ba30 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.838+0000 7f6b7ea00640 1 -- 172.21.15.46:0/3071596521 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88005020 con 0x55606b17ba30 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- 172.21.15.46:0/3071596521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 msgr2=0x55606b17b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f6b88003cc0 tx=0x7f6b88017000 comp rx=0 tx=0).stop 2024-03-08T23:14:47.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- 172.21.15.46:0/3071596521 shutdown_connections 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b17ba30 0x55606b17b3f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b172e50 0x55606b17aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3071596521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172530 0x55606b1778e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- 172.21.15.46:0/3071596521 >> 172.21.15.46:0/3071596521 conn(0x55606b16cc60 msgr2=0x55606b16d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- 172.21.15.46:0/3071596521 shutdown_connections 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- 172.21.15.46:0/3071596521 wait complete. 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 -- start start 2024-03-08T23:14:47.844 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.839+0000 7f6b983d9740 1 Processor -- start 2024-03-08T23:14:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172e50 0x55606b181430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.848 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901aedf0 con 0x55606b172530 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901b00d0 con 0x55606b17ba30 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b901af410 con 0x55606b172e50 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b983d9740 10 client.0 _ll_register_callbacks cb 0x7f6b800d5060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b8ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52750/0 (socket says 172.21.15.46:52750) 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b8ea00640 1 -- 172.21.15.46:0/3880354939 learned_addr learned my addr 172.21.15.46:0/3880354939 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:47.849 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b8f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34294/0 (socket says 172.21.15.46:34294) 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.840+0000 7f6b86000640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172e50 0x55606b181430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 -- 172.21.15.46:0/3880354939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172e50 msgr2=0x55606b181430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172e50 0x55606b181430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 -- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 msgr2=0x55606b182ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 -- 172.21.15.46:0/3880354939 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55606b1821e0 con 0x55606b17ba30 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:47.850 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b8f400640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b88019040 tx=0x7f6b88004690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d42b14696350f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b73e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88004a00 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b88004040 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f6b88004ba0 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b88004da0 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b983d9740 10 client.27056 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.841+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55606b1e2b20 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88004040 con 0x55606b17ba30 2024-03-08T23:14:47.851 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b88033500 con 0x55606b17ba30 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 10 client.27056 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 10 client.27056 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 20 client.27056 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b1e31b0 0x55606b1e55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 4 client.27056 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55606b0e4310 con 0x55606b1e31b0 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.842+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b1e31b0 0x55606b1e55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:47.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b8ea00640 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b1e31b0 0x55606b1e55d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=973e97dcbfd06391 server_cookie=237e6fb775b7d940 in_seq=0 out_seq=0 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b73e00640 10 client.27056 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b73e00640 1 -- 172.21.15.46:0/3880354939 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55606b0e4310 con 0x55606b1e31b0 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b73e00640 10 client.27056 handle_command_reply: tid=0 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 client.27056 shutdown 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 msgr2=0x55606b1819b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f6b88019040 tx=0x7f6b88004690 comp rx=0 tx=0).stop 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 shutdown_connections 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55606b17ba30 0x55606b1819b0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.853 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55606b1e31b0 0x55606b1e55d0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55606b172e50 0x55606b181430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 --2- 172.21.15.46:0/3880354939 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55606b172530 0x55606b182ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 >> 172.21.15.46:0/3880354939 conn(0x55606b16cc60 msgr2=0x55606b16b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.843+0000 7f6b73e00640 0 client.27056 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 shutdown_connections 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b983d9740 1 -- 172.21.15.46:0/3880354939 wait complete. 2024-03-08T23:14:47.854 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b983d9740 20 client.27056 trim_cache size 0 max 16384 2024-03-08T23:14:47.855 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:47.855 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 msgr2=0x7f6b740a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 0x7f6b740a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009fd0 tx=0x7f6b88009500 comp rx=0 tx=0).stop 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 msgr2=0x7f6b9019f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000ea60 tx=0x7f6b8000ef20 comp rx=0 tx=0).stop 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 shutdown_connections 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6b740a29f0 0x7f6b740a4eb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.844+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6b90199ab0 0x7f6b90199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.845+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6b901086b0 0x7f6b9019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.856 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.845+0000 7f6b8fe00640 1 --2- 172.21.15.46:0/3002909618 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6b901035e0 0x7f6b9019f900 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:47.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.845+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 >> 172.21.15.46:0/3002909618 conn(0x7f6b900fee20 msgr2=0x7f6b90101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:47.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.845+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 shutdown_connections 2024-03-08T23:14:47.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:47.845+0000 7f6b8fe00640 1 -- 172.21.15.46:0/3002909618 wait complete. 2024-03-08T23:14:47.865 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:14:47.865 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 -- 172.21.15.46:0/1273929199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 msgr2=0x7f50681018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 --2- 172.21.15.46:0/1273929199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f50681018d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5058009ca0 tx=0x7f505802c100 comp rx=0 tx=0).stop 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 -- 172.21.15.46:0/1273929199 shutdown_connections 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 --2- 172.21.15.46:0/1273929199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 --2- 172.21.15.46:0/1273929199 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5068101ea0 0x7f5068105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 --2- 172.21.15.46:0/1273929199 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f50681018d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 -- 172.21.15.46:0/1273929199 >> 172.21.15.46:0/1273929199 conn(0x7f50680786f0 msgr2=0x7f5068078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 -- 172.21.15.46:0/1273929199 shutdown_connections 2024-03-08T23:14:48.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.091+0000 7f506d800640 1 -- 172.21.15.46:0/1273929199 wait complete. 2024-03-08T23:14:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.092+0000 7f506d800640 1 Processor -- start 2024-03-08T23:14:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.092+0000 7f506d800640 1 -- start start 2024-03-08T23:14:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506819e600 0x7f506819eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068111680 con 0x7f50681014f0 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068111500 con 0x7f506819e600 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f506d800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5068111800 con 0x7f50681065c0 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f5066a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f5066000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f5066a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52774/0 (socket says 172.21.15.46:52774) 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f5066a00640 1 -- 172.21.15.46:0/3381896656 learned_addr learned my addr 172.21.15.46:0/3381896656 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:48.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.093+0000 7f5067400640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506819e600 0x7f506819eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 -- 172.21.15.46:0/3381896656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 msgr2=0x7f506819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 -- 172.21.15.46:0/3381896656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506819e600 msgr2=0x7f506819eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506819e600 0x7f506819eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 -- 172.21.15.46:0/3381896656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506819f0f0 con 0x7f50681014f0 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066000640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f5066a00640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5058005e30 tx=0x7f5058035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f505f400640 1 -- 172.21.15.46:0/3381896656 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058039040 con 0x7f50681014f0 2024-03-08T23:14:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f505f400640 1 -- 172.21.15.46:0/3381896656 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5058044070 con 0x7f50681014f0 2024-03-08T23:14:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506819f380 con 0x7f50681014f0 2024-03-08T23:14:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.094+0000 7f505f400640 1 -- 172.21.15.46:0/3381896656 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50580043c0 con 0x7f50681014f0 2024-03-08T23:14:48.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.095+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50681a6ba0 con 0x7f50681014f0 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.098+0000 7f505f400640 1 -- 172.21.15.46:0/3381896656 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5058004560 con 0x7f50681014f0 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.098+0000 7f5070431740 1 Processor -- start 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.098+0000 7f5070431740 1 -- start start 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139534310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e139536c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e13948fdd0 con 0x55e13952bc70 2024-03-08T23:14:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e139438c70 con 0x55e13952b350 2024-03-08T23:14:48.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5070431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e13948be70 con 0x55e139534850 2024-03-08T23:14:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5067400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5066a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139534310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5067400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34326/0 (socket says 172.21.15.46:34326) 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5067400640 1 -- 172.21.15.46:0/2012809853 learned_addr learned my addr 172.21.15.46:0/2012809853 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.099+0000 7f5066000640 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e139536c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 -- 172.21.15.46:0/2012809853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 msgr2=0x55e139536c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e139536c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 -- 172.21.15.46:0/2012809853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 msgr2=0x55e139534310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139534310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 -- 172.21.15.46:0/2012809853 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1394f22b0 con 0x55e13952b350 2024-03-08T23:14:48.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f505f400640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 0x7f504c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5067400640 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5060009c20 tx=0x7f5060011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53eea77062ac54b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f505f400640 1 -- 172.21.15.46:0/3381896656 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f50580f14b0 con 0x7f50681014f0 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5066a00640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 0x7f504c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5053e00640 1 -- 172.21.15.46:0/2012809853 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506001e070 con 0x55e13952b350 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5053e00640 1 -- 172.21.15.46:0/2012809853 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5060019b10 con 0x55e13952b350 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.100+0000 7f5053e00640 1 -- 172.21.15.46:0/2012809853 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506000eae0 con 0x55e13952b350 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 -- 172.21.15.46:0/2012809853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 msgr2=0x55e13952b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5060009c20 tx=0x7f5060011810 comp rx=0 tx=0).stop 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5066a00640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 0x7f504c0a4eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50580eeaa0 tx=0x7f50580f7110 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 -- 172.21.15.46:0/2012809853 shutdown_connections 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e139536c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139534310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 --2- 172.21.15.46:0/2012809853 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952b350 0x55e13952b730 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 -- 172.21.15.46:0/2012809853 >> 172.21.15.46:0/2012809853 conn(0x55e139525a80 msgr2=0x55e139525e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 -- 172.21.15.46:0/2012809853 shutdown_connections 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.101+0000 7f5070431740 1 -- 172.21.15.46:0/2012809853 wait complete. 2024-03-08T23:14:48.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 -- start start 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 Processor -- start 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e1395341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 0x55e139531af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e139537640 con 0x55e13952bc70 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e13944c0c0 con 0x55e139534850 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1395377c0 con 0x55e13952b350 2024-03-08T23:14:48.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.102+0000 7f5070431740 10 client.0 _ll_register_callbacks cb 0x55e139486ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52790/0 (socket says 172.21.15.46:52790) 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 -- 172.21.15.46:0/1964792832 learned_addr learned my addr 172.21.15.46:0/1964792832 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5067400640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e1395341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5067400640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 0x55e139531af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 -- 172.21.15.46:0/1964792832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 msgr2=0x55e1395341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e1395341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 -- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 msgr2=0x55e139531af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 0x55e139531af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 -- 172.21.15.46:0/1964792832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1391a43f0 con 0x55e13952bc70 2024-03-08T23:14:48.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5067400640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 0x55e139531af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.103+0000 7f5066000640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5054012d20 tx=0x7f505400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eac7dc8223b5d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5050c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:48.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054012110 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f5054004050 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5060019b10 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5054025050 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5070431740 10 client.16908 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e13926e7d0 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.104+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5060019b10 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5054010430 con 0x55e13952bc70 2024-03-08T23:14:48.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 10 client.16908 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 20 client.16908 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e139544980 0x55e139546d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 4 client.16908 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1394644f0 con 0x55e139544980 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.105+0000 7f5067400640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e139544980 0x55e139546d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.106+0000 7f5067400640 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e139544980 0x55e139546d70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=921a05b60c742a04 server_cookie=a1bd97fe45979c84 in_seq=0 out_seq=0 2024-03-08T23:14:48.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.106+0000 7f5050c00640 10 client.16908 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5050c00640 1 -- 172.21.15.46:0/1964792832 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e1394644f0 con 0x55e139544980 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5050c00640 10 client.16908 handle_command_reply: tid=0 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 client.16908 shutdown 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 msgr2=0x55e139531570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5054012d20 tx=0x7f505400f790 comp rx=0 tx=0).stop 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 shutdown_connections 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e139544980 0x55e139546d70 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e139534850 0x55e139531af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952bc70 0x55e139531570 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5050c00640 0 client.16908 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 --2- 172.21.15.46:0/1964792832 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e1395341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 >> 172.21.15.46:0/1964792832 conn(0x55e139525a80 msgr2=0x55e1395277d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 shutdown_connections 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 1 -- 172.21.15.46:0/1964792832 wait complete. 2024-03-08T23:14:48.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.107+0000 7f5070431740 20 client.16908 trim_cache size 0 max 16384 2024-03-08T23:14:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.120+0000 7f5070431740 1 Processor -- start 2024-03-08T23:14:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 -- start start 2024-03-08T23:14:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 0x55e139546d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e139533ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e1395341e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681b2c40 con 0x55e13952b350 2024-03-08T23:14:48.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681b07e0 con 0x55e13952bc70 2024-03-08T23:14:48.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5070431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681ae8a0 con 0x55e139534850 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5067400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 0x55e139546d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5067400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 0x55e139546d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52804/0 (socket says 172.21.15.46:52804) 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5067400640 1 -- 172.21.15.46:0/3140698393 learned_addr learned my addr 172.21.15.46:0/3140698393 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5066000640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e139533ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.121+0000 7f5067400640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e1395341e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 -- 172.21.15.46:0/3140698393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 msgr2=0x55e1395341e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e1395341e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 -- 172.21.15.46:0/3140698393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 msgr2=0x55e139546d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 0x55e139546d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 -- 172.21.15.46:0/3140698393 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e139531e60 con 0x55e13952bc70 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5067400640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e1395341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5066000640 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e139533ca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5054010800 tx=0x7f5054005970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c75a849972083dd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5053400640 1 -- 172.21.15.46:0/3140698393 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054000e60 con 0x55e13952bc70 2024-03-08T23:14:48.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.122+0000 7f5053400640 1 -- 172.21.15.46:0/3140698393 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f505400fe80 con 0x55e13952bc70 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5053400640 1 -- 172.21.15.46:0/3140698393 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50540035f0 con 0x55e13952bc70 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 -- 172.21.15.46:0/3140698393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 msgr2=0x55e139533ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e139533ca0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f5054010800 tx=0x7f5054005970 comp rx=0 tx=0).stop 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 -- 172.21.15.46:0/3140698393 shutdown_connections 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e139534850 0x55e1395341e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e139533ca0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 --2- 172.21.15.46:0/3140698393 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e13952b350 0x55e139546d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 -- 172.21.15.46:0/3140698393 >> 172.21.15.46:0/3140698393 conn(0x55e139525a80 msgr2=0x55e13954b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 -- 172.21.15.46:0/3140698393 shutdown_connections 2024-03-08T23:14:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.123+0000 7f5070431740 1 -- 172.21.15.46:0/3140698393 wait complete. 2024-03-08T23:14:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 -- start start 2024-03-08T23:14:48.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 Processor -- start 2024-03-08T23:14:48.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e13953a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e13953ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 0x55e13959ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681aeea0 con 0x55e139534850 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681b0180 con 0x55e13952bc70 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681af4c0 con 0x55e13952b350 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.124+0000 7f5070431740 10 client.0 _ll_register_callbacks cb 0x7f5060009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5067400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 0x55e13959ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e13953ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5067400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 0x55e13959ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52818/0 (socket says 172.21.15.46:52818) 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5067400640 1 -- 172.21.15.46:0/3270638510 learned_addr learned my addr 172.21.15.46:0/3270638510 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5067400640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e13953a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 -- 172.21.15.46:0/3270638510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 msgr2=0x55e13953a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e13953a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 -- 172.21.15.46:0/3270638510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 msgr2=0x55e13959ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 0x55e13959ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.134 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5066000640 1 -- 172.21.15.46:0/3270638510 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e13953b550 con 0x55e13952bc70 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.125+0000 7f5067400640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e13953a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5066000640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e13953ac90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5054010800 tx=0x7f5054000e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71040ecf3ec232a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5050c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5054036060 con 0x55e13952bc70 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e13949d310 con 0x55e13952bc70 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f50540050b0 con 0x55e13952bc70 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f505403d050 con 0x55e13952bc70 2024-03-08T23:14:48.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5070431740 10 client.27074 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.126+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e13959c4a0 con 0x55e13952bc70 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e13949d310 con 0x55e13952bc70 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5054003c50 con 0x55e13952bc70 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 10 client.27074 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 10 client.27074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 20 client.27074 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e13959ca00 0x55e13959edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:48.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 4 client.27074 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e139497660 con 0x55e13959ca00 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.127+0000 7f5067400640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e13959ca00 0x55e13959edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5067400640 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e13959ca00 0x55e13959edf0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a524af958280755 server_cookie=a07e2a367414876c in_seq=0 out_seq=0 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5050c00640 10 client.27074 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5050c00640 1 -- 172.21.15.46:0/3270638510 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e139497660 con 0x55e13959ca00 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5050c00640 10 client.27074 handle_command_reply: tid=0 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 client.27074 shutdown 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 msgr2=0x55e13953ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e13953ac90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f5054010800 tx=0x7f5054000e40 comp rx=0 tx=0).stop 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 shutdown_connections 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e13959ca00 0x55e13959edf0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e139534850 0x55e13959ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e13952bc70 0x55e13953ac90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 --2- 172.21.15.46:0/3270638510 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e13952b350 0x55e13953a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5050c00640 0 client.27074 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.128+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 >> 172.21.15.46:0/3270638510 conn(0x55e139525a80 msgr2=0x55e139526d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.138 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 shutdown_connections 2024-03-08T23:14:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f5070431740 1 -- 172.21.15.46:0/3270638510 wait complete. 2024-03-08T23:14:48.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f5070431740 20 client.27074 trim_cache size 0 max 16384 2024-03-08T23:14:48.139 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 msgr2=0x7f504c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 0x7f504c0a4eb0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50580eeaa0 tx=0x7f50580f7110 comp rx=0 tx=0).stop 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 msgr2=0x7f5068199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5058005e30 tx=0x7f5058035d90 comp rx=0 tx=0).stop 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 shutdown_connections 2024-03-08T23:14:48.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f504c0a29f0 0x7f504c0a4eb0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f506819e600 0x7f506819eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50681065c0 0x7f506819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 --2- 172.21.15.46:0/3381896656 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f50681014f0 0x7f5068199d40 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.129+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 >> 172.21.15.46:0/3381896656 conn(0x7f50680786f0 msgr2=0x7f50680ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.130+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 shutdown_connections 2024-03-08T23:14:48.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:48.130+0000 7f506d800640 1 -- 172.21.15.46:0/3381896656 wait complete. 2024-03-08T23:14:48.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:48 smithi046 ceph-mon[31859]: pgmap v215: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:48.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:48 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3010159524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:48.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:48 smithi192 ceph-mon[36700]: pgmap v215: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:48.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:48 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3010159524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:48.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:48 smithi062 ceph-mon[35761]: pgmap v215: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:48.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:48 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3010159524' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:14:49.152 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:14:49.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:14:49.152 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:14:49.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 -- 172.21.15.46:0/2219320097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d010b7c0 msgr2=0x7f93d010dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 --2- 172.21.15.46:0/2219320097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d010b7c0 0x7f93d010dc60 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f93c8009be0 tx=0x7f93c802bf80 comp rx=0 tx=0).stop 2024-03-08T23:14:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 -- 172.21.15.46:0/2219320097 shutdown_connections 2024-03-08T23:14:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 --2- 172.21.15.46:0/2219320097 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d010b7c0 0x7f93d010dc60 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 --2- 172.21.15.46:0/2219320097 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d0102cc0 0x7f93d010b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 --2- 172.21.15.46:0/2219320097 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d01023a0 0x7f93d0102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 -- 172.21.15.46:0/2219320097 >> 172.21.15.46:0/2219320097 conn(0x7f93d00fbba0 msgr2=0x7f93d00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 -- 172.21.15.46:0/2219320097 shutdown_connections 2024-03-08T23:14:49.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.391+0000 7f93cf200640 1 -- 172.21.15.46:0/2219320097 wait complete. 2024-03-08T23:14:49.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 Processor -- start 2024-03-08T23:14:49.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 -- start start 2024-03-08T23:14:49.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d0102cc0 0x7f93d019fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01115d0 con 0x7f93d010b7c0 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0111450 con 0x7f93d01023a0 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.393+0000 7f93cf200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d0111750 con 0x7f93d0102cc0 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93ce800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34372/0 (socket says 172.21.15.46:34372) 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93ce800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52832/0 (socket says 172.21.15.46:52832) 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 -- 172.21.15.46:0/1188108441 learned_addr learned my addr 172.21.15.46:0/1188108441 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cd400640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d0102cc0 0x7f93d019fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 -- 172.21.15.46:0/1188108441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d0102cc0 msgr2=0x7f93d019fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d0102cc0 0x7f93d019fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 -- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 msgr2=0x7f93d0199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93cde00640 1 -- 172.21.15.46:0/1188108441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d019a1f0 con 0x7f93d01023a0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.394+0000 7f93ce800640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93cde00640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f93c000e9f0 tx=0x7f93c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93c6a00640 1 -- 172.21.15.46:0/1188108441 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c000cd80 con 0x7f93d01023a0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93c6a00640 1 -- 172.21.15.46:0/1188108441 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c0010910 con 0x7f93d01023a0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93c6a00640 1 -- 172.21.15.46:0/1188108441 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c0018990 con 0x7f93d01023a0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93d019a4e0 con 0x7f93d01023a0 2024-03-08T23:14:49.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.395+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d01a6780 con 0x7f93d01023a0 2024-03-08T23:14:49.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.398+0000 7f93d7855740 1 Processor -- start 2024-03-08T23:14:49.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.398+0000 7f93d7855740 1 -- start start 2024-03-08T23:14:49.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93c6a00640 1 -- 172.21.15.46:0/1188108441 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f93c0010ac0 con 0x7f93d01023a0 2024-03-08T23:14:49.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92367a0 0x55cef9236b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92370c0 0x55cef923f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef919aa50 con 0x55cef923fca0 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef9143b00 con 0x55cef92367a0 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93d7855740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef9196af0 con 0x55cef92370c0 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93cd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93cd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52836/0 (socket says 172.21.15.46:52836) 2024-03-08T23:14:49.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93cd400640 1 -- 172.21.15.46:0/986342811 learned_addr learned my addr 172.21.15.46:0/986342811 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93cde00640 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92370c0 0x55cef923f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.399+0000 7f93ce800640 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92367a0 0x55cef9236b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 -- 172.21.15.46:0/986342811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92370c0 msgr2=0x55cef923f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92370c0 0x55cef923f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 -- 172.21.15.46:0/986342811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92367a0 msgr2=0x55cef9236b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92367a0 0x55cef9236b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 -- 172.21.15.46:0/986342811 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cef91611e0 con 0x55cef923fca0 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93cd400640 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009bc0 tx=0x7f93bc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6349c7a5f47f5613 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93c6a00640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 0x7f93b40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93bb000640 1 -- 172.21.15.46:0/986342811 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc004230 con 0x55cef923fca0 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93bb000640 1 -- 172.21.15.46:0/986342811 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93bc009e30 con 0x55cef923fca0 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93bb000640 1 -- 172.21.15.46:0/986342811 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93bc0197c0 con 0x55cef923fca0 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.400+0000 7f93c6a00640 1 -- 172.21.15.46:0/1188108441 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f93c00cf520 con 0x7f93d01023a0 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93ce800640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 0x7f93b40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 -- 172.21.15.46:0/986342811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 msgr2=0x55cef9242090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f93bc009bc0 tx=0x7f93bc011810 comp rx=0 tx=0).stop 2024-03-08T23:14:49.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 -- 172.21.15.46:0/986342811 shutdown_connections 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef9242090 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92370c0 0x55cef923f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 --2- 172.21.15.46:0/986342811 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92367a0 0x55cef9236b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 -- 172.21.15.46:0/986342811 >> 172.21.15.46:0/986342811 conn(0x55cef9230ed0 msgr2=0x55cef92312b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93ce800640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 0x7f93b40a4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f93c8006010 tx=0x7f93c80092a0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 -- 172.21.15.46:0/986342811 shutdown_connections 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.401+0000 7f93d7855740 1 -- 172.21.15.46:0/986342811 wait complete. 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.402+0000 7f93d7855740 1 -- start start 2024-03-08T23:14:49.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.402+0000 7f93d7855740 1 Processor -- start 2024-03-08T23:14:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef923d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef923fca0 0x55cef923d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef9242a90 con 0x55cef92367a0 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef9156f60 con 0x55cef92370c0 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cef9242c10 con 0x55cef923fca0 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93d7855740 10 client.0 _ll_register_callbacks cb 0x55cef9142b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93cde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52838/0 (socket says 172.21.15.46:52838) 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93cde00640 1 -- 172.21.15.46:0/2649122516 learned_addr learned my addr 172.21.15.46:0/2649122516 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93cd400640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef923d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93ce800640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef923fca0 0x55cef923d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.403+0000 7f93cde00640 1 -- 172.21.15.46:0/2649122516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef923fca0 msgr2=0x55cef923d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93cde00640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef923fca0 0x55cef923d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93cde00640 1 -- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 msgr2=0x55cef923d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93cde00640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef923d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93cde00640 1 -- 172.21.15.46:0/2649122516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cef8f777e0 con 0x55cef92367a0 2024-03-08T23:14:49.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93cde00640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93c000e9c0 tx=0x7f93c0097470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94be0b7fbed4371e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93af400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c009b170 con 0x55cef92367a0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f93c00ccad0 con 0x55cef92367a0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93bc009e30 con 0x55cef92367a0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93c00cf320 con 0x55cef92367a0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93d7855740 10 client.16926 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.404+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55cef8fd7f70 con 0x55cef92367a0 2024-03-08T23:14:49.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93bc009e30 con 0x55cef92367a0 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f93c0018c50 con 0x55cef92367a0 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 10 client.16926 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 20 client.16926 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef924fee0 0x55cef92522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 4 client.16926 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cef91a7f90 con 0x55cef924fee0 2024-03-08T23:14:49.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.405+0000 7f93cd400640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef924fee0 0x55cef92522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.406+0000 7f93cd400640 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef924fee0 0x55cef92522d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5759ed784484ae88 server_cookie=f80bafe8ae68e4ec in_seq=0 out_seq=0 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.406+0000 7f93af400640 10 client.16926 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93af400640 1 -- 172.21.15.46:0/2649122516 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cef91a7f90 con 0x55cef924fee0 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93af400640 10 client.16926 handle_command_reply: tid=0 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 client.16926 shutdown 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 msgr2=0x55cef923cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f93c000e9c0 tx=0x7f93c0097470 comp rx=0 tx=0).stop 2024-03-08T23:14:49.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 shutdown_connections 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef924fee0 0x55cef92522d0 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef923fca0 0x55cef923d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93af400640 0 client.16926 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef923d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 --2- 172.21.15.46:0/2649122516 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92367a0 0x55cef923cbe0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 >> 172.21.15.46:0/2649122516 conn(0x55cef9230ed0 msgr2=0x55cef9232c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 shutdown_connections 2024-03-08T23:14:49.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 1 -- 172.21.15.46:0/2649122516 wait complete. 2024-03-08T23:14:49.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.407+0000 7f93d7855740 20 client.16926 trim_cache size 0 max 16384 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 Processor -- start 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 -- start start 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef923f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef9239e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01b2ba0 con 0x55cef923fca0 2024-03-08T23:14:49.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01b0740 con 0x55cef92370c0 2024-03-08T23:14:49.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.424+0000 7f93d7855740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01ae800 con 0x55cef92367a0 2024-03-08T23:14:49.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93ce800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef923f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93ce800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52840/0 (socket says 172.21.15.46:52840) 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93ce800640 1 -- 172.21.15.46:0/1460812544 learned_addr learned my addr 172.21.15.46:0/1460812544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cd400640 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef9239e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 -- 172.21.15.46:0/1460812544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 msgr2=0x55cef9239e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef9239e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 -- 172.21.15.46:0/1460812544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 msgr2=0x55cef923a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93cde00640 1 -- 172.21.15.46:0/1460812544 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cef9252640 con 0x55cef92367a0 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.425+0000 7f93ce800640 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93cde00640 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef923f310 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f93c0099ac0 tx=0x7f93c00cf7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51ed8362b8e4fe41 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93ba600640 1 -- 172.21.15.46:0/1460812544 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c009af70 con 0x55cef92367a0 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93ba600640 1 -- 172.21.15.46:0/1460812544 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c009f8b0 con 0x55cef92367a0 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93ba600640 1 -- 172.21.15.46:0/1460812544 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c00afb40 con 0x55cef92367a0 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93d7855740 1 -- 172.21.15.46:0/1460812544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 msgr2=0x55cef923f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.426+0000 7f93d7855740 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef923f310 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f93c0099ac0 tx=0x7f93c00cf7a0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 -- 172.21.15.46:0/1460812544 shutdown_connections 2024-03-08T23:14:49.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef923fca0 0x55cef923a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef92370c0 0x55cef9239e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 --2- 172.21.15.46:0/1460812544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef923f310 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 -- 172.21.15.46:0/1460812544 >> 172.21.15.46:0/1460812544 conn(0x55cef9230ed0 msgr2=0x55cef923bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 -- 172.21.15.46:0/1460812544 shutdown_connections 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 -- 172.21.15.46:0/1460812544 wait complete. 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 -- start start 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.427+0000 7f93d7855740 1 Processor -- start 2024-03-08T23:14:49.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92370c0 0x55cef92a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef923fca0 0x55cef92462d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01aee00 con 0x55cef92370c0 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01b00e0 con 0x55cef923fca0 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d01af420 con 0x55cef92367a0 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93d7855740 10 client.0 _ll_register_callbacks cb 0x7f93bc0097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93cde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93cd400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92370c0 0x55cef92a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93cde00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39328/0 (socket says 172.21.15.46:39328) 2024-03-08T23:14:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93cde00640 1 -- 172.21.15.46:0/3000446213 learned_addr learned my addr 172.21.15.46:0/3000446213 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.428+0000 7f93ce800640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef923fca0 0x55cef92462d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 -- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef923fca0 msgr2=0x55cef92462d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef923fca0 0x55cef92462d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 -- 172.21.15.46:0/3000446213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92370c0 msgr2=0x55cef92a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92370c0 0x55cef92a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 -- 172.21.15.46:0/3000446213 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cef8e6f170 con 0x55cef92367a0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93cde00640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c6a0 tx=0x7f93c0009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35c100cf4d3524eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93af400640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c009b810 con 0x55cef92367a0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f93c0018b30 con 0x55cef92367a0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93c009f8b0 con 0x55cef92367a0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93c00a3f00 con 0x55cef92367a0 2024-03-08T23:14:49.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93d7855740 10 client.27064 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55cef9166050 con 0x55cef92367a0 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.429+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93c009f8b0 con 0x55cef92367a0 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f93c009a740 con 0x55cef92367a0 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 10 client.27064 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 10 client.27064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 20 client.27064 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef92a80b0 0x55cef92aa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 4 client.27064 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cef91a22e0 con 0x55cef92a80b0 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.430+0000 7f93cd400640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef92a80b0 0x55cef92aa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.431+0000 7f93cd400640 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef92a80b0 0x55cef92aa4a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc0851273c1ad3d server_cookie=b0692c80d0312c6 in_seq=0 out_seq=0 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.431+0000 7f93af400640 10 client.27064 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93af400640 1 -- 172.21.15.46:0/3000446213 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cef91a22e0 con 0x55cef92a80b0 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93af400640 10 client.27064 handle_command_reply: tid=0 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 client.27064 shutdown 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 msgr2=0x55cef92a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c6a0 tx=0x7f93c0009700 comp rx=0 tx=0).stop 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 shutdown_connections 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55cef92a80b0 0x55cef92aa4a0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55cef923fca0 0x55cef92462d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55cef92370c0 0x55cef92a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 --2- 172.21.15.46:0/3000446213 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55cef92367a0 0x55cef92a09b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93af400640 0 client.27064 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 >> 172.21.15.46:0/3000446213 conn(0x55cef9230ed0 msgr2=0x55cef922f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 shutdown_connections 2024-03-08T23:14:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 1 -- 172.21.15.46:0/3000446213 wait complete. 2024-03-08T23:14:49.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.432+0000 7f93d7855740 20 client.27064 trim_cache size 0 max 16384 2024-03-08T23:14:49.442 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:49.442 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:14:49.442 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "57130115-d1e8-4864-b45a-0673d21ec239", 2024-03-08T23:14:49.443 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:14:49.443 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 msgr2=0x7f93b40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 0x7f93b40a4eb0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f93c8006010 tx=0x7f93c80092a0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 msgr2=0x7f93d019f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f93c000e9f0 tx=0x7f93c000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 shutdown_connections 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f93b40a29f0 0x7f93b40a4eb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f93d010b7c0 0x7f93d0199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f93d0102cc0 0x7f93d019fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 --2- 172.21.15.46:0/1188108441 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f93d01023a0 0x7f93d019f8c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 >> 172.21.15.46:0/1188108441 conn(0x7f93d00fbba0 msgr2=0x7f93d00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 shutdown_connections 2024-03-08T23:14:49.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.433+0000 7f93cf200640 1 -- 172.21.15.46:0/1188108441 wait complete. 2024-03-08T23:14:49.460 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:14:49.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:14:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 -- 172.21.15.46:0/3107942095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe194101e80 msgr2=0x7fe194105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 --2- 172.21.15.46:0/3107942095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe194101e80 0x7fe194105ec0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe180009900 tx=0x7fe18002bc30 comp rx=0 tx=0).stop 2024-03-08T23:14:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 -- 172.21.15.46:0/3107942095 shutdown_connections 2024-03-08T23:14:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 --2- 172.21.15.46:0/3107942095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe1941065a0 0x7fe19410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 --2- 172.21.15.46:0/3107942095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe194101e80 0x7fe194105ec0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 --2- 172.21.15.46:0/3107942095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 0x7fe1941018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 -- 172.21.15.46:0/3107942095 >> 172.21.15.46:0/3107942095 conn(0x7fe1940fcd50 msgr2=0x7fe1940ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 -- 172.21.15.46:0/3107942095 shutdown_connections 2024-03-08T23:14:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.746+0000 7fe19b200640 1 -- 172.21.15.46:0/3107942095 wait complete. 2024-03-08T23:14:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 Processor -- start 2024-03-08T23:14:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 -- start start 2024-03-08T23:14:49.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 0x7fe194199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19419e6e0 0x7fe19419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe194111660 con 0x7fe1941014d0 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941114e0 con 0x7fe19419e6e0 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.748+0000 7fe19b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941117e0 con 0x7fe1941065a0 2024-03-08T23:14:49.752 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.753 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 0x7fe194199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39352/0 (socket says 172.21.15.46:39352) 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193400640 1 -- 172.21.15.46:0/1603759399 learned_addr learned my addr 172.21.15.46:0/1603759399 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe199200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19419e6e0 0x7fe19419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193400640 1 -- 172.21.15.46:0/1603759399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19419e6e0 msgr2=0x7fe19419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.749+0000 7fe193400640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19419e6e0 0x7fe19419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.750+0000 7fe193400640 1 -- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 msgr2=0x7fe194199e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.750+0000 7fe193400640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 0x7fe194199e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.750+0000 7fe193400640 1 -- 172.21.15.46:0/1603759399 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19419f1d0 con 0x7fe1941065a0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.750+0000 7fe193400640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe18002c240 tx=0x7fe180004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.751+0000 7fe190c00640 1 -- 172.21.15.46:0/1603759399 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180043020 con 0x7fe1941065a0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.751+0000 7fe190c00640 1 -- 172.21.15.46:0/1603759399 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1800054e0 con 0x7fe1941065a0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.751+0000 7fe190c00640 1 -- 172.21.15.46:0/1603759399 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180003bb0 con 0x7fe1941065a0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.751+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19419f460 con 0x7fe1941065a0 2024-03-08T23:14:49.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.751+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1941a6c30 con 0x7fe1941065a0 2024-03-08T23:14:49.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.755+0000 7fe19de08740 1 Processor -- start 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 -- start start 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b573540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b573a80 0x55b24b575e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b52c410 con 0x55b24b56a580 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b4ced50 con 0x55b24b56aea0 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe19de08740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b477b80 con 0x55b24b573a80 2024-03-08T23:14:49.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe199200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe199200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52874/0 (socket says 172.21.15.46:52874) 2024-03-08T23:14:49.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.756+0000 7fe199200640 1 -- 172.21.15.46:0/1511624670 learned_addr learned my addr 172.21.15.46:0/1511624670 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe193e00640 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b573540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe190c00640 1 -- 172.21.15.46:0/1603759399 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe180046020 con 0x7fe1941065a0 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe199200640 1 -- 172.21.15.46:0/1511624670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b573a80 msgr2=0x55b24b575e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe199200640 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b573a80 0x55b24b575e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe199200640 1 -- 172.21.15.46:0/1511624670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 msgr2=0x55b24b573540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe199200640 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b573540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe199200640 1 -- 172.21.15.46:0/1511624670 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b24b2ad7d0 con 0x55b24b56a580 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.757+0000 7fe193e00640 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b573540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.758+0000 7fe199200640 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009f40 tx=0x7fe18c010860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4fcbaf424bcb3f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.758+0000 7fe189600640 1 -- 172.21.15.46:0/1511624670 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c003b70 con 0x55b24b56a580 2024-03-08T23:14:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.758+0000 7fe189600640 1 -- 172.21.15.46:0/1511624670 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe18c003d10 con 0x55b24b56a580 2024-03-08T23:14:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.758+0000 7fe189600640 1 -- 172.21.15.46:0/1511624670 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18c011af0 con 0x55b24b56a580 2024-03-08T23:14:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.759+0000 7fe190c00640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 0x7fe1780a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.759+0000 7fe190c00640 1 -- 172.21.15.46:0/1603759399 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fe1800f1180 con 0x7fe1941065a0 2024-03-08T23:14:49.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.759+0000 7fe193e00640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 0x7fe1780a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.759+0000 7fe19de08740 1 -- 172.21.15.46:0/1511624670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 msgr2=0x55b24b56a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.759+0000 7fe19de08740 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009f40 tx=0x7fe18c010860 comp rx=0 tx=0).stop 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 -- 172.21.15.46:0/1511624670 shutdown_connections 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b573a80 0x55b24b575e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b573540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 --2- 172.21.15.46:0/1511624670 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b56a960 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 -- 172.21.15.46:0/1511624670 >> 172.21.15.46:0/1511624670 conn(0x55b24b564cb0 msgr2=0x55b24b565090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.760+0000 7fe19de08740 1 -- 172.21.15.46:0/1511624670 shutdown_connections 2024-03-08T23:14:49.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.761+0000 7fe193e00640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 0x7fe1780a4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55b24b56a050 tx=0x7fe184005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.761+0000 7fe19de08740 1 -- 172.21.15.46:0/1511624670 wait complete. 2024-03-08T23:14:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.761+0000 7fe19de08740 1 -- start start 2024-03-08T23:14:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.761+0000 7fe19de08740 1 Processor -- start 2024-03-08T23:14:49.767 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 0x55b24b571600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b48aee0 con 0x55b24b56a580 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b4cadf0 con 0x55b24b573a80 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b24b576870 con 0x55b24b56aea0 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.762+0000 7fe19de08740 10 client.0 _ll_register_callbacks cb 0x55b24b4918d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe199200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 0x55b24b571600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52876/0 (socket says 172.21.15.46:52876) 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193e00640 1 -- 172.21.15.46:0/2670135058 learned_addr learned my addr 172.21.15.46:0/2670135058 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 -- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 msgr2=0x55b24b571600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 0x55b24b571600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 -- 172.21.15.46:0/2670135058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 msgr2=0x55b24b570b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193400640 1 -- 172.21.15.46:0/2670135058 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b24b1e33f0 con 0x55b24b56aea0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe199200640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 0x55b24b571600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.763+0000 7fe193e00640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe193400640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5710c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe1800ee5c0 tx=0x7fe180003680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e45cb390f2b2d74b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe17e000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe180005690 con 0x55b24b56aea0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fe1800ee730 con 0x55b24b56aea0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe18c003d10 con 0x55b24b56aea0 2024-03-08T23:14:49.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe1800ee930 con 0x55b24b56aea0 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe19de08740 10 client.27073 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.764+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b24b1a30f0 con 0x55b24b56aea0 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe18c003d10 con 0x55b24b56aea0 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe1800c57e0 con 0x55b24b56aea0 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 10 client.27073 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 10 client.27073 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 20 client.27073 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:49.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b58c450 0x55b24b58e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 4 client.27073 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:14:49.772 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.765+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b24b4dc290 con 0x55b24b58c450 2024-03-08T23:14:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.766+0000 7fe193e00640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b58c450 0x55b24b58e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.766+0000 7fe193e00640 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b58c450 0x55b24b58e840 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5604258afc97ff7 server_cookie=a8d9624f0c279239 in_seq=0 out_seq=0 2024-03-08T23:14:49.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.766+0000 7fe17e000640 10 client.27073 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.767+0000 7fe17e000640 1 -- 172.21.15.46:0/2670135058 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b24b4dc290 con 0x55b24b58c450 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.767+0000 7fe17e000640 10 client.27073 handle_command_reply: tid=0 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.767+0000 7fe19de08740 1 client.27073 shutdown 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 msgr2=0x55b24b5710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5710c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe1800ee5c0 tx=0x7fe180003680 comp rx=0 tx=0).stop 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 shutdown_connections 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b58c450 0x55b24b58e840 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b573a80 0x55b24b571600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5710c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 --2- 172.21.15.46:0/2670135058 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b56a580 0x55b24b570b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 >> 172.21.15.46:0/2670135058 conn(0x55b24b564cb0 msgr2=0x55b24b563f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.768+0000 7fe17e000640 0 client.27073 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.769+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 shutdown_connections 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.769+0000 7fe19de08740 1 -- 172.21.15.46:0/2670135058 wait complete. 2024-03-08T23:14:49.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.769+0000 7fe19de08740 20 client.27073 trim_cache size 0 max 16384 2024-03-08T23:14:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.791+0000 7fe19de08740 1 Processor -- start 2024-03-08T23:14:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.791+0000 7fe19de08740 1 -- start start 2024-03-08T23:14:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b571420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b57ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941ae970 con 0x55b24b573a80 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941ac510 con 0x55b24b56aea0 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe19de08740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941aa5f0 con 0x55b24b56a580 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe193e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe193400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b571420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.796 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe193e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39390/0 (socket says 172.21.15.46:39390) 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe193400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b571420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:34464/0 (socket says 172.21.15.46:34464) 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe193e00640 1 -- 172.21.15.46:0/1154332095 learned_addr learned my addr 172.21.15.46:0/1154332095 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.792+0000 7fe199200640 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b57ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 -- 172.21.15.46:0/1154332095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 msgr2=0x55b24b571420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b571420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 -- 172.21.15.46:0/1154332095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 msgr2=0x55b24b57ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b57ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.797 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 -- 172.21.15.46:0/1154332095 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b24b56acd0 con 0x55b24b56a580 2024-03-08T23:14:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe199200640 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b57ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe193e00640 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe184009460 tx=0x7fe18401a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9bcfacaa52b0c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe188c00640 1 -- 172.21.15.46:0/1154332095 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184003e80 con 0x55b24b56a580 2024-03-08T23:14:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe188c00640 1 -- 172.21.15.46:0/1154332095 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe18402a890 con 0x55b24b56a580 2024-03-08T23:14:49.798 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.793+0000 7fe188c00640 1 -- 172.21.15.46:0/1154332095 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840047c0 con 0x55b24b56a580 2024-03-08T23:14:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 -- 172.21.15.46:0/1154332095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 msgr2=0x55b24b5732b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe184009460 tx=0x7fe18401a000 comp rx=0 tx=0).stop 2024-03-08T23:14:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 -- 172.21.15.46:0/1154332095 shutdown_connections 2024-03-08T23:14:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b57ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56aea0 0x55b24b571420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 --2- 172.21.15.46:0/1154332095 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56a580 0x55b24b5732b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.794+0000 7fe19de08740 1 -- 172.21.15.46:0/1154332095 >> 172.21.15.46:0/1154332095 conn(0x55b24b564cb0 msgr2=0x55b24b56be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.795+0000 7fe19de08740 1 -- 172.21.15.46:0/1154332095 shutdown_connections 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.795+0000 7fe19de08740 1 -- 172.21.15.46:0/1154332095 wait complete. 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.795+0000 7fe19de08740 1 -- start start 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.796+0000 7fe19de08740 1 Processor -- start 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.796+0000 7fe19de08740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 0x55b24b5d6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.796+0000 7fe19de08740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.796+0000 7fe19de08740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b5e18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.796+0000 7fe19de08740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941aabf0 con 0x55b24b573a80 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe19de08740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941abeb0 con 0x55b24b56a580 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe19de08740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1941ab1f0 con 0x55b24b56aea0 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe19de08740 10 client.0 _ll_register_callbacks cb 0x7fe18c0099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 0x55b24b5d6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39394/0 (socket says 172.21.15.46:39394) 2024-03-08T23:14:49.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193400640 1 -- 172.21.15.46:0/2209569068 learned_addr learned my addr 172.21.15.46:0/2209569068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe199200640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b5e18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 -- 172.21.15.46:0/2209569068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 msgr2=0x55b24b5d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 -- 172.21.15.46:0/2209569068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 msgr2=0x55b24b5e18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b5e18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193e00640 1 -- 172.21.15.46:0/2209569068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b24b5e1df0 con 0x55b24b56a580 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe199200640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b5e18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:14:49.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.797+0000 7fe193400640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:14:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.798+0000 7fe193e00640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 0x55b24b5d6830 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe184012cb0 tx=0x7fe184003e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85c581d33c88d4c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:14:49.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.798+0000 7fe17e000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:14:49.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.798+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184037030 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.798+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fe18402f050 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.799+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe18402a890 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.799+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe184044050 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.799+0000 7fe19de08740 10 client.27110 fetch_fsmap learned FSMap version 31 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.799+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b24b5e3030 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.799+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe18402a890 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe184033c30 con 0x55b24b56a580 2024-03-08T23:14:49.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 10 client.27110 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 10 client.27110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 20 client.27110 populate_metadata read hostname 'smithi046' 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b5e36a0 0x55b24b5e5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 4 client.27110 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b24b4d65e0 con 0x55b24b5e36a0 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.800+0000 7fe193400640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b5e36a0 0x55b24b5e5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.801+0000 7fe193400640 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b5e36a0 0x55b24b5e5ac0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab3746d7473d3bf6 server_cookie=d73e70e02bd2efac in_seq=0 out_seq=0 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.801+0000 7fe17e000640 10 client.27110 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.801+0000 7fe17e000640 1 -- 172.21.15.46:0/2209569068 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b24b4d65e0 con 0x55b24b5e36a0 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.801+0000 7fe17e000640 10 client.27110 handle_command_reply: tid=0 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.801+0000 7fe19de08740 1 client.27110 shutdown 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 msgr2=0x55b24b5d6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 0x55b24b5d6830 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe184012cb0 tx=0x7fe184003e80 comp rx=0 tx=0).stop 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 shutdown_connections 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b24b5e36a0 0x55b24b5e5ac0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b24b573a80 0x55b24b5e18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b24b56aea0 0x55b24b5d6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 --2- 172.21.15.46:0/2209569068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b24b56a580 0x55b24b5d6830 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 >> 172.21.15.46:0/2209569068 conn(0x55b24b564cb0 msgr2=0x55b24b56f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe17e000640 0 client.27110 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 shutdown_connections 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.802+0000 7fe19de08740 1 -- 172.21.15.46:0/2209569068 wait complete. 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.803+0000 7fe19de08740 20 client.27110 trim_cache size 0 max 16384 2024-03-08T23:14:49.809 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "57130115-d1e8-4864-b45a-0673d21ec239": { 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "57130115-d1e8-4864-b45a-0673d21ec239", 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:14:49.810 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:14:49.811 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:14:49.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 msgr2=0x7fe1780a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 0x7fe1780a4eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x55b24b56a050 tx=0x7fe184005f90 comp rx=0 tx=0).stop 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 msgr2=0x7fe19419a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe18002c240 tx=0x7fe180004360 comp rx=0 tx=0).stop 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 shutdown_connections 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe1780a29f0 0x7fe1780a4eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe19419e6e0 0x7fe19419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe1941065a0 0x7fe19419a360 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 --2- 172.21.15.46:0/1603759399 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe1941014d0 0x7fe194199e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:14:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 >> 172.21.15.46:0/1603759399 conn(0x7fe1940fcd50 msgr2=0x7fe1940fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:14:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 shutdown_connections 2024-03-08T23:14:49.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:14:49.804+0000 7fe19b200640 1 -- 172.21.15.46:0/1603759399 wait complete. 2024-03-08T23:14:49.831 INFO:tasks.cephfs.filesystem:scrub status for tag:57130115-d1e8-4864-b45a-0673d21ec239 - {'path': '/', 'tag': '57130115-d1e8-4864-b45a-0673d21ec239', 'options': 'recursive,force'} 2024-03-08T23:14:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: pgmap v216: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:14:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:50.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:50.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:14:50.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.564 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:50 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: pgmap v216: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:14:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:14:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:50 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: pgmap v216: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:14:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:14:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:14:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:50.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:50 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:14:52.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:52 smithi192 ceph-mon[36700]: pgmap v217: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:52.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:52 smithi062 ceph-mon[35761]: pgmap v217: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:52.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:52 smithi046 ceph-mon[31859]: pgmap v217: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:14:54.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:54 smithi192 ceph-mon[36700]: pgmap v218: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:54.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:54 smithi062 ceph-mon[35761]: pgmap v218: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:54.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:54 smithi046 ceph-mon[31859]: pgmap v218: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:14:56.565 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:56 smithi192 ceph-mon[36700]: pgmap v219: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:14:56.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:56 smithi062 ceph-mon[35761]: pgmap v219: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:14:56.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:56 smithi046 ceph-mon[31859]: pgmap v219: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:14:58.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:14:58 smithi192 ceph-mon[36700]: pgmap v220: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:14:58.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:14:58 smithi062 ceph-mon[35761]: pgmap v220: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:14:58.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:14:58 smithi046 ceph-mon[31859]: pgmap v220: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:00.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:00 smithi062 ceph-mon[35761]: pgmap v221: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:00.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:00 smithi046 ceph-mon[31859]: pgmap v221: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:00.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:00 smithi192 ceph-mon[36700]: pgmap v221: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:02.615 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:02 smithi192 ceph-mon[36700]: pgmap v222: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:02.616 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:02 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:02.616 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:02 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:02.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:02 smithi062 ceph-mon[35761]: pgmap v222: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:02.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:02 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:02.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:02 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:02.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:02 smithi046 ceph-mon[31859]: pgmap v222: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:02.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:02 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:02.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:02 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:03.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:03 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:03.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:03 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:03.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:03 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:04.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:04 smithi062 ceph-mon[35761]: pgmap v223: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:15:04.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:04 smithi046 ceph-mon[31859]: pgmap v223: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:15:04.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:04 smithi192 ceph-mon[36700]: pgmap v223: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:15:05.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:05 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:15:05.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:05 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:15:05.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:05 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:15:06.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:06 smithi062 ceph-mon[35761]: pgmap v224: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:06.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:06 smithi046 ceph-mon[31859]: pgmap v224: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:06.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:06 smithi192 ceph-mon[36700]: pgmap v224: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:07.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:07 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:15:07.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:07 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:15:07.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:07 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:15:07.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:07 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:15:07.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:07 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:15:07.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:07 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:15:07.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:07 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:15:07.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:07 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:15:07.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:07 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:15:08.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:08 smithi062 ceph-mon[35761]: pgmap v225: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:08.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:08 smithi062 ceph-mon[35761]: osdmap e97: 12 total, 12 up, 12 in 2024-03-08T23:15:08.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:08 smithi046 ceph-mon[31859]: pgmap v225: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:08.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:08 smithi046 ceph-mon[31859]: osdmap e97: 12 total, 12 up, 12 in 2024-03-08T23:15:08.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:08 smithi192 ceph-mon[36700]: pgmap v225: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:15:08.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:08 smithi192 ceph-mon[36700]: osdmap e97: 12 total, 12 up, 12 in 2024-03-08T23:15:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:10 smithi062 ceph-mon[35761]: pgmap v227: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:10 smithi046 ceph-mon[31859]: pgmap v227: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:10.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:10 smithi192 ceph-mon[36700]: pgmap v227: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:11.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:11 smithi046 ceph-mon[31859]: pgmap v228: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:11.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:11 smithi192 ceph-mon[36700]: pgmap v228: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:11.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:11 smithi062 ceph-mon[35761]: pgmap v228: 129 pgs: 129 active+clean; 78 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:15:14.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:13 smithi192 ceph-mon[36700]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:15:14.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:13 smithi062 ceph-mon[35761]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:15:14.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:13 smithi046 ceph-mon[31859]: pgmap v229: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:15:15.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:15 smithi192 ceph-mon[36700]: osdmap e98: 12 total, 12 up, 12 in 2024-03-08T23:15:15.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:15 smithi062 ceph-mon[35761]: osdmap e98: 12 total, 12 up, 12 in 2024-03-08T23:15:15.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:15 smithi046 ceph-mon[31859]: osdmap e98: 12 total, 12 up, 12 in 2024-03-08T23:15:16.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:16 smithi192 ceph-mon[36700]: pgmap v231: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:15:16.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:16 smithi062 ceph-mon[35761]: pgmap v231: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:15:16.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:16 smithi046 ceph-mon[31859]: pgmap v231: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:15:18.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:18 smithi192 ceph-mon[36700]: pgmap v232: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2024-03-08T23:15:18.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:18 smithi062 ceph-mon[35761]: pgmap v232: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2024-03-08T23:15:18.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:18 smithi046 ceph-mon[31859]: pgmap v232: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 223 B/s wr, 6 op/s 2024-03-08T23:15:19.833 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:15:19.833 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:15:20.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.068+0000 7f1ebf200640 1 -- 172.21.15.46:0/2302256620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01035e0 msgr2=0x7f1ec01039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.068+0000 7f1ebf200640 1 --2- 172.21.15.46:0/2302256620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01035e0 0x7f1ec01039c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0009900 tx=0x7f1eb002bc40 comp rx=0 tx=0).stop 2024-03-08T23:15:20.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 -- 172.21.15.46:0/2302256620 shutdown_connections 2024-03-08T23:15:20.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 --2- 172.21.15.46:0/2302256620 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01086b0 0x7f1ec0110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 --2- 172.21.15.46:0/2302256620 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0103f90 0x7f1ec0107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 --2- 172.21.15.46:0/2302256620 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01035e0 0x7f1ec01039c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 -- 172.21.15.46:0/2302256620 >> 172.21.15.46:0/2302256620 conn(0x7f1ec00fee20 msgr2=0x7f1ec0101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 -- 172.21.15.46:0/2302256620 shutdown_connections 2024-03-08T23:15:20.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.069+0000 7f1ebf200640 1 -- 172.21.15.46:0/2302256620 wait complete. 2024-03-08T23:15:20.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 Processor -- start 2024-03-08T23:15:20.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 -- start start 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01086b0 0x7f1ec019fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0199ad0 0x7f1ec0199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0113790 con 0x7f1ec01035e0 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0113610 con 0x7f1ec0199ad0 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.070+0000 7f1ebf200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec0113910 con 0x7f1ec01086b0 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55454/0 (socket says 172.21.15.46:55454) 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 -- 172.21.15.46:0/3381631676 learned_addr learned my addr 172.21.15.46:0/3381631676 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1eb5400640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01086b0 0x7f1ec019fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebe800640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0199ad0 0x7f1ec0199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 -- 172.21.15.46:0/3381631676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01086b0 msgr2=0x7f1ec019fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01086b0 0x7f1ec019fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 -- 172.21.15.46:0/3381631676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0199ad0 msgr2=0x7f1ec0199f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0199ad0 0x7f1ec0199f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.071+0000 7f1ebde00640 1 -- 172.21.15.46:0/3381631676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec019a5f0 con 0x7f1ec01035e0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005ab0 tx=0x7f1eb00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1eb7400640 1 -- 172.21.15.46:0/3381631676 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0041070 con 0x7f1ec01035e0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1eb7400640 1 -- 172.21.15.46:0/3381631676 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1eb002cca0 con 0x7f1ec01035e0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec019a880 con 0x7f1ec01035e0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1eb7400640 1 -- 172.21.15.46:0/3381631676 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb0003d90 con 0x7f1ec01035e0 2024-03-08T23:15:20.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.072+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec01a6a70 con 0x7f1ec01035e0 2024-03-08T23:15:20.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.075+0000 7f1ec7d6d740 1 Processor -- start 2024-03-08T23:15:20.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.075+0000 7f1eb7400640 1 -- 172.21.15.46:0/3381631676 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1eb0044070 con 0x7f1ec01035e0 2024-03-08T23:15:20.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 -- start start 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5ac1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5acd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed5a83410 con 0x564ed5ac20b0 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed5a25d50 con 0x564ed5acac90 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed59ceb80 con 0x564ed5ac1790 2024-03-08T23:15:20.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5acd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55466/0 (socket says 172.21.15.46:55466) 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ebde00640 1 -- 172.21.15.46:0/2133321523 learned_addr learned my addr 172.21.15.46:0/2133321523 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.076+0000 7f1ebe800640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5ac1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 -- 172.21.15.46:0/2133321523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 msgr2=0x564ed5ac1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5ac1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 -- 172.21.15.46:0/2133321523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 msgr2=0x564ed5aca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 -- 172.21.15.46:0/2133321523 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ed58047d0 con 0x564ed5acac90 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1ebde00640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1ebe800640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5ac1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb5400640 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5acd080 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1ec019ac10 tx=0x7f1ea8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=148b5ce9c0d4f05e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eb7400640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 0x7f1ea40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.077+0000 7f1eaf400640 1 -- 172.21.15.46:0/2133321523 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8009d70 con 0x564ed5acac90 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1eb7400640 1 -- 172.21.15.46:0/3381631676 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f1eb00f0aa0 con 0x7f1ec01035e0 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1eaf400640 1 -- 172.21.15.46:0/2133321523 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ea80198f0 con 0x564ed5acac90 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1eaf400640 1 -- 172.21.15.46:0/2133321523 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea80188e0 con 0x564ed5acac90 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ebe800640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 0x7f1ea40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/2133321523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 msgr2=0x564ed5acd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5acd080 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1ec019ac10 tx=0x7f1ea8011810 comp rx=0 tx=0).stop 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ebe800640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 0x7f1ea40a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009fd0 tx=0x7f1eb8009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/2133321523 shutdown_connections 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5acd080 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5aca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/2133321523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5ac1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.078+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/2133321523 >> 172.21.15.46:0/2133321523 conn(0x564ed5abbec0 msgr2=0x564ed5abc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.079+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/2133321523 shutdown_connections 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.079+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/2133321523 wait complete. 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.079+0000 7f1ec7d6d740 1 -- start start 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.079+0000 7f1ec7d6d740 1 Processor -- start 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 0x564ed5ac8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5acac90 0x564ed5ac8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed59e1ee0 con 0x564ed5ac1790 2024-03-08T23:15:20.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed5a21df0 con 0x564ed5ac20b0 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ed5acda80 con 0x564ed5acac90 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ec7d6d740 10 client.0 _ll_register_callbacks cb 0x564ed59e88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ebde00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55472/0 (socket says 172.21.15.46:55472) 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ebde00640 1 -- 172.21.15.46:0/1850894891 learned_addr learned my addr 172.21.15.46:0/1850894891 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 0x564ed5ac8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ebe800640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5acac90 0x564ed5ac8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 -- 172.21.15.46:0/1850894891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5acac90 msgr2=0x564ed5ac8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5acac90 0x564ed5ac8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 -- 172.21.15.46:0/1850894891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 msgr2=0x564ed5ac7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1eb5400640 1 -- 172.21.15.46:0/1850894891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ed573a3f0 con 0x564ed5ac20b0 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.080+0000 7f1ebde00640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1eb5400640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 0x564ed5ac8250 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x564ed5abd050 tx=0x7f1ea80184c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1fede269f5d5285 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ea3e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:20.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8027030 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ea80198f0 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f1ea8016040 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ea8031050 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ec7d6d740 10 client.27122 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.081+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564ed56fa0f0 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ea80198f0 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1ea8004950 con 0x564ed5ac20b0 2024-03-08T23:15:20.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 10 client.27122 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 10 client.27122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 20 client.27122 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5adae80 0x564ed5add270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 4 client.27122 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ed5a33290 con 0x564ed5adae80 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.082+0000 7f1ebde00640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5adae80 0x564ed5add270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.083+0000 7f1ea3e00640 10 client.27122 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:20.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.083+0000 7f1ebde00640 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5adae80 0x564ed5add270 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c45da834b0387b3 server_cookie=8e7a1720a922ae95 in_seq=0 out_seq=0 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ea3e00640 1 -- 172.21.15.46:0/1850894891 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564ed5a33290 con 0x564ed5adae80 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ea3e00640 10 client.27122 handle_command_reply: tid=0 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 client.27122 shutdown 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 msgr2=0x564ed5ac8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 0x564ed5ac8250 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x564ed5abd050 tx=0x7f1ea80184c0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 shutdown_connections 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5adae80 0x564ed5add270 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5acac90 0x564ed5ac8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5ac20b0 0x564ed5ac8250 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/1850894891 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac1790 0x564ed5ac7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 >> 172.21.15.46:0/1850894891 conn(0x564ed5abbec0 msgr2=0x564ed5abdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ea3e00640 0 client.27122 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 shutdown_connections 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/1850894891 wait complete. 2024-03-08T23:15:20.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.084+0000 7f1ec7d6d740 20 client.27122 trim_cache size 0 max 16384 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 Processor -- start 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 -- start start 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5aca440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5ad2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01b2b80 con 0x564ed5ac20b0 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01b0720 con 0x564ed5acac90 2024-03-08T23:15:20.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01ae7e0 con 0x564ed5ac1790 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55474/0 (socket says 172.21.15.46:55474) 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.101+0000 7f1eb5400640 1 -- 172.21.15.46:0/421722718 learned_addr learned my addr 172.21.15.46:0/421722718 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1ebde00640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5aca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1ebe800640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5ad2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 -- 172.21.15.46:0/421722718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 msgr2=0x564ed5aca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5aca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 -- 172.21.15.46:0/421722718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 msgr2=0x564ed5ad2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5ad2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 -- 172.21.15.46:0/421722718 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ed5acd580 con 0x564ed5ac20b0 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1ebde00640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5aca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1ebe800640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5ad2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eb5400640 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000c00 tx=0x7f1ea8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1622efb046ec8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eaea00640 1 -- 172.21.15.46:0/421722718 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea803a070 con 0x564ed5ac20b0 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.102+0000 7f1eaea00640 1 -- 172.21.15.46:0/421722718 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ea8016030 con 0x564ed5ac20b0 2024-03-08T23:15:20.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1eaea00640 1 -- 172.21.15.46:0/421722718 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea8027040 con 0x564ed5ac20b0 2024-03-08T23:15:20.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/421722718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 msgr2=0x564ed5ad1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8000c00 tx=0x7f1ea8009f80 comp rx=0 tx=0).stop 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/421722718 shutdown_connections 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5ad2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5ad1cc0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/421722718 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5aca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/421722718 >> 172.21.15.46:0/421722718 conn(0x564ed5abbec0 msgr2=0x564ed5ac2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/421722718 shutdown_connections 2024-03-08T23:15:20.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.103+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/421722718 wait complete. 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 -- start start 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 Processor -- start 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5b29510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5b25150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01aede0 con 0x564ed5ac20b0 2024-03-08T23:15:20.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01b00c0 con 0x564ed5acac90 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec01af400 con 0x564ed5ac1790 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.104+0000 7f1ec7d6d740 10 client.0 _ll_register_callbacks cb 0x7f1ea8009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55488/0 (socket says 172.21.15.46:55488) 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 -- 172.21.15.46:0/3554733812 learned_addr learned my addr 172.21.15.46:0/3554733812 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5b29510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1ebe800640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5b25150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 -- 172.21.15.46:0/3554733812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 msgr2=0x564ed5b29510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5b29510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 -- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 msgr2=0x564ed5b25150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5b25150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 -- 172.21.15.46:0/3554733812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ed5b25980 con 0x564ed5ac20b0 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1eb5400640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009460 tx=0x7f1ea80092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=958ad745c64d712c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.105+0000 7f1ea3e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:20.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea803a070 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f1ea80183b0 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ea8016030 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ea8027050 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ec7d6d740 10 client.16968 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564ed5b31dd0 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ea8016030 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.106+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1ea80037b0 con 0x564ed5ac20b0 2024-03-08T23:15:20.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 10 client.16968 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 10 client.16968 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 20 client.16968 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5b32440 0x564ed5b34860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 4 client.16968 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564ed5a2d5e0 con 0x564ed5b32440 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5b32440 0x564ed5b34860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ebde00640 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5b32440 0x564ed5b34860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3cbde6b33bf34bfe server_cookie=aa56af8e98e7669b in_seq=0 out_seq=0 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.107+0000 7f1ea3e00640 10 client.16968 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ea3e00640 1 -- 172.21.15.46:0/3554733812 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564ed5a2d5e0 con 0x564ed5b32440 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ea3e00640 10 client.16968 handle_command_reply: tid=0 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 client.16968 shutdown 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 msgr2=0x564ed5b29a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8009460 tx=0x7f1ea80092e0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 shutdown_connections 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564ed5b32440 0x564ed5b34860 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564ed5acac90 0x564ed5b25150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564ed5ac20b0 0x564ed5b29a50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 --2- 172.21.15.46:0/3554733812 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564ed5ac1790 0x564ed5b29510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ea3e00640 0 client.16968 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 >> 172.21.15.46:0/3554733812 conn(0x564ed5abbec0 msgr2=0x564ed5ac2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 shutdown_connections 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 1 -- 172.21.15.46:0/3554733812 wait complete. 2024-03-08T23:15:20.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.108+0000 7f1ec7d6d740 20 client.16968 trim_cache size 0 max 16384 2024-03-08T23:15:20.121 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:20.121 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:15:20.121 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:15:20.121 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 msgr2=0x7f1ea40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 0x7f1ea40a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1eb8009fd0 tx=0x7f1eb8009500 comp rx=0 tx=0).stop 2024-03-08T23:15:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 msgr2=0x7f1ec019f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0005ab0 tx=0x7f1eb00042f0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 shutdown_connections 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1ea40a2970 0x7f1ea40a4e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1ec0199ad0 0x7f1ec0199f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ec01086b0 0x7f1ec019fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 --2- 172.21.15.46:0/3381631676 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1ec01035e0 0x7f1ec019f920 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 >> 172.21.15.46:0/3381631676 conn(0x7f1ec00fee20 msgr2=0x7f1ec0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 shutdown_connections 2024-03-08T23:15:20.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.109+0000 7f1ebf200640 1 -- 172.21.15.46:0/3381631676 wait complete. 2024-03-08T23:15:20.132 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:15:20.132 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:15:20.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 -- 172.21.15.46:0/3097866026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d010b750 msgr2=0x7f54d010db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 --2- 172.21.15.46:0/3097866026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d010b750 0x7f54d010db40 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f54c8009ca0 tx=0x7f54c802c100 comp rx=0 tx=0).stop 2024-03-08T23:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 -- 172.21.15.46:0/3097866026 shutdown_connections 2024-03-08T23:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 --2- 172.21.15.46:0/3097866026 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d010b750 0x7f54d010db40 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 --2- 172.21.15.46:0/3097866026 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d0100100 0x7f54d010b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 --2- 172.21.15.46:0/3097866026 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d00ff7e0 0x7f54d00ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 -- 172.21.15.46:0/3097866026 >> 172.21.15.46:0/3097866026 conn(0x7f54d0074e60 msgr2=0x7f54d0075270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 -- 172.21.15.46:0/3097866026 shutdown_connections 2024-03-08T23:15:20.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.352+0000 7f54cfe00640 1 -- 172.21.15.46:0/3097866026 wait complete. 2024-03-08T23:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.353+0000 7f54cfe00640 1 Processor -- start 2024-03-08T23:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.353+0000 7f54cfe00640 1 -- start start 2024-03-08T23:15:20.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 0x7f54d019dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d010b750 0x7f54d0197790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d01114b0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0111330 con 0x7f54d0100100 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cfe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0111630 con 0x7f54d010b750 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55514/0 (socket says 172.21.15.46:55514) 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cea00640 1 -- 172.21.15.46:0/293612471 learned_addr learned my addr 172.21.15.46:0/293612471 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54ce000640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 0x7f54d019dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.354+0000 7f54cf400640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d010b750 0x7f54d0197790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 -- 172.21.15.46:0/293612471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d010b750 msgr2=0x7f54d0197790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d010b750 0x7f54d0197790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 -- 172.21.15.46:0/293612471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 msgr2=0x7f54d019dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 0x7f54d019dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 -- 172.21.15.46:0/293612471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0198050 con 0x7f54d00ff7e0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54ce000640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 0x7f54d019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cea00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f54c000d950 tx=0x7f54c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c0017070 con 0x7f54d00ff7e0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54c0009c30 con 0x7f54d00ff7e0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d0198340 con 0x7f54d00ff7e0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.355+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c000f6b0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.356+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d01a4600 con 0x7f54d00ff7e0 2024-03-08T23:15:20.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.357+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d0100790 con 0x7f54d00ff7e0 2024-03-08T23:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.359+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f54c00109d0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.360+0000 7f54d4a00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 0x7f54b40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.360+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f54c00ceda0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.360+0000 7f54ce000640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 0x7f54b40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.361+0000 7f54ce000640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 0x7f54b40a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54d0198f90 tx=0x7f54bc005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.365+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f54c0009de0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:20 smithi046 ceph-mon[31859]: pgmap v233: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:20.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:20 smithi192 ceph-mon[36700]: pgmap v233: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:20.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.610+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54d01a47d0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.612+0000 7f54d4a00640 1 -- 172.21.15.46:0/293612471 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f54c009b0f0 con 0x7f54d00ff7e0 2024-03-08T23:15:20.615 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:15:20.615 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:15:20.616 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:15:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.614+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 msgr2=0x7f54b40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.614+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 0x7f54b40a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f54d0198f90 tx=0x7f54bc005f90 comp rx=0 tx=0).stop 2024-03-08T23:15:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.614+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 msgr2=0x7f54d019d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.614+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f54c000d950 tx=0x7f54c000de10 comp rx=0 tx=0).stop 2024-03-08T23:15:20.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 shutdown_connections 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f54b40a2970 0x7f54b40a4e30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f54d010b750 0x7f54d0197790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f54d0100100 0x7f54d019dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 --2- 172.21.15.46:0/293612471 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f54d00ff7e0 0x7f54d019d680 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 >> 172.21.15.46:0/293612471 conn(0x7f54d0074e60 msgr2=0x7f54d00fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 shutdown_connections 2024-03-08T23:15:20.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.615+0000 7f54cfe00640 1 -- 172.21.15.46:0/293612471 wait complete. 2024-03-08T23:15:20.636 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:20.636 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:15:20.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:20 smithi062 ceph-mon[35761]: pgmap v233: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:20.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 -- 172.21.15.46:0/3644546438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f8102380 msgr2=0x7f48f8102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 --2- 172.21.15.46:0/3644546438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f8102380 0x7f48f8102760 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f48e8009ca0 tx=0x7f48e802c190 comp rx=0 tx=0).stop 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 -- 172.21.15.46:0/3644546438 shutdown_connections 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 --2- 172.21.15.46:0/3644546438 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f810b7a0 0x7f48f810dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 --2- 172.21.15.46:0/3644546438 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102ca0 0x7f48f810b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 --2- 172.21.15.46:0/3644546438 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f8102380 0x7f48f8102760 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 -- 172.21.15.46:0/3644546438 >> 172.21.15.46:0/3644546438 conn(0x7f48f80fbba0 msgr2=0x7f48f80fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 -- 172.21.15.46:0/3644546438 shutdown_connections 2024-03-08T23:15:20.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.864+0000 7f48fd000640 1 -- 172.21.15.46:0/3644546438 wait complete. 2024-03-08T23:15:20.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 Processor -- start 2024-03-08T23:15:20.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 -- start start 2024-03-08T23:15:20.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f8102ca0 0x7f48f819dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f810b7a0 0x7f48f8197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8111580 con 0x7f48f8102380 2024-03-08T23:15:20.878 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8111400 con 0x7f48f810b7a0 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.865+0000 7f48fd000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f8111700 con 0x7f48f8102ca0 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55526/0 (socket says 172.21.15.46:55526) 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 -- 172.21.15.46:0/2148374477 learned_addr learned my addr 172.21.15.46:0/2148374477 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f8102ca0 0x7f48f819dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f810b7a0 0x7f48f8197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 -- 172.21.15.46:0/2148374477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f8102ca0 msgr2=0x7f48f819dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.879 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f8102ca0 0x7f48f819dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 -- 172.21.15.46:0/2148374477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f810b7a0 msgr2=0x7f48f8197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f810b7a0 0x7f48f8197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.866+0000 7f48f7400640 1 -- 172.21.15.46:0/2148374477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8198080 con 0x7f48f8102380 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48f7400640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f48e8005e30 tx=0x7f48e8035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48efe00640 1 -- 172.21.15.46:0/2148374477 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e8039040 con 0x7f48f8102380 2024-03-08T23:15:20.880 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f8198310 con 0x7f48f8102380 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48efe00640 1 -- 172.21.15.46:0/2148374477 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48e8044070 con 0x7f48f8102380 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48efe00640 1 -- 172.21.15.46:0/2148374477 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e80328d0 con 0x7f48f8102380 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.867+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f81a45c0 con 0x7f48f8102380 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f48efe00640 1 -- 172.21.15.46:0/2148374477 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f48e8032a70 con 0x7f48f8102380 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 Processor -- start 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 -- start start 2024-03-08T23:15:20.881 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707e50 0x55d38c7104f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c712e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c66bdd0 con 0x55d38c710a30 2024-03-08T23:15:20.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c614c70 con 0x55d38c707e50 2024-03-08T23:15:20.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f49013cc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c667e70 con 0x55d38c707530 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c712e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.871+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53442/0 (socket says 172.21.15.46:53442) 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c712e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55538/0 (socket says 172.21.15.46:55538) 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 -- 172.21.15.46:0/2897009164 learned_addr learned my addr 172.21.15.46:0/2897009164 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.883 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7400640 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707e50 0x55d38c7104f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 -- 172.21.15.46:0/2897009164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707e50 msgr2=0x55d38c7104f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707e50 0x55d38c7104f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 -- 172.21.15.46:0/2897009164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 msgr2=0x55d38c712e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c712e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 -- 172.21.15.46:0/2897009164 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d38c6ce2b0 con 0x55d38c707530 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48efe00640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 0x7f48dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009c20 tx=0x7f48f0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9ce59070d37c0ed server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.884 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.872+0000 7f48efe00640 1 -- 172.21.15.46:0/2148374477 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f48e80f1370 con 0x7f48f8102380 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f48f7400640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 0x7f48dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f48ecc00640 1 -- 172.21.15.46:0/2897009164 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f001e070 con 0x55d38c707530 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f48ecc00640 1 -- 172.21.15.46:0/2897009164 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f0019b10 con 0x55d38c707530 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f48ecc00640 1 -- 172.21.15.46:0/2897009164 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f000eae0 con 0x55d38c707530 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 -- 172.21.15.46:0/2897009164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 msgr2=0x55d38c707910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f48f0009c20 tx=0x7f48f0011810 comp rx=0 tx=0).stop 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f48f7400640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 0x7f48dc0a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48e80ee8c0 tx=0x7f48e8009800 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 -- 172.21.15.46:0/2897009164 shutdown_connections 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c712e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707e50 0x55d38c7104f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 --2- 172.21.15.46:0/2897009164 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707530 0x55d38c707910 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.873+0000 7f49013cc740 1 -- 172.21.15.46:0/2897009164 >> 172.21.15.46:0/2897009164 conn(0x55d38c701c60 msgr2=0x55d38c702040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.874+0000 7f49013cc740 1 -- 172.21.15.46:0/2897009164 shutdown_connections 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.874+0000 7f49013cc740 1 -- 172.21.15.46:0/2897009164 wait complete. 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.874+0000 7f49013cc740 1 -- start start 2024-03-08T23:15:20.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.874+0000 7f49013cc740 1 Processor -- start 2024-03-08T23:15:20.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c710360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c70d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.890 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c70dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c713820 con 0x55d38c710a30 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c6280c0 con 0x55d38c707530 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d38c7139a0 con 0x55d38c707e50 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f49013cc740 10 client.0 _ll_register_callbacks cb 0x55d38c662ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c710360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c70dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c70d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.891 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c70dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55548/0 (socket says 172.21.15.46:55548) 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f7e00640 1 -- 172.21.15.46:0/2695034449 learned_addr learned my addr 172.21.15.46:0/2695034449 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 -- 172.21.15.46:0/2695034449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 msgr2=0x55d38c710360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c710360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 -- 172.21.15.46:0/2695034449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 msgr2=0x55d38c70dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c70dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.875+0000 7f48f6a00640 1 -- 172.21.15.46:0/2695034449 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d38c3803f0 con 0x55d38c707e50 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c70d6f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48f80fe230 tx=0x7f48e0012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ae419845f2740c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f48e5600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e000f8f0 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48f0019b10 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f48e000fa90 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48e000fc90 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f49013cc740 10 client.27115 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.876+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d38c44a7d0 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f0019b10 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f48e001c420 con 0x55d38c707e50 2024-03-08T23:15:20.893 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 10 client.27115 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 10 client.27115 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 20 client.27115 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c720ab0 0x55d38c722ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 4 client.27115 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.877+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d38c6404f0 con 0x55d38c720ab0 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.878+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c720ab0 0x55d38c722ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.878+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c720ab0 0x55d38c722ed0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85b2f41db8a7fdb6 server_cookie=2159e21a22af4f06 in_seq=0 out_seq=0 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.878+0000 7f48e5600640 10 client.27115 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f48e5600640 1 -- 172.21.15.46:0/2695034449 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d38c6404f0 con 0x55d38c720ab0 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f48e5600640 10 client.27115 handle_command_reply: tid=0 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 client.27115 shutdown 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 msgr2=0x55d38c70d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c70d6f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f48f80fe230 tx=0x7f48e0012b60 comp rx=0 tx=0).stop 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 shutdown_connections 2024-03-08T23:15:20.895 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c720ab0 0x55d38c722ed0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c710a30 0x55d38c70dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c70d6f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 --2- 172.21.15.46:0/2695034449 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c710360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f48e5600640 0 client.27115 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 >> 172.21.15.46:0/2695034449 conn(0x55d38c701c60 msgr2=0x55d38c7039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 shutdown_connections 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 1 -- 172.21.15.46:0/2695034449 wait complete. 2024-03-08T23:15:20.896 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.879+0000 7f49013cc740 20 client.27115 trim_cache size 0 max 16384 2024-03-08T23:15:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.892+0000 7f49013cc740 1 Processor -- start 2024-03-08T23:15:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.892+0000 7f49013cc740 1 -- start start 2024-03-08T23:15:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c717fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.900 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c710a30 0x55d38c710360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81b0940 con 0x55d38c707e50 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81ae4e0 con 0x55d38c707530 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f49013cc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81ac5a0 con 0x55d38c710a30 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55560/0 (socket says 172.21.15.46:55560) 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f48f6a00640 1 -- 172.21.15.46:0/1129696557 learned_addr learned my addr 172.21.15.46:0/1129696557 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.901 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f48f7e00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c717fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.893+0000 7f48f7e00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c710a30 0x55d38c710360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 -- 172.21.15.46:0/1129696557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c710a30 msgr2=0x55d38c710360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c710a30 0x55d38c710360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 -- 172.21.15.46:0/1129696557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 msgr2=0x55d38c717fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c717fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 -- 172.21.15.46:0/1129696557 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d38c701a90 con 0x55d38c707e50 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f7e00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c717fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:20.902 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48f6a00640 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f48e0001eb0 tx=0x7f48e00095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b909deff8a4316 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48e7e00640 1 -- 172.21.15.46:0/1129696557 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e0009db0 con 0x55d38c707e50 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48e7e00640 1 -- 172.21.15.46:0/1129696557 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48e00055b0 con 0x55d38c707e50 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.894+0000 7f48e7e00640 1 -- 172.21.15.46:0/1129696557 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e00035f0 con 0x55d38c707e50 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- 172.21.15.46:0/1129696557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 msgr2=0x55d38c70fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f48e0001eb0 tx=0x7f48e00095b0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- 172.21.15.46:0/1129696557 shutdown_connections 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c710a30 0x55d38c710360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.903 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707e50 0x55d38c70fe20 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 --2- 172.21.15.46:0/1129696557 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c707530 0x55d38c717fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- 172.21.15.46:0/1129696557 >> 172.21.15.46:0/1129696557 conn(0x55d38c701c60 msgr2=0x55d38c70bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- 172.21.15.46:0/1129696557 shutdown_connections 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- 172.21.15.46:0/1129696557 wait complete. 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.895+0000 7f49013cc740 1 -- start start 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 Processor -- start 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707530 0x55d38c7752c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.904 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 0x55d38c76f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81acba0 con 0x55d38c707530 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81ade80 con 0x55d38c710a30 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48f81ad1c0 con 0x55d38c707e50 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f49013cc740 10 client.0 _ll_register_callbacks cb 0x7f48f0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f48f7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707530 0x55d38c7752c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f48f6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53462/0 (socket says 172.21.15.46:53462) 2024-03-08T23:15:20.905 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.896+0000 7f48f6a00640 1 -- 172.21.15.46:0/2435583233 learned_addr learned my addr 172.21.15.46:0/2435583233 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 0x55d38c76f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 -- 172.21.15.46:0/2435583233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 msgr2=0x55d38c76f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 0x55d38c76f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 -- 172.21.15.46:0/2435583233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707530 msgr2=0x55d38c7752c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707530 0x55d38c7752c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 -- 172.21.15.46:0/2435583233 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d38c712e20 con 0x55d38c707e50 2024-03-08T23:15:20.906 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 0x55d38c76f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:15:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48f6a00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f48e0000c00 tx=0x7f48e0005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1777dfb0a01e543e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.897+0000 7f48e5600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:15:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48e001c420 con 0x55d38c707e50 2024-03-08T23:15:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f48e001c5c0 con 0x55d38c707e50 2024-03-08T23:15:20.909 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f48e00055b0 con 0x55d38c707e50 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f48e00050d0 con 0x55d38c707e50 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f49013cc740 10 client.27121 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d38c77c900 con 0x55d38c707e50 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48e00055b0 con 0x55d38c707e50 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.898+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f48e0029030 con 0x55d38c707e50 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 10 client.27121 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 10 client.27121 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:15:20.910 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 20 client.27121 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c77cf60 0x55d38c77f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 4 client.27121 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d38c679310 con 0x55d38c77cf60 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c77cf60 0x55d38c77f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f48f7e00640 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c77cf60 0x55d38c77f380 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9d10d53303f92665 server_cookie=8d5a6a17a06f0dd4 in_seq=0 out_seq=0 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.899+0000 7f48e5600640 10 client.27121 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f48e5600640 1 -- 172.21.15.46:0/2435583233 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d38c679310 con 0x55d38c77cf60 2024-03-08T23:15:20.911 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f48e5600640 10 client.27121 handle_command_reply: tid=0 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 client.27121 shutdown 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 msgr2=0x55d38c775800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f48e0000c00 tx=0x7f48e0005f10 comp rx=0 tx=0).stop 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 shutdown_connections 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55d38c77cf60 0x55d38c77f380 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.912 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d38c710a30 0x55d38c76f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d38c707e50 0x55d38c775800 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f48e5600640 0 client.27121 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 --2- 172.21.15.46:0/2435583233 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d38c707530 0x55d38c7752c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 >> 172.21.15.46:0/2435583233 conn(0x55d38c701c60 msgr2=0x55d38c70c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 shutdown_connections 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 1 -- 172.21.15.46:0/2435583233 wait complete. 2024-03-08T23:15:20.913 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.900+0000 7f49013cc740 20 client.27121 trim_cache size 0 max 16384 2024-03-08T23:15:20.914 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 msgr2=0x7f48dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 0x7f48dc0a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f48e80ee8c0 tx=0x7f48e8009800 comp rx=0 tx=0).stop 2024-03-08T23:15:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 msgr2=0x7f48f819d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f48e8005e30 tx=0x7f48e8035d90 comp rx=0 tx=0).stop 2024-03-08T23:15:20.915 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 shutdown_connections 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f48dc0a2970 0x7f48dc0a4e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f48f810b7a0 0x7f48f8197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48f8102ca0 0x7f48f819dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 --2- 172.21.15.46:0/2148374477 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f48f8102380 0x7f48f819d700 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 >> 172.21.15.46:0/2148374477 conn(0x7f48f80fbba0 msgr2=0x7f48f80fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 shutdown_connections 2024-03-08T23:15:20.916 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:20.901+0000 7f48fd000640 1 -- 172.21.15.46:0/2148374477 wait complete. 2024-03-08T23:15:20.921 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:20.922 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:15:21.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.131+0000 7fd37a000640 1 -- 172.21.15.46:0/2029138013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 msgr2=0x7fd374102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.131+0000 7fd37a000640 1 --2- 172.21.15.46:0/2029138013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd374102700 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ca0 tx=0x7fd36402c190 comp rx=0 tx=0).stop 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 -- 172.21.15.46:0/2029138013 shutdown_connections 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 --2- 172.21.15.46:0/2029138013 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd37410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 --2- 172.21.15.46:0/2029138013 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37410b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 --2- 172.21.15.46:0/2029138013 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd374102700 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 -- 172.21.15.46:0/2029138013 >> 172.21.15.46:0/2029138013 conn(0x7fd3740fbb40 msgr2=0x7fd3740fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 -- 172.21.15.46:0/2029138013 shutdown_connections 2024-03-08T23:15:21.137 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.132+0000 7fd37a000640 1 -- 172.21.15.46:0/2029138013 wait complete. 2024-03-08T23:15:21.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 Processor -- start 2024-03-08T23:15:21.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 -- start start 2024-03-08T23:15:21.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd37419d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37419dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd374111520 con 0x7fd37410b740 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741113a0 con 0x7fd374102c40 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd37a000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741116a0 con 0x7fd374102320 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd379600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd379600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55580/0 (socket says 172.21.15.46:55580) 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd379600640 1 -- 172.21.15.46:0/2547679992 learned_addr learned my addr 172.21.15.46:0/2547679992 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.142 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd373e00640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37419dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.133+0000 7fd378c00640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd37419d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 -- 172.21.15.46:0/2547679992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 msgr2=0x7fd37419d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd37419d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 -- 172.21.15.46:0/2547679992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 msgr2=0x7fd37419dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37419dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 -- 172.21.15.46:0/2547679992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3741980e0 con 0x7fd37410b740 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd373e00640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37419dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:21.143 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd379600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00e970 tx=0x7fd36c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd371600640 1 -- 172.21.15.46:0/2547679992 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c00ccc0 con 0x7fd37410b740 2024-03-08T23:15:21.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3741983d0 con 0x7fd37410b740 2024-03-08T23:15:21.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd371600640 1 -- 172.21.15.46:0/2547679992 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd36c00ce60 con 0x7fd37410b740 2024-03-08T23:15:21.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.134+0000 7fd371600640 1 -- 172.21.15.46:0/2547679992 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36c010490 con 0x7fd37410b740 2024-03-08T23:15:21.144 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.135+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3741a4620 con 0x7fd37410b740 2024-03-08T23:15:21.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.138+0000 7fd371600640 1 -- 172.21.15.46:0/2547679992 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd36c005020 con 0x7fd37410b740 2024-03-08T23:15:21.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.138+0000 7fd37e3ca740 1 Processor -- start 2024-03-08T23:15:21.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.138+0000 7fd37e3ca740 1 -- start start 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.138+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b6230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56446880add0 con 0x5644688a5f10 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644687b3c70 con 0x5644688b3e40 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564468806e70 con 0x5644688a68c0 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd378c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55582/0 (socket says 172.21.15.46:55582) 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 -- 172.21.15.46:0/1517790446 learned_addr learned my addr 172.21.15.46:0/1517790446 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd378c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53480/0 (socket says 172.21.15.46:53480) 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd371600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 0x7fd3580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 -- 172.21.15.46:0/1517790446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 msgr2=0x5644688b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 -- 172.21.15.46:0/1517790446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 msgr2=0x5644688b6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.139+0000 7fd373e00640 1 -- 172.21.15.46:0/1517790446 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56446886d2b0 con 0x5644688a5f10 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd371600640 1 -- 172.21.15.46:0/2547679992 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd36c0cf660 con 0x7fd37410b740 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd379600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 0x7fd3580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd373e00640 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd360009660 tx=0x7fd360009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fca9765cc091a330 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd369c00640 1 -- 172.21.15.46:0/1517790446 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360004050 con 0x5644688a5f10 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd369c00640 1 -- 172.21.15.46:0/1517790446 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd36000fda0 con 0x5644688a5f10 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.140+0000 7fd369c00640 1 -- 172.21.15.46:0/1517790446 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360003bc0 con 0x5644688a5f10 2024-03-08T23:15:21.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1517790446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 msgr2=0x5644688a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd379600640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 0x7fd3580a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd36c09b5a0 tx=0x7fd36c009720 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd360009660 tx=0x7fd360009bd0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1517790446 shutdown_connections 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1517790446 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688a62f0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1517790446 >> 172.21.15.46:0/1517790446 conn(0x5644688a0c60 msgr2=0x5644688a1040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1517790446 shutdown_connections 2024-03-08T23:15:21.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.141+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1517790446 wait complete. 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 -- start start 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 Processor -- start 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 0x5644688b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644688b6c00 con 0x5644688a68c0 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644687c70c0 con 0x5644688b3e40 2024-03-08T23:15:21.155 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644688b6d80 con 0x5644688a5f10 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd37e3ca740 10 client.0 _ll_register_callbacks cb 0x564468801ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.142+0000 7fd378c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55594/0 (socket says 172.21.15.46:55594) 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 -- 172.21.15.46:0/1423168851 learned_addr learned my addr 172.21.15.46:0/1423168851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd373e00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 -- 172.21.15.46:0/1423168851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 msgr2=0x5644688b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd373e00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 0x5644688b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 0x5644688b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.156 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 -- 172.21.15.46:0/1423168851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 msgr2=0x5644688b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 -- 172.21.15.46:0/1423168851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56446851f3f0 con 0x5644688a68c0 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd373e00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 0x5644688b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd378c00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd364009430 tx=0x7fd364003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f75b940b5d5ecd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.143+0000 7fd35e000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364044070 con 0x5644688a68c0 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd364004040 con 0x5644688a68c0 2024-03-08T23:15:21.157 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd36000fda0 con 0x5644688a68c0 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd364051050 con 0x5644688a68c0 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 10 client.17004 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5644685e97d0 con 0x5644688a68c0 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd36000fda0 con 0x5644688a68c0 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd364032c20 con 0x5644688a68c0 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 10 client.17004 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 10 client.17004 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:15:21.158 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 20 client.17004 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5644688c3e80 0x5644688c6270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 4 client.17004 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.144+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644687df4f0 con 0x5644688c3e80 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.145+0000 7fd373e00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5644688c3e80 0x5644688c6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.145+0000 7fd373e00640 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5644688c3e80 0x5644688c6270 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4def62ab06b15cf8 server_cookie=30f1ebdf71a2429c in_seq=0 out_seq=0 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.145+0000 7fd35e000640 10 client.17004 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd35e000640 1 -- 172.21.15.46:0/1423168851 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5644687df4f0 con 0x5644688c3e80 2024-03-08T23:15:21.159 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd35e000640 10 client.17004 handle_command_reply: tid=0 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 client.17004 shutdown 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 msgr2=0x5644688b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd364009430 tx=0x7fd364003e50 comp rx=0 tx=0).stop 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 shutdown_connections 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688b3e40 0x5644688b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a68c0 0x5644688b0af0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5644688c3e80 0x5644688c6270 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1423168851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a5f10 0x5644688b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.160 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 >> 172.21.15.46:0/1423168851 conn(0x5644688a0c60 msgr2=0x5644688a29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.146+0000 7fd35e000640 0 client.17004 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:21.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.147+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 shutdown_connections 2024-03-08T23:15:21.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.147+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1423168851 wait complete. 2024-03-08T23:15:21.161 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.147+0000 7fd37e3ca740 20 client.17004 trim_cache size 0 max 16384 2024-03-08T23:15:21.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 Processor -- start 2024-03-08T23:15:21.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 -- start start 2024-03-08T23:15:21.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a68c0 0x5644688bb030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688b3e40 0x5644688bb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741ae670 con 0x5644688a5f10 2024-03-08T23:15:21.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741ac210 con 0x5644688a68c0 2024-03-08T23:15:21.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.160+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741aa2d0 con 0x5644688b3e40 2024-03-08T23:15:21.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55596/0 (socket says 172.21.15.46:55596) 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 -- 172.21.15.46:0/1426269108 learned_addr learned my addr 172.21.15.46:0/1426269108 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd378c00640 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a68c0 0x5644688bb030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 -- 172.21.15.46:0/1426269108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688b3e40 msgr2=0x5644688bb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688b3e40 0x5644688bb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 -- 172.21.15.46:0/1426269108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a68c0 msgr2=0x5644688bb030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a68c0 0x5644688bb030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 -- 172.21.15.46:0/1426269108 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644688a6660 con 0x5644688a5f10 2024-03-08T23:15:21.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.161+0000 7fd373e00640 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd360003900 tx=0x7fd36000ff50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a5474dd543de67 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd369200640 1 -- 172.21.15.46:0/1426269108 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3600134c0 con 0x5644688a5f10 2024-03-08T23:15:21.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd369200640 1 -- 172.21.15.46:0/1426269108 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd360013170 con 0x5644688a5f10 2024-03-08T23:15:21.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd369200640 1 -- 172.21.15.46:0/1426269108 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd360003930 con 0x5644688a5f10 2024-03-08T23:15:21.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1426269108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 msgr2=0x5644688b3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fd360003900 tx=0x7fd36000ff50 comp rx=0 tx=0).stop 2024-03-08T23:15:21.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1426269108 shutdown_connections 2024-03-08T23:15:21.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688b3e40 0x5644688bb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a68c0 0x5644688bb030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/1426269108 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688a5f10 0x5644688b3220 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1426269108 >> 172.21.15.46:0/1426269108 conn(0x5644688a0c60 msgr2=0x5644688b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1426269108 shutdown_connections 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.162+0000 7fd37e3ca740 1 -- 172.21.15.46:0/1426269108 wait complete. 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.163+0000 7fd37e3ca740 1 -- start start 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.163+0000 7fd37e3ca740 1 Processor -- start 2024-03-08T23:15:21.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.163+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 0x5644688b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.163+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd37e3ca740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688b3e40 0x5644688ba4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741aa8d0 con 0x5644688b3e40 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741abbb0 con 0x5644688a5f10 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd37e3ca740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3741aaef0 con 0x5644688a68c0 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd37e3ca740 10 client.0 _ll_register_callbacks cb 0x7fd360009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53498/0 (socket says 172.21.15.46:53498) 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 -- 172.21.15.46:0/662902722 learned_addr learned my addr 172.21.15.46:0/662902722 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd373e00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688b3e40 0x5644688ba4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd373e00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 0x5644688b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 -- 172.21.15.46:0/662902722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 msgr2=0x5644688b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 0x5644688b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 -- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688b3e40 msgr2=0x5644688ba4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688b3e40 0x5644688ba4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd378c00640 1 -- 172.21.15.46:0/662902722 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56446891a350 con 0x5644688a68c0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.164+0000 7fd373e00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 0x5644688b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd378c00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ca0 tx=0x7fd364005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdd29fb4a69f9fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd364044070 con 0x5644688a68c0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd360013170 con 0x5644688a68c0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd364034c30 con 0x5644688a68c0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd364039040 con 0x5644688a68c0 2024-03-08T23:15:21.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd37e3ca740 10 client.27133 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56446891b650 con 0x5644688a68c0 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd360013170 con 0x5644688a68c0 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.165+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd364004050 con 0x5644688a68c0 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 10 client.27133 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 10 client.27133 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 20 client.27133 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56446891bd40 0x56446891e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 4 client.27133 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:21.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564468818310 con 0x56446891bd40 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd373e00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56446891bd40 0x56446891e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd373e00640 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56446891bd40 0x56446891e160 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ee6cdea4a16a4fc0 server_cookie=ba02f426c7cd5444 in_seq=0 out_seq=0 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.166+0000 7fd35e000640 10 client.27133 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd35e000640 1 -- 172.21.15.46:0/662902722 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564468818310 con 0x56446891bd40 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd35e000640 10 client.27133 handle_command_reply: tid=0 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 client.27133 shutdown 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 msgr2=0x5644688b9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fd364009ca0 tx=0x7fd364005bd0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 shutdown_connections 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5644688b3e40 0x5644688ba4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x56446891bd40 0x56446891e160 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5644688a68c0 0x5644688b9f60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 --2- 172.21.15.46:0/662902722 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5644688a5f10 0x5644688b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd35e000640 0 client.27133 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 >> 172.21.15.46:0/662902722 conn(0x5644688a0c60 msgr2=0x5644688ab8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 shutdown_connections 2024-03-08T23:15:21.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 1 -- 172.21.15.46:0/662902722 wait complete. 2024-03-08T23:15:21.183 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.167+0000 7fd37e3ca740 20 client.27133 trim_cache size 0 max 16384 2024-03-08T23:15:21.183 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 msgr2=0x7fd3580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 0x7fd3580a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd36c09b5a0 tx=0x7fd36c009720 comp rx=0 tx=0).stop 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 msgr2=0x7fd374197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00e970 tx=0x7fd36c00ed70 comp rx=0 tx=0).stop 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 shutdown_connections 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd3580a2970 0x7fd3580a4e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd37410b740 0x7fd374197960 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd374102c40 0x7fd37419dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 --2- 172.21.15.46:0/2547679992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd374102320 0x7fd37419d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 >> 172.21.15.46:0/2547679992 conn(0x7fd3740fbb40 msgr2=0x7fd3740fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 shutdown_connections 2024-03-08T23:15:21.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.168+0000 7fd37a000640 1 -- 172.21.15.46:0/2547679992 wait complete. 2024-03-08T23:15:21.190 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:21.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:15:21.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 -- 172.21.15.46:0/2525686205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 msgr2=0x7f951410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 --2- 172.21.15.46:0/2525686205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951410d460 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9500009860 tx=0x7f950002bc10 comp rx=0 tx=0).stop 2024-03-08T23:15:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 -- 172.21.15.46:0/2525686205 shutdown_connections 2024-03-08T23:15:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 --2- 172.21.15.46:0/2525686205 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f951410d9a0 0x7f951410fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 --2- 172.21.15.46:0/2525686205 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951410d460 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 --2- 172.21.15.46:0/2525686205 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f9514073720 0x7f9514073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 -- 172.21.15.46:0/2525686205 >> 172.21.15.46:0/2525686205 conn(0x7f95140fcc50 msgr2=0x7f95140ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 -- 172.21.15.46:0/2525686205 shutdown_connections 2024-03-08T23:15:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.412+0000 7f9513200640 1 -- 172.21.15.46:0/2525686205 wait complete. 2024-03-08T23:15:21.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.413+0000 7f9513200640 1 Processor -- start 2024-03-08T23:15:21.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.413+0000 7f9513200640 1 -- start start 2024-03-08T23:15:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.413+0000 7f9513200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 0x7f951419f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.413+0000 7f9513200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9513200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951410d9a0 0x7f9514199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9513200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9514113710 con 0x7f951410d9a0 2024-03-08T23:15:21.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9513200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9514113590 con 0x7f9514073720 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9513200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9514113890 con 0x7f9514074040 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:53522/0 (socket says 172.21.15.46:53522) 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 -- 172.21.15.46:0/3061795815 learned_addr learned my addr 172.21.15.46:0/3061795815 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9512800640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951410d9a0 0x7f9514199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511e00640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 0x7f951419f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 -- 172.21.15.46:0/3061795815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 msgr2=0x7f951419f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 0x7f951419f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 -- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951410d9a0 msgr2=0x7f9514199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951410d9a0 0x7f9514199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 -- 172.21.15.46:0/3061795815 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951419a210 con 0x7f9514074040 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511e00640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 0x7f951419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.414+0000 7f9511400640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9500037040 tx=0x7f95000344c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.415+0000 7f950aa00640 1 -- 172.21.15.46:0/3061795815 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500041020 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.415+0000 7f950aa00640 1 -- 172.21.15.46:0/3061795815 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950002ca90 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.415+0000 7f950aa00640 1 -- 172.21.15.46:0/3061795815 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500039040 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.415+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951419a4a0 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.415+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95141a67a0 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.418+0000 7f950aa00640 1 -- 172.21.15.46:0/3061795815 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9500044020 con 0x7f9514074040 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.418+0000 7f951bd6d740 1 Processor -- start 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 -- start start 2024-03-08T23:15:21.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4ef910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb453dd0 con 0x55d6bb4efe50 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb3fcc70 con 0x55d6bb4ef530 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f951bd6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb44fe70 con 0x55d6bb4f8a30 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f9511e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f9512800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f9511e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55640/0 (socket says 172.21.15.46:55640) 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f9511e00640 1 -- 172.21.15.46:0/3305020360 learned_addr learned my addr 172.21.15.46:0/3305020360 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.419+0000 7f9511400640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 -- 172.21.15.46:0/3305020360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 msgr2=0x55d6bb4fae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4fae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 -- 172.21.15.46:0/3305020360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 msgr2=0x55d6bb4f84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 -- 172.21.15.46:0/3305020360 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6bb4b62b0 con 0x55d6bb4ef530 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9511e00640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:21.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9511400640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4fae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f950aa00640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 0x7f94f80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9512800640 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4ef910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f950c009f00 tx=0x7f950c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fdc558c95785481 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f950aa00640 1 -- 172.21.15.46:0/3061795815 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f95000f08a0 con 0x7f9514074040 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f94ff400640 1 -- 172.21.15.46:0/3305020360 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c004d20 con 0x55d6bb4ef530 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f9511e00640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 0x7f94f80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f94ff400640 1 -- 172.21.15.46:0/3305020360 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950c004050 con 0x55d6bb4ef530 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.420+0000 7f94ff400640 1 -- 172.21.15.46:0/3305020360 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c005280 con 0x55d6bb4ef530 2024-03-08T23:15:21.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 -- 172.21.15.46:0/3305020360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 msgr2=0x55d6bb4ef910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4ef910 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f950c009f00 tx=0x7f950c011810 comp rx=0 tx=0).stop 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f9511e00640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 0x7f94f80a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55d6bb4ef000 tx=0x7f9504009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 -- 172.21.15.46:0/3305020360 shutdown_connections 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4fae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 --2- 172.21.15.46:0/3305020360 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4ef910 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 -- 172.21.15.46:0/3305020360 >> 172.21.15.46:0/3305020360 conn(0x55d6bb4e9c60 msgr2=0x55d6bb4ea040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 -- 172.21.15.46:0/3305020360 shutdown_connections 2024-03-08T23:15:21.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.421+0000 7f951bd6d740 1 -- 172.21.15.46:0/3305020360 wait complete. 2024-03-08T23:15:21.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.422+0000 7f951bd6d740 1 -- start start 2024-03-08T23:15:21.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.422+0000 7f951bd6d740 1 Processor -- start 2024-03-08T23:15:21.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f5750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb4fb820 con 0x55d6bb4f8a30 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb4100c0 con 0x55d6bb4ef530 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d6bb4fb9a0 con 0x55d6bb4efe50 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f951bd6d740 10 client.0 _ll_register_callbacks cb 0x55d6bb44aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55656/0 (socket says 172.21.15.46:55656) 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 -- 172.21.15.46:0/126277030 learned_addr learned my addr 172.21.15.46:0/126277030 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9511e00640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9511400640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 -- 172.21.15.46:0/126277030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 msgr2=0x55d6bb4f5750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f5750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 -- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 msgr2=0x55d6bb4f8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.423+0000 7f9512800640 1 -- 172.21.15.46:0/126277030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6bb1683f0 con 0x55d6bb4f8a30 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f9512800640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f950c01b040 tx=0x7f950c0044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac91860859cc74c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f94f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950c028020 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f950c004c50 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f950c004050 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950c033050 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f951bd6d740 10 client.17022 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.424+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d6bb2327d0 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950c004050 con 0x55d6bb4f8a30 2024-03-08T23:15:21.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f950c023470 con 0x55d6bb4f8a30 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 10 client.17022 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 20 client.17022 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb508b30 0x55d6bb50af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 4 client.17022 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.425+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d6bb4284f0 con 0x55d6bb508b30 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.426+0000 7f9511e00640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb508b30 0x55d6bb50af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.426+0000 7f9511e00640 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb508b30 0x55d6bb50af20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f95d43a1ce9544c0 server_cookie=13e032d0f8cff974 in_seq=0 out_seq=0 2024-03-08T23:15:21.440 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.426+0000 7f94f3e00640 10 client.17022 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f94f3e00640 1 -- 172.21.15.46:0/126277030 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d6bb4284f0 con 0x55d6bb508b30 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f94f3e00640 10 client.17022 handle_command_reply: tid=0 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 client.17022 shutdown 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 msgr2=0x55d6bb4f5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f950c01b040 tx=0x7f950c0044b0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 shutdown_connections 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb508b30 0x55d6bb50af20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f5e90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f5750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 --2- 172.21.15.46:0/126277030 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f94f3e00640 0 client.17022 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 >> 172.21.15.46:0/126277030 conn(0x55d6bb4e9c60 msgr2=0x55d6bb4eb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 shutdown_connections 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.427+0000 7f951bd6d740 1 -- 172.21.15.46:0/126277030 wait complete. 2024-03-08T23:15:21.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.428+0000 7f951bd6d740 20 client.17022 trim_cache size 0 max 16384 2024-03-08T23:15:21.444 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 Processor -- start 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 -- start start 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141b2aa0 con 0x55d6bb4efe50 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141b0640 con 0x55d6bb4f8a30 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f951bd6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141ae700 con 0x55d6bb4ef530 2024-03-08T23:15:21.445 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f9511400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f9511400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55660/0 (socket says 172.21.15.46:55660) 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f9511400640 1 -- 172.21.15.46:0/4056481464 learned_addr learned my addr 172.21.15.46:0/4056481464 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f9511e00640 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.441+0000 7f9512800640 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 -- 172.21.15.46:0/4056481464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4ef530 msgr2=0x55d6bb4f7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 -- 172.21.15.46:0/4056481464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 msgr2=0x55d6bb4f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.447 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 -- 172.21.15.46:0/4056481464 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d6bb4f3ca0 con 0x55d6bb4efe50 2024-03-08T23:15:21.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f9511400640 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f95000b90b0 tx=0x7f950002bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff236a06dfc899 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f94fea00640 1 -- 172.21.15.46:0/4056481464 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000bc5b0 con 0x55d6bb4efe50 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f94fea00640 1 -- 172.21.15.46:0/4056481464 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95000f6030 con 0x55d6bb4efe50 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.442+0000 7f94fea00640 1 -- 172.21.15.46:0/4056481464 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95000bafe0 con 0x55d6bb4efe50 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- 172.21.15.46:0/4056481464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 msgr2=0x55d6bb4f84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f95000b90b0 tx=0x7f950002bc10 comp rx=0 tx=0).stop 2024-03-08T23:15:21.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- 172.21.15.46:0/4056481464 shutdown_connections 2024-03-08T23:15:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb4f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4efe50 0x55d6bb4f84d0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 --2- 172.21.15.46:0/4056481464 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4ef530 0x55d6bb4f7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- 172.21.15.46:0/4056481464 >> 172.21.15.46:0/4056481464 conn(0x55d6bb4e9c60 msgr2=0x55d6bb4f4780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- 172.21.15.46:0/4056481464 shutdown_connections 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- 172.21.15.46:0/4056481464 wait complete. 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.443+0000 7f951bd6d740 1 -- start start 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 Processor -- start 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4feeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb55eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141aed00 con 0x55d6bb4ef530 2024-03-08T23:15:21.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141affe0 con 0x55d6bb4f8a30 2024-03-08T23:15:21.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95141af320 con 0x55d6bb4efe50 2024-03-08T23:15:21.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f951bd6d740 10 client.0 _ll_register_callbacks cb 0x7f950c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:21.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f9511e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f9512800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb55eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f9511e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55666/0 (socket says 172.21.15.46:55666) 2024-03-08T23:15:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f9511e00640 1 -- 172.21.15.46:0/2745987114 learned_addr learned my addr 172.21.15.46:0/2745987114 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.444+0000 7f9511400640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4feeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 -- 172.21.15.46:0/2745987114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 msgr2=0x55d6bb4feeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4feeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 -- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 msgr2=0x55d6bb55eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb55eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 -- 172.21.15.46:0/2745987114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d6bb4ff740 con 0x55d6bb4ef530 2024-03-08T23:15:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9512800640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb55eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:21.458 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511400640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4feeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f9511e00640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55d6bb50b6f0 tx=0x7f95040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b003f3fee7065d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.445+0000 7f94f3e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9504005100 con 0x55d6bb4ef530 2024-03-08T23:15:21.459 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f95040052a0 con 0x55d6bb4ef530 2024-03-08T23:15:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95000f6030 con 0x55d6bb4ef530 2024-03-08T23:15:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950403b050 con 0x55d6bb4ef530 2024-03-08T23:15:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 10 client.17028 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:21.460 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55d6bb560540 con 0x55d6bb4ef530 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95000f6030 con 0x55d6bb4ef530 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f95040116a0 con 0x55d6bb4ef530 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 10 client.17028 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 10 client.17028 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 20 client.17028 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.446+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb560c50 0x55d6bb563070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f951bd6d740 4 client.17028 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:21.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d6bb461310 con 0x55d6bb560c50 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f9511400640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb560c50 0x55d6bb563070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f9511400640 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb560c50 0x55d6bb563070 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=609bd06728b09e1d server_cookie=6df7010f2ef2b460 in_seq=0 out_seq=0 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f94f3e00640 10 client.17028 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f94f3e00640 1 -- 172.21.15.46:0/2745987114 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d6bb461310 con 0x55d6bb560c50 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f94f3e00640 10 client.17028 handle_command_reply: tid=0 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.447+0000 7f951bd6d740 1 client.17028 shutdown 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 msgr2=0x55d6bb4fe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.462 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55d6bb50b6f0 tx=0x7f95040046c0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 shutdown_connections 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55d6bb560c50 0x55d6bb563070 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55d6bb4f8a30 0x55d6bb55eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55d6bb4efe50 0x55d6bb4feeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 --2- 172.21.15.46:0/2745987114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55d6bb4ef530 0x55d6bb4fe970 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f94f3e00640 0 client.17028 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 >> 172.21.15.46:0/2745987114 conn(0x55d6bb4e9c60 msgr2=0x55d6bb4f0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 shutdown_connections 2024-03-08T23:15:21.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 1 -- 172.21.15.46:0/2745987114 wait complete. 2024-03-08T23:15:21.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.448+0000 7f951bd6d740 20 client.17028 trim_cache size 0 max 16384 2024-03-08T23:15:21.464 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 msgr2=0x7f94f80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 0x7f94f80a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x55d6bb4ef000 tx=0x7f9504009260 comp rx=0 tx=0).stop 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 msgr2=0x7f951419fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9500037040 tx=0x7f95000344c0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 shutdown_connections 2024-03-08T23:15:21.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f94f80a2970 0x7f94f80a4e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f951410d9a0 0x7f9514199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9514074040 0x7f951419fe20 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 --2- 172.21.15.46:0/3061795815 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f9514073720 0x7f951419f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 >> 172.21.15.46:0/3061795815 conn(0x7f95140fcc50 msgr2=0x7f95140fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 shutdown_connections 2024-03-08T23:15:21.466 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:21.449+0000 7f9513200640 1 -- 172.21.15.46:0/3061795815 wait complete. 2024-03-08T23:15:21.467 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:21 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/293612471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:21.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:21 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/293612471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:21.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:21 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/293612471' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:22.470 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:15:22.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:15:22.471 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:15:22.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:22 smithi062 ceph-mon[35761]: pgmap v234: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:22.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 -- 172.21.15.46:0/828521960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641014f0 msgr2=0x7fa2641018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 --2- 172.21.15.46:0/828521960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641014f0 0x7fa2641018d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fa254009900 tx=0x7fa25402bc50 comp rx=0 tx=0).stop 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 -- 172.21.15.46:0/828521960 shutdown_connections 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 --2- 172.21.15.46:0/828521960 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641065c0 0x7fa26410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 --2- 172.21.15.46:0/828521960 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264101ea0 0x7fa264105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 --2- 172.21.15.46:0/828521960 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641014f0 0x7fa2641018d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 -- 172.21.15.46:0/828521960 >> 172.21.15.46:0/828521960 conn(0x7fa2640786f0 msgr2=0x7fa264078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 -- 172.21.15.46:0/828521960 shutdown_connections 2024-03-08T23:15:22.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.696+0000 7fa26a200640 1 -- 172.21.15.46:0/828521960 wait complete. 2024-03-08T23:15:22.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 Processor -- start 2024-03-08T23:15:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 -- start start 2024-03-08T23:15:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641014f0 0x7fa2641081b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264108c30 0x7fa2641aa3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641116d0 con 0x7fa2641065c0 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264111550 con 0x7fa264108c30 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.697+0000 7fa26a200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa264111850 con 0x7fa2641014f0 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55682/0 (socket says 172.21.15.46:55682) 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 -- 172.21.15.46:0/520170319 learned_addr learned my addr 172.21.15.46:0/520170319 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa263400640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641014f0 0x7fa2641081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa263e00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264108c30 0x7fa2641aa3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 -- 172.21.15.46:0/520170319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641014f0 msgr2=0x7fa2641081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641014f0 0x7fa2641081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 -- 172.21.15.46:0/520170319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264108c30 msgr2=0x7fa2641aa3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264108c30 0x7fa2641aa3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.698+0000 7fa262a00640 1 -- 172.21.15.46:0/520170319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2641aa8e0 con 0x7fa2641065c0 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa262a00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005fd0 tx=0x7fa24c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa25be00640 1 -- 172.21.15.46:0/520170319 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c017070 con 0x7fa2641065c0 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2641aabd0 con 0x7fa2641065c0 2024-03-08T23:15:22.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa25be00640 1 -- 172.21.15.46:0/520170319 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa24c009c30 con 0x7fa2641065c0 2024-03-08T23:15:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa25be00640 1 -- 172.21.15.46:0/520170319 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c010460 con 0x7fa2641065c0 2024-03-08T23:15:22.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.699+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2641ab140 con 0x7fa2641065c0 2024-03-08T23:15:22.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.702+0000 7fa26ce07740 1 Processor -- start 2024-03-08T23:15:22.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.702+0000 7fa25be00640 1 -- 172.21.15.46:0/520170319 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa24c003e70 con 0x7fa2641065c0 2024-03-08T23:15:22.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.702+0000 7fa26ce07740 1 -- start start 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x5559258169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x555925821ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55592577aa50 con 0x555925816f00 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555925723b00 con 0x55592581fae0 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa26ce07740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555925776af0 con 0x5559258165e0 2024-03-08T23:15:22.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa263400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa263400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55692/0 (socket says 172.21.15.46:55692) 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa263400640 1 -- 172.21.15.46:0/2820271780 learned_addr learned my addr 172.21.15.46:0/2820271780 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa262a00640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x555925821ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.703+0000 7fa263e00640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x5559258169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 -- 172.21.15.46:0/2820271780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 msgr2=0x5559258169c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x5559258169c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 -- 172.21.15.46:0/2820271780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 msgr2=0x555925821ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x555925821ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 -- 172.21.15.46:0/2820271780 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559257411e0 con 0x555925816f00 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa262a00640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x555925821ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa25be00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 0x7fa2480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263400640 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa254009230 tx=0x7fa254005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4ff6e3e47469a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa25be00640 1 -- 172.21.15.46:0/520170319 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fa24c0cedf0 con 0x7fa2641065c0 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa263e00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 0x7fa2480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa253e00640 1 -- 172.21.15.46:0/2820271780 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254042070 con 0x555925816f00 2024-03-08T23:15:22.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.704+0000 7fa253e00640 1 -- 172.21.15.46:0/2820271780 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa254004240 con 0x555925816f00 2024-03-08T23:15:22.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa253e00640 1 -- 172.21.15.46:0/2820271780 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254004f60 con 0x555925816f00 2024-03-08T23:15:22.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 -- 172.21.15.46:0/2820271780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 msgr2=0x55592581f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fa254009230 tx=0x7fa254005dd0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa263e00640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 0x7fa2480a4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa25c005fd0 tx=0x7fa25c009600 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 -- 172.21.15.46:0/2820271780 shutdown_connections 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x555925821ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581f5a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2820271780 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x5559258169c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 -- 172.21.15.46:0/2820271780 >> 172.21.15.46:0/2820271780 conn(0x555925810d10 msgr2=0x5559258110f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.705+0000 7fa26ce07740 1 -- 172.21.15.46:0/2820271780 shutdown_connections 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.706+0000 7fa26ce07740 1 -- 172.21.15.46:0/2820271780 wait complete. 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.706+0000 7fa26ce07740 1 -- start start 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.706+0000 7fa26ce07740 1 Processor -- start 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x55592581caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x55592581d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555925822870 con 0x555925816f00 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555925736f60 con 0x55592581fae0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559258229f0 con 0x5559258165e0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa26ce07740 10 client.0 _ll_register_callbacks cb 0x555925722b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55698/0 (socket says 172.21.15.46:55698) 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 -- 172.21.15.46:0/1059244645 learned_addr learned my addr 172.21.15.46:0/1059244645 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa263e00640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x55592581d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa263400640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x55592581caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 -- 172.21.15.46:0/1059244645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 msgr2=0x55592581caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x55592581caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 -- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 msgr2=0x55592581d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x55592581d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa262a00640 1 -- 172.21.15.46:0/1059244645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559255577e0 con 0x555925816f00 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.707+0000 7fa263e00640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x55592581d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa262a00640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0cc180 tx=0x7fa24c0d2430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4bdad80aea0a569 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c09a990 con 0x555925816f00 2024-03-08T23:15:22.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fa24c095ba0 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa254004240 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24c0cc5a0 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa26ce07740 10 client.17040 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5559255b7f70 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.708+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa254004240 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa24c0b9b30 con 0x555925816f00 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 10 client.17040 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:22.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 20 client.17040 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55592582ff00 0x555925832320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 4 client.17040 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555925787f90 con 0x55592582ff00 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa263400640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55592582ff00 0x555925832320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa263400640 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55592582ff00 0x555925832320 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c7d875cd88f0cf1 server_cookie=11b136444bff111a in_seq=0 out_seq=0 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.709+0000 7fa250c00640 10 client.17040 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:22.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.710+0000 7fa250c00640 1 -- 172.21.15.46:0/1059244645 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555925787f90 con 0x55592582ff00 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.710+0000 7fa250c00640 10 client.17040 handle_command_reply: tid=0 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.710+0000 7fa26ce07740 1 client.17040 shutdown 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.710+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 msgr2=0x55592581cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.710+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa24c0cc180 tx=0x7fa24c0d2430 comp rx=0 tx=0).stop 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 shutdown_connections 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55592582ff00 0x555925832320 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x55592581d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581cfe0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 --2- 172.21.15.46:0/1059244645 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x55592581caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 >> 172.21.15.46:0/1059244645 conn(0x555925810d10 msgr2=0x555925812a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa250c00640 0 client.17040 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 shutdown_connections 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 1 -- 172.21.15.46:0/1059244645 wait complete. 2024-03-08T23:15:22.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.711+0000 7fa26ce07740 20 client.17040 trim_cache size 0 max 16384 2024-03-08T23:15:22.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:22 smithi046 ceph-mon[31859]: pgmap v234: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:22.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 Processor -- start 2024-03-08T23:15:22.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 -- start start 2024-03-08T23:15:22.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5559258165e0 0x55592581f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55592581fae0 0x55592581b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b6fb0 con 0x555925816f00 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b4b50 con 0x5559258165e0 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa26ce07740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b2c10 con 0x55592581fae0 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.728+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55706/0 (socket says 172.21.15.46:55706) 2024-03-08T23:15:22.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 -- 172.21.15.46:0/3725791067 learned_addr learned my addr 172.21.15.46:0/3725791067 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa263400640 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5559258165e0 0x55592581f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa263e00640 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55592581fae0 0x55592581b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 -- 172.21.15.46:0/3725791067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55592581fae0 msgr2=0x55592581b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55592581fae0 0x55592581b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 -- 172.21.15.46:0/3725791067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5559258165e0 msgr2=0x55592581f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5559258165e0 0x55592581f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 -- 172.21.15.46:0/3725791067 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55592544f170 con 0x555925816f00 2024-03-08T23:15:22.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa262a00640 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009610 tx=0x7fa24c00b550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=252d6a80fca175b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.729+0000 7fa253400640 1 -- 172.21.15.46:0/3725791067 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c009de0 con 0x555925816f00 2024-03-08T23:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa253400640 1 -- 172.21.15.46:0/3725791067 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa24c09b030 con 0x555925816f00 2024-03-08T23:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa253400640 1 -- 172.21.15.46:0/3725791067 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c095ba0 con 0x555925816f00 2024-03-08T23:15:22.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 -- 172.21.15.46:0/3725791067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 msgr2=0x55592581b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa24c009610 tx=0x7fa24c00b550 comp rx=0 tx=0).stop 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 -- 172.21.15.46:0/3725791067 shutdown_connections 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55592581fae0 0x55592581b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x55592581b370 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 --2- 172.21.15.46:0/3725791067 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5559258165e0 0x55592581f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 -- 172.21.15.46:0/3725791067 >> 172.21.15.46:0/3725791067 conn(0x555925810d10 msgr2=0x555925817d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 -- 172.21.15.46:0/3725791067 shutdown_connections 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.730+0000 7fa26ce07740 1 -- 172.21.15.46:0/3725791067 wait complete. 2024-03-08T23:15:22.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.731+0000 7fa26ce07740 1 -- start start 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.731+0000 7fa26ce07740 1 Processor -- start 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.731+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x555925826c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.731+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.731+0000 7fa26ce07740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x5559258251f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa26ce07740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b3210 con 0x555925816f00 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa26ce07740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b44f0 con 0x55592581fae0 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa26ce07740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2641b3830 con 0x5559258165e0 2024-03-08T23:15:22.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa26ce07740 10 client.0 _ll_register_callbacks cb 0x7fa254005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55710/0 (socket says 172.21.15.46:55710) 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 -- 172.21.15.46:0/2038004328 learned_addr learned my addr 172.21.15.46:0/2038004328 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa263e00640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x5559258251f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa263400640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x555925826c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 -- 172.21.15.46:0/2038004328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 msgr2=0x555925826c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x555925826c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 -- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 msgr2=0x5559258251f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x5559258251f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.732+0000 7fa262a00640 1 -- 172.21.15.46:0/2038004328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555925746050 con 0x555925816f00 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa262a00640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa24c012040 tx=0x7fa24c009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75548acc2daa451 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa24c009de0 con 0x555925816f00 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fa24c095da0 con 0x555925816f00 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa24c09b030 con 0x555925816f00 2024-03-08T23:15:22.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa24c0cc5b0 con 0x555925816f00 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa26ce07740 10 client.17046 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555925746050 con 0x555925816f00 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa24c079e50 con 0x555925816f00 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.733+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa24c0840e0 con 0x555925816f00 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 10 client.17046 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 10 client.17046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 20 client.17046 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:22.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555925887060 0x555925889480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 4 client.17046 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5559257822e0 con 0x555925887060 2024-03-08T23:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa263400640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555925887060 0x555925889480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa263400640 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555925887060 0x555925889480 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dd7885d0b15b434 server_cookie=67ce236a7db76f02 in_seq=0 out_seq=0 2024-03-08T23:15:22.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.734+0000 7fa250c00640 10 client.17046 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa250c00640 1 -- 172.21.15.46:0/2038004328 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5559257822e0 con 0x555925887060 2024-03-08T23:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa250c00640 10 client.17046 handle_command_reply: tid=0 2024-03-08T23:15:22.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 client.17046 shutdown 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 msgr2=0x555925827190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa24c012040 tx=0x7fa24c009740 comp rx=0 tx=0).stop 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 shutdown_connections 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555925887060 0x555925889480 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55592581fae0 0x5559258251f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555925816f00 0x555925827190 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 --2- 172.21.15.46:0/2038004328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5559258165e0 0x555925826c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa250c00640 0 client.17046 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.736+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 >> 172.21.15.46:0/2038004328 conn(0x555925810d10 msgr2=0x55592581b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 shutdown_connections 2024-03-08T23:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26ce07740 1 -- 172.21.15.46:0/2038004328 wait complete. 2024-03-08T23:15:22.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26ce07740 20 client.17046 trim_cache size 0 max 16384 2024-03-08T23:15:22.747 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:22.748 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:15:22.748 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "6d96a2d0-0a21-4aac-bf66-d18e2cad1b76", 2024-03-08T23:15:22.748 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:15:22.748 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 msgr2=0x7fa2480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 0x7fa2480a4eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa25c005fd0 tx=0x7fa25c009600 comp rx=0 tx=0).stop 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 msgr2=0x7fa2641086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fa24c005fd0 tx=0x7fa24c00dde0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 shutdown_connections 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2480a29f0 0x7fa2480a4eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa264108c30 0x7fa2641aa3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa2641065c0 0x7fa2641086f0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 --2- 172.21.15.46:0/520170319 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa2641014f0 0x7fa2641081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.737+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 >> 172.21.15.46:0/520170319 conn(0x7fa2640786f0 msgr2=0x7fa2640ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.738+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 shutdown_connections 2024-03-08T23:15:22.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:22.738+0000 7fa26a200640 1 -- 172.21.15.46:0/520170319 wait complete. 2024-03-08T23:15:22.762 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:15:22.762 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:15:22.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:22 smithi192 ceph-mon[36700]: pgmap v234: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:15:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 -- 172.21.15.46:0/2476338269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1013d0 msgr2=0x7f35ec1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 --2- 172.21.15.46:0/2476338269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1013d0 0x7f35ec1017b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009900 tx=0x7f35dc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:15:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 -- 172.21.15.46:0/2476338269 shutdown_connections 2024-03-08T23:15:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 --2- 172.21.15.46:0/2476338269 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.052 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 --2- 172.21.15.46:0/2476338269 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec101d80 0x7f35ec105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 --2- 172.21.15.46:0/2476338269 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1013d0 0x7f35ec1017b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 -- 172.21.15.46:0/2476338269 >> 172.21.15.46:0/2476338269 conn(0x7f35ec0fcc90 msgr2=0x7f35ec0ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.048+0000 7f35ebe00640 1 -- 172.21.15.46:0/2476338269 shutdown_connections 2024-03-08T23:15:23.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.049+0000 7f35ebe00640 1 -- 172.21.15.46:0/2476338269 wait complete. 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.050+0000 7f35ebe00640 1 Processor -- start 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.050+0000 7f35ebe00640 1 -- start start 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec19dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1978a0 0x7f35ec197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec111580 con 0x7f35ec1064a0 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec111400 con 0x7f35ec1978a0 2024-03-08T23:15:23.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ebe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec111700 con 0x7f35ec1013d0 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:60770/0 (socket says 172.21.15.46:60770) 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 -- 172.21.15.46:0/692345746 learned_addr learned my addr 172.21.15.46:0/692345746 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eb400640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1978a0 0x7f35ec197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 -- 172.21.15.46:0/692345746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1978a0 msgr2=0x7f35ec197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1978a0 0x7f35ec197d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35eaa00640 1 -- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 msgr2=0x7f35ec19dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.051+0000 7f35ea000640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec19dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35eaa00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec19dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35eaa00640 1 -- 172.21.15.46:0/692345746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ec198430 con 0x7f35ec1013d0 2024-03-08T23:15:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35ea000640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec19dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:15:23.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35eaa00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35dc02c240 tx=0x7f35dc041c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35e3400640 1 -- 172.21.15.46:0/692345746 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc004450 con 0x7f35ec1013d0 2024-03-08T23:15:23.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35e3400640 1 -- 172.21.15.46:0/692345746 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35dc005000 con 0x7f35ec1013d0 2024-03-08T23:15:23.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35e3400640 1 -- 172.21.15.46:0/692345746 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc003a70 con 0x7f35ec1013d0 2024-03-08T23:15:23.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.052+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35ec1986c0 con 0x7f35ec1013d0 2024-03-08T23:15:23.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.053+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35ec0737c0 con 0x7f35ec1013d0 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 Processor -- start 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 -- start start 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa65760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa68090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4fa0d410 con 0x55ef4fa65ca0 2024-03-08T23:15:23.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f9afd50 con 0x55ef4fa4b170 2024-03-08T23:15:23.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.057+0000 7f35f49aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f958b80 con 0x55ef4fa4bb20 2024-03-08T23:15:23.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43712/0 (socket says 172.21.15.46:43712) 2024-03-08T23:15:23.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 -- 172.21.15.46:0/2768816433 learned_addr learned my addr 172.21.15.46:0/2768816433 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:23.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35ea000640 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa68090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 -- 172.21.15.46:0/2768816433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 msgr2=0x55ef4fa65760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:15:23.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa65760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 -- 172.21.15.46:0/2768816433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 msgr2=0x55ef4fa68090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa68090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35eb400640 1 -- 172.21.15.46:0/2768816433 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef4f78e7d0 con 0x55ef4fa4b170 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.058+0000 7f35e3400640 1 -- 172.21.15.46:0/692345746 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f35dc0051b0 con 0x7f35ec1013d0 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.059+0000 7f35eb400640 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009f50 tx=0x7f35e40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff88f4a2f2c9f85e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.059+0000 7f35d7e00640 1 -- 172.21.15.46:0/2768816433 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4004b20 con 0x55ef4fa4b170 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.059+0000 7f35d7e00640 1 -- 172.21.15.46:0/2768816433 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35e4004cc0 con 0x55ef4fa4b170 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.059+0000 7f35d7e00640 1 -- 172.21.15.46:0/2768816433 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e4005020 con 0x55ef4fa4b170 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 -- 172.21.15.46:0/2768816433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 msgr2=0x55ef4fa4b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009f50 tx=0x7f35e40118d0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 -- 172.21.15.46:0/2768816433 shutdown_connections 2024-03-08T23:15:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa68090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa65760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2768816433 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa4b550 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.060+0000 7f35f49aa740 1 -- 172.21.15.46:0/2768816433 >> 172.21.15.46:0/2768816433 conn(0x55ef4fa45ec0 msgr2=0x55ef4fa462a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.061+0000 7f35f49aa740 1 -- 172.21.15.46:0/2768816433 shutdown_connections 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.061+0000 7f35f49aa740 1 -- 172.21.15.46:0/2768816433 wait complete. 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.061+0000 7f35e3400640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 0x7f35d00a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.062+0000 7f35f49aa740 1 -- start start 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.062+0000 7f35e3400640 1 -- 172.21.15.46:0/692345746 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f35dc0f8f00 con 0x7f35ec1013d0 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.062+0000 7f35f49aa740 1 Processor -- start 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.062+0000 7f35eb400640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 0x7f35d00a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa63250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa63790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f96bee0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4f9abdf0 con 0x55ef4fa4b170 2024-03-08T23:15:23.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ef4fa68a60 con 0x55ef4fa65ca0 2024-03-08T23:15:23.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35f49aa740 10 client.0 _ll_register_callbacks cb 0x55ef4f9728d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:23.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43718/0 (socket says 172.21.15.46:43718) 2024-03-08T23:15:23.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35eaa00640 1 -- 172.21.15.46:0/1335279141 learned_addr learned my addr 172.21.15.46:0/1335279141 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:23.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35ea000640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa63250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.063+0000 7f35eb400640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa63790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 -- 172.21.15.46:0/1335279141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 msgr2=0x55ef4fa63790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa63790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 -- 172.21.15.46:0/1335279141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 msgr2=0x55ef4fa63250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa63250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 -- 172.21.15.46:0/1335279141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef4f6c43f0 con 0x55ef4fa4b170 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eb400640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa63790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35ea000640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa63250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eb400640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 0x7f35d00a4e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f35e4004440 tx=0x7f35e4000f40 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35eaa00640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0c4f50 tx=0x7f35dc0c4e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3066c96c96c973cf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0c5250 con 0x55ef4fa4b170 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f35dc0e50b0 con 0x55ef4fa4b170 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.064+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35e4004cc0 con 0x55ef4fa4b170 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35dc0c93a0 con 0x55ef4fa4b170 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 10 client.27215 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ef4f6840f0 con 0x55ef4fa4b170 2024-03-08T23:15:23.076 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35e4004cc0 con 0x55ef4fa4b170 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f35dc003c10 con 0x55ef4fa4b170 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 10 client.27215 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 10 client.27215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 20 client.27215 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4fa75db0 0x55ef4fa781a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 4 client.27215 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.065+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ef4f9bd290 con 0x55ef4fa75db0 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.066+0000 7f35ea000640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4fa75db0 0x55ef4fa781a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.077 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.067+0000 7f35ea000640 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4fa75db0 0x55ef4fa781a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dfb550e672db428 server_cookie=fee3d2d342c1033a in_seq=0 out_seq=0 2024-03-08T23:15:23.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.067+0000 7f35d4c00640 10 client.27215 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.067+0000 7f35d4c00640 1 -- 172.21.15.46:0/1335279141 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ef4f9bd290 con 0x55ef4fa75db0 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.067+0000 7f35d4c00640 10 client.27215 handle_command_reply: tid=0 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.067+0000 7f35f49aa740 1 client.27215 shutdown 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 msgr2=0x55ef4fa62d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0c4f50 tx=0x7f35dc0c4e50 comp rx=0 tx=0).stop 2024-03-08T23:15:23.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 shutdown_connections 2024-03-08T23:15:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4fa75db0 0x55ef4fa781a0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa63790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa63250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35d4c00640 0 client.27215 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1335279141 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa62d10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 >> 172.21.15.46:0/1335279141 conn(0x55ef4fa45ec0 msgr2=0x55ef4fa47c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 shutdown_connections 2024-03-08T23:15:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 1 -- 172.21.15.46:0/1335279141 wait complete. 2024-03-08T23:15:23.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.068+0000 7f35f49aa740 20 client.27215 trim_cache size 0 max 16384 2024-03-08T23:15:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.101+0000 7f35f49aa740 1 Processor -- start 2024-03-08T23:15:23.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 -- start start 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa635b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa6cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec1a5f80 con 0x55ef4fa65ca0 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec072bc0 con 0x55ef4fa4b170 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35f49aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec070c80 con 0x55ef4fa4bb20 2024-03-08T23:15:23.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43732/0 (socket says 172.21.15.46:43732) 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35eaa00640 1 -- 172.21.15.46:0/2098387215 learned_addr learned my addr 172.21.15.46:0/2098387215 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.102+0000 7f35ea000640 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa635b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eb400640 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa6cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 -- 172.21.15.46:0/2098387215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 msgr2=0x55ef4fa635b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa635b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 -- 172.21.15.46:0/2098387215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 msgr2=0x55ef4fa6cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa6cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 -- 172.21.15.46:0/2098387215 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ef4fa7d3b0 con 0x55ef4fa4b170 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35eaa00640 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0098d0 tx=0x7f35dc0c3760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14feb02765c3791b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35d7400640 1 -- 172.21.15.46:0/2098387215 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0c5250 con 0x55ef4fa4b170 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35d7400640 1 -- 172.21.15.46:0/2098387215 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35dc0cd9f0 con 0x55ef4fa4b170 2024-03-08T23:15:23.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.103+0000 7f35d7400640 1 -- 172.21.15.46:0/2098387215 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc0c3390 con 0x55ef4fa4b170 2024-03-08T23:15:23.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 -- 172.21.15.46:0/2098387215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 msgr2=0x55ef4fa65440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f35dc0098d0 tx=0x7f35dc0c3760 comp rx=0 tx=0).stop 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 -- 172.21.15.46:0/2098387215 shutdown_connections 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa65ca0 0x55ef4fa6cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa4bb20 0x55ef4fa635b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 --2- 172.21.15.46:0/2098387215 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4b170 0x55ef4fa65440 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 -- 172.21.15.46:0/2098387215 >> 172.21.15.46:0/2098387215 conn(0x55ef4fa45ec0 msgr2=0x55ef4fa5e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 -- 172.21.15.46:0/2098387215 shutdown_connections 2024-03-08T23:15:23.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.104+0000 7f35f49aa740 1 -- 172.21.15.46:0/2098387215 wait complete. 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 -- start start 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 Processor -- start 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 0x55ef4fac0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 0x55ef4fac0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fac52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec071280 con 0x55ef4fa4b170 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.105+0000 7f35f49aa740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec072560 con 0x55ef4fa4bb20 2024-03-08T23:15:23.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35f49aa740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35ec0718a0 con 0x55ef4fa65ca0 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35f49aa740 10 client.0 _ll_register_callbacks cb 0x7f35e4009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 0x55ef4fac0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35eaa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 0x55ef4fac0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:52454/0 (socket says 172.21.15.46:52454) 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35eaa00640 1 -- 172.21.15.46:0/1054481270 learned_addr learned my addr 172.21.15.46:0/1054481270 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 0x55ef4fac0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35eb400640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fac52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 -- 172.21.15.46:0/1054481270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 msgr2=0x55ef4fac52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fac52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 -- 172.21.15.46:0/1054481270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 msgr2=0x55ef4fac0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 0x55ef4fac0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 -- 172.21.15.46:0/1054481270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ef4fac58b0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35eb400640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fac52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.106+0000 7f35ea000640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 0x55ef4fac0960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55ef4fa472e0 tx=0x7f35d801af40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf57d4cd414b7d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35d4c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d8004250 con 0x55ef4fa4bb20 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35dc0cd9f0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f35d80043f0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35d8005030 con 0x55ef4fa4bb20 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35f49aa740 10 client.27221 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ef4facccd0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.107+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35dc0cd9f0 con 0x55ef4fa4bb20 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f35d800f040 con 0x55ef4fa4bb20 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 10 client.27221 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:23.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 10 client.27221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 20 client.27221 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4facd350 0x55ef4facf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 4 client.27221 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ef4f9b75e0 con 0x55ef4facd350 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.108+0000 7f35eaa00640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4facd350 0x55ef4facf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35eaa00640 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4facd350 0x55ef4facf740 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aecdf50fec16adf server_cookie=3a7f25fd79327314 in_seq=0 out_seq=0 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35d4c00640 10 client.27221 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35d4c00640 1 -- 172.21.15.46:0/1054481270 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ef4f9b75e0 con 0x55ef4facd350 2024-03-08T23:15:23.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35d4c00640 10 client.27221 handle_command_reply: tid=0 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 client.27221 shutdown 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 msgr2=0x55ef4fac0960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 0x55ef4fac0960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55ef4fa472e0 tx=0x7f35d801af40 comp rx=0 tx=0).stop 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 shutdown_connections 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55ef4facd350 0x55ef4facf740 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ef4fa65ca0 0x55ef4fac52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55ef4fa4bb20 0x55ef4fac0960 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 --2- 172.21.15.46:0/1054481270 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55ef4fa4b170 0x55ef4fac0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 >> 172.21.15.46:0/1054481270 conn(0x55ef4fa45ec0 msgr2=0x55ef4fa446b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35d4c00640 0 client.27221 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 shutdown_connections 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 1 -- 172.21.15.46:0/1054481270 wait complete. 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.109+0000 7f35f49aa740 20 client.27221 trim_cache size 0 max 16384 2024-03-08T23:15:23.121 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "6d96a2d0-0a21-4aac-bf66-d18e2cad1b76": { 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "6d96a2d0-0a21-4aac-bf66-d18e2cad1b76", 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:15:23.122 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:15:23.123 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:15:23.123 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 msgr2=0x7f35d00a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 0x7f35d00a4e90 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f35e4004440 tx=0x7f35e4000f40 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 msgr2=0x7f35ec19d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f35dc02c240 tx=0x7f35dc041c80 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 shutdown_connections 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f35d00a29d0 0x7f35d00a4e90 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f35ec1978a0 0x7f35ec197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f35ec1064a0 0x7f35ec19dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 --2- 172.21.15.46:0/692345746 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f35ec1013d0 0x7f35ec19d6e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:23.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 >> 172.21.15.46:0/692345746 conn(0x7f35ec0fcc90 msgr2=0x7f35ec0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 shutdown_connections 2024-03-08T23:15:23.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:23.110+0000 7f35ebe00640 1 -- 172.21.15.46:0/692345746 wait complete. 2024-03-08T23:15:23.131 INFO:tasks.cephfs.filesystem:scrub status for tag:6d96a2d0-0a21-4aac-bf66-d18e2cad1b76 - {'path': '/', 'tag': '6d96a2d0-0a21-4aac-bf66-d18e2cad1b76', 'options': 'recursive,force'} 2024-03-08T23:15:23.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:15:23.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:23.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:23.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:23.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:23.476 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:23 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:23.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:15:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:23.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:23 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:23.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:15:23.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:23.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:23.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:23.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:23.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:23 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:24 smithi062 ceph-mon[35761]: pgmap v235: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:15:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:24 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:24 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:24 smithi046 ceph-mon[31859]: pgmap v235: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:15:24.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:24 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:24 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:24 smithi192 ceph-mon[36700]: pgmap v235: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:15:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:24 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:24 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:26 smithi062 ceph-mon[35761]: pgmap v236: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:15:26.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:26 smithi046 ceph-mon[31859]: pgmap v236: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:15:26.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:26 smithi192 ceph-mon[36700]: pgmap v236: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:15:28.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:28 smithi062 ceph-mon[35761]: pgmap v237: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:28.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:28 smithi046 ceph-mon[31859]: pgmap v237: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:28.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:28 smithi192 ceph-mon[36700]: pgmap v237: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:30.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:30 smithi062 ceph-mon[35761]: pgmap v238: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:30.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:30 smithi046 ceph-mon[31859]: pgmap v238: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:30.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:30 smithi192 ceph-mon[36700]: pgmap v238: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:32.615 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:32 smithi192 ceph-mon[36700]: pgmap v239: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:32.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:32 smithi062 ceph-mon[35761]: pgmap v239: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:32.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:32 smithi046 ceph-mon[31859]: pgmap v239: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:33.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:33 smithi046 ceph-mon[31859]: pgmap v240: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:33.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:33 smithi192 ceph-mon[36700]: pgmap v240: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:33.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:33 smithi062 ceph-mon[35761]: pgmap v240: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:36.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:36 smithi192 ceph-mon[36700]: pgmap v241: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:36.593 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:36 smithi062 ceph-mon[35761]: pgmap v241: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:36.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:36 smithi046 ceph-mon[31859]: pgmap v241: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:37.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:37 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:37.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:37 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:37.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:37 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:37 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:37.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:37 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:37 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:38.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:38 smithi192 ceph-mon[36700]: pgmap v242: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:38.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:38 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:15:38.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:38 smithi062 ceph-mon[35761]: pgmap v242: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:38.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:38 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:15:38.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:38 smithi046 ceph-mon[31859]: pgmap v242: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:15:38.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:38 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:15:40.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:40 smithi062 ceph-mon[35761]: pgmap v243: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:40.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:40 smithi046 ceph-mon[31859]: pgmap v243: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:40.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:40 smithi192 ceph-mon[36700]: pgmap v243: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:42.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:42 smithi062 ceph-mon[35761]: pgmap v244: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:42.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:42 smithi046 ceph-mon[31859]: pgmap v244: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:42.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:42 smithi192 ceph-mon[36700]: pgmap v244: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:44.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:44 smithi062 ceph-mon[35761]: pgmap v245: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:44.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:44 smithi046 ceph-mon[31859]: pgmap v245: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:44.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:44 smithi192 ceph-mon[36700]: pgmap v245: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:46.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:46 smithi062 ceph-mon[35761]: pgmap v246: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:46.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:46 smithi046 ceph-mon[31859]: pgmap v246: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:46.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:46 smithi192 ceph-mon[36700]: pgmap v246: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:48.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:48 smithi062 ceph-mon[35761]: pgmap v247: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:48.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:48 smithi046 ceph-mon[31859]: pgmap v247: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:48.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:48 smithi192 ceph-mon[36700]: pgmap v247: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:50.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:50 smithi062 ceph-mon[35761]: pgmap v248: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:50.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:50 smithi046 ceph-mon[31859]: pgmap v248: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:50.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:50 smithi192 ceph-mon[36700]: pgmap v248: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:52.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:52 smithi062 ceph-mon[35761]: pgmap v249: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:52.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:52 smithi046 ceph-mon[31859]: pgmap v249: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:52.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:52 smithi192 ceph-mon[36700]: pgmap v249: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:53.134 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:15:53.134 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:15:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.356+0000 7f9602000640 1 -- 172.21.15.46:0/3104629729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc076140 msgr2=0x7f95fc111ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.356+0000 7f9602000640 1 --2- 172.21.15.46:0/3104629729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc076140 0x7f95fc111ad0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009930 tx=0x7f95e802bc30 comp rx=0 tx=0).stop 2024-03-08T23:15:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 -- 172.21.15.46:0/3104629729 shutdown_connections 2024-03-08T23:15:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 --2- 172.21.15.46:0/3104629729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc112010 0x7f95fc114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 --2- 172.21.15.46:0/3104629729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc076140 0x7f95fc111ad0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 --2- 172.21.15.46:0/3104629729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc075820 0x7f95fc075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 -- 172.21.15.46:0/3104629729 >> 172.21.15.46:0/3104629729 conn(0x7f95fc0fee00 msgr2=0x7f95fc101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 -- 172.21.15.46:0/3104629729 shutdown_connections 2024-03-08T23:15:53.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.357+0000 7f9602000640 1 -- 172.21.15.46:0/3104629729 wait complete. 2024-03-08T23:15:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.358+0000 7f9602000640 1 Processor -- start 2024-03-08T23:15:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.358+0000 7f9602000640 1 -- start start 2024-03-08T23:15:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc076140 0x7f95fc19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc112010 0x7f95fc1a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc117bc0 con 0x7f95fc112010 2024-03-08T23:15:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc117a40 con 0x7f95fc075820 2024-03-08T23:15:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9602000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc117d40 con 0x7f95fc076140 2024-03-08T23:15:53.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56552/0 (socket says 172.21.15.46:56552) 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 -- 172.21.15.46:0/1310093509 learned_addr learned my addr 172.21.15.46:0/1310093509 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9601600640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc112010 0x7f95fc1a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f95fbe00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc076140 0x7f95fc19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 -- 172.21.15.46:0/1310093509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc076140 msgr2=0x7f95fc19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc076140 0x7f95fc19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 -- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc112010 msgr2=0x7f95fc1a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc112010 0x7f95fc1a2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.359+0000 7f9600c00640 1 -- 172.21.15.46:0/1310093509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc1a30c0 con 0x7f95fc075820 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f9600c00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f95f000e9f0 tx=0x7f95f000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f95f9600640 1 -- 172.21.15.46:0/1310093509 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f000cd80 con 0x7f95fc075820 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc1a33b0 con 0x7f95fc075820 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f95f9600640 1 -- 172.21.15.46:0/1310093509 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f0010910 con 0x7f95fc075820 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f95f9600640 1 -- 172.21.15.46:0/1310093509 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f00188f0 con 0x7f95fc075820 2024-03-08T23:15:53.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.360+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc1aac40 con 0x7f95fc075820 2024-03-08T23:15:53.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.363+0000 7f95f9600640 1 -- 172.21.15.46:0/1310093509 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f95f0010ac0 con 0x7f95fc075820 2024-03-08T23:15:53.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 Processor -- start 2024-03-08T23:15:53.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 -- start start 2024-03-08T23:15:53.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f6135a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f6135d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f61313410 con 0x564f61351790 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f612b5d50 con 0x564f613520b0 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9606366740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f6125eb80 con 0x564f6135ac90 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44928/0 (socket says 172.21.15.46:44928) 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.364+0000 7f9601600640 1 -- 172.21.15.46:0/2315212293 learned_addr learned my addr 172.21.15.46:0/2315212293 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9600c00640 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f6135a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f95fbe00640 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f6135d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 -- 172.21.15.46:0/2315212293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 msgr2=0x564f6135d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f6135d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 -- 172.21.15.46:0/2315212293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 msgr2=0x564f6135a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f6135a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 -- 172.21.15.46:0/2315212293 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f610947d0 con 0x564f61351790 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f95f9600640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 0x7f95e40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f95f9600640 1 -- 172.21.15.46:0/1310093509 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f95f00cf5d0 con 0x7f95fc075820 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9601600640 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009f40 tx=0x7f95f4010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249f0aeb78905370 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.365+0000 7f9600c00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 0x7f95e40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f95edc00640 1 -- 172.21.15.46:0/2315212293 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4016070 con 0x564f61351790 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f95edc00640 1 -- 172.21.15.46:0/2315212293 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4004d20 con 0x564f61351790 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f95edc00640 1 -- 172.21.15.46:0/2315212293 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f40127c0 con 0x564f61351790 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f9606366740 1 -- 172.21.15.46:0/2315212293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 msgr2=0x564f61351b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f9606366740 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009f40 tx=0x7f95f4010800 comp rx=0 tx=0).stop 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.366+0000 7f9600c00640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 0x7f95e40a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f95f00cca10 tx=0x7f95f00d5550 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 -- 172.21.15.46:0/2315212293 shutdown_connections 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f6135d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f6135a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 --2- 172.21.15.46:0/2315212293 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61351b70 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 -- 172.21.15.46:0/2315212293 >> 172.21.15.46:0/2315212293 conn(0x564f6134bec0 msgr2=0x564f6134c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 -- 172.21.15.46:0/2315212293 shutdown_connections 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 -- 172.21.15.46:0/2315212293 wait complete. 2024-03-08T23:15:53.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 -- start start 2024-03-08T23:15:53.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.367+0000 7f9606366740 1 Processor -- start 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61357c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61358180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f613586c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f61271ee0 con 0x564f61351790 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f612b1df0 con 0x564f613520b0 2024-03-08T23:15:53.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f6135da80 con 0x564f6135ac90 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9606366740 10 client.0 _ll_register_callbacks cb 0x564f612788d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61357c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61357c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44930/0 (socket says 172.21.15.46:44930) 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f9601600640 1 -- 172.21.15.46:0/56976063 learned_addr learned my addr 172.21.15.46:0/56976063 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.368+0000 7f95fbe00640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61358180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f9601600640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f613586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 -- 172.21.15.46:0/56976063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 msgr2=0x564f613586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f613586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 -- 172.21.15.46:0/56976063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 msgr2=0x564f61357c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61357c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 -- 172.21.15.46:0/56976063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f60fca3f0 con 0x564f613520b0 2024-03-08T23:15:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95fbe00640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61358180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036040 tx=0x7f95e8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5763e664e1c7a86f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.369+0000 7f95e2000640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:53.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e802cbd0 con 0x564f613520b0 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f95e80046c0 con 0x564f613520b0 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95f4004d20 con 0x564f613520b0 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95e8054050 con 0x564f613520b0 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f9606366740 10 client.27233 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564f60f8a0f0 con 0x564f613520b0 2024-03-08T23:15:53.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4004d20 con 0x564f613520b0 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f95e80035f0 con 0x564f613520b0 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f9606366740 10 client.27233 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.370+0000 7f9606366740 10 client.27233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9606366740 20 client.27233 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f6136ae60 0x564f6136d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9606366740 4 client.27233 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:53.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f612c3290 con 0x564f6136ae60 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9601600640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f6136ae60 0x564f6136d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f9601600640 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f6136ae60 0x564f6136d250 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc941954cbe30a0 server_cookie=486efffdbd1608ac in_seq=0 out_seq=0 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.371+0000 7f95e2000640 10 client.27233 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f95e2000640 1 -- 172.21.15.46:0/56976063 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564f612c3290 con 0x564f6136ae60 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f95e2000640 10 client.27233 handle_command_reply: tid=0 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 client.27233 shutdown 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 msgr2=0x564f61358180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61358180 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f95e8036040 tx=0x7f95e8009560 comp rx=0 tx=0).stop 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 shutdown_connections 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f6136ae60 0x564f6136d250 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f613586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f95e2000640 0 client.27233 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61358180 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 --2- 172.21.15.46:0/56976063 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f61357c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.372+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 >> 172.21.15.46:0/56976063 conn(0x564f6134bec0 msgr2=0x564f6134dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.373+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 shutdown_connections 2024-03-08T23:15:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.373+0000 7f9606366740 1 -- 172.21.15.46:0/56976063 wait complete. 2024-03-08T23:15:53.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.373+0000 7f9606366740 20 client.27233 trim_cache size 0 max 16384 2024-03-08T23:15:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.389+0000 7f9606366740 1 Processor -- start 2024-03-08T23:15:53.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.389+0000 7f9606366740 1 -- start start 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61361cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f61362200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b6f70 con 0x564f61351790 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b4b10 con 0x564f613520b0 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9606366740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b2bd0 con 0x564f6135ac90 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44936/0 (socket says 172.21.15.46:44936) 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 -- 172.21.15.46:0/1947526247 learned_addr learned my addr 172.21.15.46:0/1947526247 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f95fbe00640 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61361cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 -- 172.21.15.46:0/1947526247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 msgr2=0x564f61362200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f61362200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 -- 172.21.15.46:0/1947526247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 msgr2=0x564f61361cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61361cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.390+0000 7f9601600640 1 -- 172.21.15.46:0/1947526247 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f613723d0 con 0x564f61351790 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f9601600640 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f95f4011010 tx=0x7f95f4005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d8aba9907aa9bd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f95ed200640 1 -- 172.21.15.46:0/1947526247 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4024030 con 0x564f61351790 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f95ed200640 1 -- 172.21.15.46:0/1947526247 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4025040 con 0x564f61351790 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f95ed200640 1 -- 172.21.15.46:0/1947526247 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4016070 con 0x564f61351790 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f9606366740 1 -- 172.21.15.46:0/1947526247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 msgr2=0x564f6135a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.391+0000 7f9606366740 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f95f4011010 tx=0x7f95f4005c80 comp rx=0 tx=0).stop 2024-03-08T23:15:53.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 -- 172.21.15.46:0/1947526247 shutdown_connections 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f6135ac90 0x564f61362200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f613520b0 0x564f61361cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 --2- 172.21.15.46:0/1947526247 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f61351790 0x564f6135a370 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 -- 172.21.15.46:0/1947526247 >> 172.21.15.46:0/1947526247 conn(0x564f6134bec0 msgr2=0x564f61352f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 -- 172.21.15.46:0/1947526247 shutdown_connections 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 -- 172.21.15.46:0/1947526247 wait complete. 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 -- start start 2024-03-08T23:15:53.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.392+0000 7f9606366740 1 Processor -- start 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 0x564f613b5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f6135ac90 0x564f613ba5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b31d0 con 0x564f613520b0 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b44b0 con 0x564f6135ac90 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1b37f0 con 0x564f61351790 2024-03-08T23:15:53.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9606366740 10 client.0 _ll_register_callbacks cb 0x7f95f4009a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9601600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 0x564f613b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9601600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 0x564f613b5590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56528/0 (socket says 172.21.15.46:56528) 2024-03-08T23:15:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f95fbe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44940/0 (socket says 172.21.15.46:44940) 2024-03-08T23:15:53.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9601600640 1 -- 172.21.15.46:0/173737652 learned_addr learned my addr 172.21.15.46:0/173737652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.393+0000 7f9601600640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f6135ac90 0x564f613ba5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 -- 172.21.15.46:0/173737652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 msgr2=0x564f613b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 0x564f613b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 -- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f6135ac90 msgr2=0x564f613ba5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f6135ac90 0x564f613ba5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 -- 172.21.15.46:0/173737652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564f613baca0 con 0x564f613520b0 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95fbe00640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009850 tx=0x7f95e80095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=997ba0b0a0f96f25 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95e2000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95e8005850 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f95e80035f0 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95f4025040 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.394+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95e80046c0 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 10 client.17076 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564f613c20e0 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4025040 con 0x564f613520b0 2024-03-08T23:15:53.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f95e8005150 con 0x564f613520b0 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 10 client.17076 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 20 client.17076 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f613c2830 0x564f613c4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 4 client.17076 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564f612bd5e0 con 0x564f613c2830 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.395+0000 7f9601600640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f613c2830 0x564f613c4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9601600640 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f613c2830 0x564f613c4c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7630d97329bce2 server_cookie=40765a13324f0b36 in_seq=0 out_seq=0 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f95e2000640 10 client.17076 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f95e2000640 1 -- 172.21.15.46:0/173737652 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x564f612bd5e0 con 0x564f613c2830 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f95e2000640 10 client.17076 handle_command_reply: tid=0 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 client.17076 shutdown 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 msgr2=0x564f613b5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f95e8009850 tx=0x7f95e80095d0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 shutdown_connections 2024-03-08T23:15:53.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x564f613c2830 0x564f613c4c50 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x564f6135ac90 0x564f613ba5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x564f613520b0 0x564f613b5ad0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f95e2000640 0 client.17076 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 --2- 172.21.15.46:0/173737652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564f61351790 0x564f613b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 >> 172.21.15.46:0/173737652 conn(0x564f6134bec0 msgr2=0x564f6134a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.396+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 shutdown_connections 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9606366740 1 -- 172.21.15.46:0/173737652 wait complete. 2024-03-08T23:15:53.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9606366740 20 client.17076 trim_cache size 0 max 16384 2024-03-08T23:15:53.408 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:53.408 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:15:53.408 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:15:53.408 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:53.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 msgr2=0x7f95e40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 0x7f95e40a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f95f00cca10 tx=0x7f95f00d5550 comp rx=0 tx=0).stop 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 msgr2=0x7f95fc19e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f95f000e9f0 tx=0x7f95f000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 shutdown_connections 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f95e40a2970 0x7f95e40a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.410 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f95fc112010 0x7f95fc1a2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f95fc076140 0x7f95fc19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 --2- 172.21.15.46:0/1310093509 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f95fc075820 0x7f95fc19e120 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 >> 172.21.15.46:0/1310093509 conn(0x7f95fc0fee00 msgr2=0x7f95fc1136d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.397+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 shutdown_connections 2024-03-08T23:15:53.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.398+0000 7f9602000640 1 -- 172.21.15.46:0/1310093509 wait complete. 2024-03-08T23:15:53.419 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:15:53.420 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:15:53.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.631+0000 7f6d8c400640 1 -- 172.21.15.46:0/3253137558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d84108260 msgr2=0x7f6d84114770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.631+0000 7f6d8c400640 1 --2- 172.21.15.46:0/3253137558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d84108260 0x7f6d84114770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009cd0 tx=0x7f6d7c02c0e0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.631+0000 7f6d8c400640 1 -- 172.21.15.46:0/3253137558 shutdown_connections 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.631+0000 7f6d8c400640 1 --2- 172.21.15.46:0/3253137558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d84108260 0x7f6d84114770 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.632+0000 7f6d8c400640 1 --2- 172.21.15.46:0/3253137558 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d841078a0 0x7f6d84107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.632+0000 7f6d8c400640 1 --2- 172.21.15.46:0/3253137558 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 0x7f6d8410dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.632+0000 7f6d8c400640 1 -- 172.21.15.46:0/3253137558 >> 172.21.15.46:0/3253137558 conn(0x7f6d84075840 msgr2=0x7f6d84077c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.632+0000 7f6d8c400640 1 -- 172.21.15.46:0/3253137558 shutdown_connections 2024-03-08T23:15:53.635 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.632+0000 7f6d8c400640 1 -- 172.21.15.46:0/3253137558 wait complete. 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 Processor -- start 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 -- start start 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d841078a0 0x7f6d8419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 0x7f6d841a2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84117cc0 con 0x7f6d84108260 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84117b40 con 0x7f6d8410d8b0 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d8c400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d84117e40 con 0x7f6d841078a0 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d89000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d89a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d841078a0 0x7f6d8419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d89000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44962/0 (socket says 172.21.15.46:44962) 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.633+0000 7f6d89000640 1 -- 172.21.15.46:0/790523825 learned_addr learned my addr 172.21.15.46:0/790523825 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d8a400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 0x7f6d841a2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 -- 172.21.15.46:0/790523825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d841078a0 msgr2=0x7f6d8419e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d841078a0 0x7f6d8419e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 -- 172.21.15.46:0/790523825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 msgr2=0x7f6d841a2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 0x7f6d841a2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 -- 172.21.15.46:0/790523825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d841a3140 con 0x7f6d84108260 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d89000640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9f0 tx=0x7f6d7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.634+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7000cd80 con 0x7f6d84108260 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.635+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d841a3430 con 0x7f6d84108260 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.635+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d70010910 con 0x7f6d84108260 2024-03-08T23:15:53.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.635+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70018990 con 0x7f6d84108260 2024-03-08T23:15:53.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.635+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d841aacc0 con 0x7f6d84108260 2024-03-08T23:15:53.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.636+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d84068530 con 0x7f6d84108260 2024-03-08T23:15:53.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.638+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6d70010ac0 con 0x7f6d84108260 2024-03-08T23:15:53.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.640+0000 7f6d82000640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 0x7f6d6c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:53.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.640+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f6d700cf520 con 0x7f6d84108260 2024-03-08T23:15:53.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.640+0000 7f6d89a00640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 0x7f6d6c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:53.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.641+0000 7f6d89a00640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 0x7f6d6c0a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009b50 tx=0x7f6d78005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:53.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.645+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6d7009b810 con 0x7f6d84108260 2024-03-08T23:15:53.892 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.890+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d8402ce70 con 0x7f6d84108260 2024-03-08T23:15:53.894 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.892+0000 7f6d82000640 1 -- 172.21.15.46:0/790523825 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f6d7009f8b0 con 0x7f6d84108260 2024-03-08T23:15:53.894 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:15:53.894 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:15:53.895 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 msgr2=0x7f6d6c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 0x7f6d6c0a4eb0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d78009b50 tx=0x7f6d78005f90 comp rx=0 tx=0).stop 2024-03-08T23:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 msgr2=0x7f6d8419e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000e9f0 tx=0x7f6d7000eeb0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.897 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 shutdown_connections 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f6d6c0a29f0 0x7f6d6c0a4eb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f6d8410d8b0 0x7f6d841a2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f6d84108260 0x7f6d8419e6e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 --2- 172.21.15.46:0/790523825 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6d841078a0 0x7f6d8419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.894+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 >> 172.21.15.46:0/790523825 conn(0x7f6d84075840 msgr2=0x7f6d84076050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.895+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 shutdown_connections 2024-03-08T23:15:53.898 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:53.895+0000 7f6d8c400640 1 -- 172.21.15.46:0/790523825 wait complete. 2024-03-08T23:15:53.918 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:53.919 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:15:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 -- 172.21.15.46:0/1951806847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70107890 msgr2=0x7f4b70107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 --2- 172.21.15.46:0/1951806847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70107890 0x7f4b70107d10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c009c10 tx=0x7f4b5c02bef0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.146 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 -- 172.21.15.46:0/1951806847 shutdown_connections 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 --2- 172.21.15.46:0/1951806847 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 0x7f4b70114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 --2- 172.21.15.46:0/1951806847 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b70107890 0x7f4b70107d10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 --2- 172.21.15.46:0/1951806847 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b7010d8a0 0x7f4b7010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.143+0000 7f4b76e00640 1 -- 172.21.15.46:0/1951806847 >> 172.21.15.46:0/1951806847 conn(0x7f4b70075830 msgr2=0x7f4b70077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.144+0000 7f4b76e00640 1 -- 172.21.15.46:0/1951806847 shutdown_connections 2024-03-08T23:15:54.147 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.144+0000 7f4b76e00640 1 -- 172.21.15.46:0/1951806847 wait complete. 2024-03-08T23:15:54.149 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.145+0000 7f4b76e00640 1 Processor -- start 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.145+0000 7f4b76e00640 1 -- start start 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.145+0000 7f4b76e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b76e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 0x7f4b7019e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b76e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7010d8a0 0x7f4b701a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b76e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70117d00 con 0x7f4b70107890 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b76e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70117b80 con 0x7f4b70108250 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b76e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70117e80 con 0x7f4b7010d8a0 2024-03-08T23:15:54.150 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44980/0 (socket says 172.21.15.46:44980) 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 -- 172.21.15.46:0/4152919200 learned_addr learned my addr 172.21.15.46:0/4152919200 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6f400640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 0x7f4b7019e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b74e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7010d8a0 0x7f4b701a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 -- 172.21.15.46:0/4152919200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7010d8a0 msgr2=0x7f4b701a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7010d8a0 0x7f4b701a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 -- 172.21.15.46:0/4152919200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 msgr2=0x7f4b7019e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.151 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 0x7f4b7019e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.146+0000 7f4b6fe00640 1 -- 172.21.15.46:0/4152919200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b701a3220 con 0x7f4b70107890 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b6fe00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000e9c0 tx=0x7f4b6000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b6cc00640 1 -- 172.21.15.46:0/4152919200 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000ccc0 con 0x7f4b70107890 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b701a3510 con 0x7f4b70107890 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b6cc00640 1 -- 172.21.15.46:0/4152919200 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b60005500 con 0x7f4b70107890 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b6cc00640 1 -- 172.21.15.46:0/4152919200 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b60010430 con 0x7f4b70107890 2024-03-08T23:15:54.152 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.147+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b701aada0 con 0x7f4b70107890 2024-03-08T23:15:54.153 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.151+0000 7f4b799fd740 1 Processor -- start 2024-03-08T23:15:54.154 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 -- start start 2024-03-08T23:15:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b6cc00640 1 -- 172.21.15.46:0/4152919200 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4b600046c0 con 0x7f4b70107890 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c1632ddd0 con 0x560c163c9c70 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c162d6c70 con 0x560c163c9350 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.152+0000 7f4b799fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c16329e70 con 0x560c163d2850 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.165 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b6fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44982/0 (socket says 172.21.15.46:44982) 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b6fe00640 1 -- 172.21.15.46:0/319117092 learned_addr learned my addr 172.21.15.46:0/319117092 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b6f400640 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 -- 172.21.15.46:0/319117092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 msgr2=0x560c163d4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d4c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 -- 172.21.15.46:0/319117092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 msgr2=0x560c163d2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 -- 172.21.15.46:0/319117092 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c163902b0 con 0x560c163c9350 2024-03-08T23:15:54.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.153+0000 7f4b74e00640 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c9730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009b80 tx=0x7f4b68011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70b9230e52764990 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b6cc00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 0x7f4b540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b65600640 1 -- 172.21.15.46:0/319117092 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004030 con 0x560c163c9350 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b65600640 1 -- 172.21.15.46:0/319117092 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b68005500 con 0x560c163c9350 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b65600640 1 -- 172.21.15.46:0/319117092 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004dd0 con 0x560c163c9350 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b6cc00640 1 -- 172.21.15.46:0/4152919200 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4b600cf4f0 con 0x7f4b70107890 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.154+0000 7f4b6fe00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 0x7f4b540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- 172.21.15.46:0/319117092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 msgr2=0x560c163c9730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c9730 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009b80 tx=0x7f4b68011810 comp rx=0 tx=0).stop 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b6fe00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 0x7f4b540a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4b60097b90 tx=0x7f4b600d5060 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- 172.21.15.46:0/319117092 shutdown_connections 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 --2- 172.21.15.46:0/319117092 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c9730 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- 172.21.15.46:0/319117092 >> 172.21.15.46:0/319117092 conn(0x560c163c3a80 msgr2=0x560c163c3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- 172.21.15.46:0/319117092 shutdown_connections 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- 172.21.15.46:0/319117092 wait complete. 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.155+0000 7f4b799fd740 1 -- start start 2024-03-08T23:15:54.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 Processor -- start 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c163d5640 con 0x560c163c9c70 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c162ea0c0 con 0x560c163c9350 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c163d57c0 con 0x560c163d2850 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b799fd740 10 client.0 _ll_register_callbacks cb 0x560c16324ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.156+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44986/0 (socket says 172.21.15.46:44986) 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 -- 172.21.15.46:0/1256213792 learned_addr learned my addr 172.21.15.46:0/1256213792 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b74e00640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b74e00640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 -- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 msgr2=0x560c163cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163cfb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 -- 172.21.15.46:0/1256213792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 msgr2=0x560c163d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 -- 172.21.15.46:0/1256213792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c160423f0 con 0x560c163c9c70 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b74e00640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b6f400640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c004af0 tx=0x7f4b5c0098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8efd59d6fe0fd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.157+0000 7f4b5a000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c033640 con 0x560c163c9c70 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4b5c0337e0 con 0x560c163c9c70 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b68005500 con 0x560c163c9c70 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b5c052050 con 0x560c163c9c70 2024-03-08T23:15:54.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b799fd740 10 client.17094 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560c1610c7d0 con 0x560c163c9c70 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b68005500 con 0x560c163c9c70 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.158+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4b5c004040 con 0x560c163c9c70 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 10 client.17094 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 10 client.17094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 20 client.17094 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c163e2910 0x560c163e4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.172 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 4 client.17094 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c163024f0 con 0x560c163e2910 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.159+0000 7f4b74e00640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c163e2910 0x560c163e4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.160+0000 7f4b74e00640 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c163e2910 0x560c163e4d00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c6418bdcd986da3 server_cookie=3509ba9b4f4e3d27 in_seq=0 out_seq=0 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.160+0000 7f4b5a000640 10 client.17094 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.160+0000 7f4b5a000640 1 -- 172.21.15.46:0/1256213792 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560c163024f0 con 0x560c163e2910 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.160+0000 7f4b5a000640 10 client.17094 handle_command_reply: tid=0 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 client.17094 shutdown 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 msgr2=0x560c163cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.173 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c004af0 tx=0x7f4b5c0098d0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 shutdown_connections 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c163e2910 0x560c163e4d00 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163cfb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163cf590 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b5a000640 0 client.17094 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 --2- 172.21.15.46:0/1256213792 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 >> 172.21.15.46:0/1256213792 conn(0x560c163c3a80 msgr2=0x560c163c57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 shutdown_connections 2024-03-08T23:15:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 1 -- 172.21.15.46:0/1256213792 wait complete. 2024-03-08T23:15:54.175 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.161+0000 7f4b799fd740 20 client.17094 trim_cache size 0 max 16384 2024-03-08T23:15:54.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.174+0000 7f4b799fd740 1 Processor -- start 2024-03-08T23:15:54.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.174+0000 7f4b799fd740 1 -- start start 2024-03-08T23:15:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c2910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d2200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.178 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b7170 con 0x560c163c9c70 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b4d10 con 0x560c163c9350 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b799fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b2dd0 con 0x560c163d2850 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44992/0 (socket says 172.21.15.46:44992) 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 -- 172.21.15.46:0/2652642652 learned_addr learned my addr 172.21.15.46:0/2652642652 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b74e00640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b74e00640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 -- 172.21.15.46:0/2652642652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 msgr2=0x560c163d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 -- 172.21.15.46:0/2652642652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 msgr2=0x560c163c2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b6f400640 1 -- 172.21.15.46:0/2652642652 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c163d4fb0 con 0x560c163c9c70 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.175+0000 7f4b74e00640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.176+0000 7f4b6f400640 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c000f80 tx=0x7f4b5c02bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b10a75a9897917 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.176+0000 7f4b64c00640 1 -- 172.21.15.46:0/2652642652 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c033840 con 0x560c163c9c70 2024-03-08T23:15:54.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.176+0000 7f4b64c00640 1 -- 172.21.15.46:0/2652642652 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b5c004700 con 0x560c163c9c70 2024-03-08T23:15:54.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.176+0000 7f4b64c00640 1 -- 172.21.15.46:0/2652642652 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c003bd0 con 0x560c163c9c70 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- 172.21.15.46:0/2652642652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 msgr2=0x560c163d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c000f80 tx=0x7f4b5c02bef0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- 172.21.15.46:0/2652642652 shutdown_connections 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9c70 0x560c163d1cc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 --2- 172.21.15.46:0/2652642652 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9350 0x560c163c2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- 172.21.15.46:0/2652642652 >> 172.21.15.46:0/2652642652 conn(0x560c163c3a80 msgr2=0x560c163c2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.185 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- 172.21.15.46:0/2652642652 shutdown_connections 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- 172.21.15.46:0/2652642652 wait complete. 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 -- start start 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.177+0000 7f4b799fd740 1 Processor -- start 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b33d0 con 0x560c163c9350 2024-03-08T23:15:54.186 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b46b0 con 0x560c163c9c70 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701b39f0 con 0x560c163d2850 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b799fd740 10 client.0 _ll_register_callbacks cb 0x7f4b68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b74e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44994/0 (socket says 172.21.15.46:44994) 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b74e00640 1 -- 172.21.15.46:0/3099914439 learned_addr learned my addr 172.21.15.46:0/3099914439 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b6f400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56682/0 (socket says 172.21.15.46:56682) 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.178+0000 7f4b74e00640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.187 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 -- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 msgr2=0x560c163d9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 -- 172.21.15.46:0/3099914439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 msgr2=0x560c163d8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 -- 172.21.15.46:0/3099914439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c16439090 con 0x560c163c9350 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b6f400640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b74e00640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4b680048a0 tx=0x7f4b68005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6fafb62b343f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b5a000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6801e040 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4b6802b160 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4b5c004700 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b6802b360 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b799fd740 10 client.17100 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.179+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560c1643a3d0 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4b5c004700 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4b68024040 con 0x560c163c9350 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 10 client.17100 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.189 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 10 client.17100 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 20 client.17100 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c1643a910 0x560c1643cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 4 client.17100 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560c1633b310 con 0x560c1643a910 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.180+0000 7f4b6f400640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c1643a910 0x560c1643cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b6f400640 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c1643a910 0x560c1643cd00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=413ff12614e37083 server_cookie=fc933c1e5a56d188 in_seq=0 out_seq=0 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b5a000640 10 client.17100 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:54.190 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b5a000640 1 -- 172.21.15.46:0/3099914439 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560c1633b310 con 0x560c1643a910 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b5a000640 10 client.17100 handle_command_reply: tid=0 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b799fd740 1 client.17100 shutdown 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 msgr2=0x560c163d86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.181+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4b680048a0 tx=0x7f4b68005f90 comp rx=0 tx=0).stop 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 shutdown_connections 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x560c1643a910 0x560c1643cd00 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560c163d2850 0x560c163d9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560c163c9c70 0x560c163d8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.191 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 --2- 172.21.15.46:0/3099914439 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560c163c9350 0x560c163d86d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b5a000640 0 client.17100 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 >> 172.21.15.46:0/3099914439 conn(0x560c163c3a80 msgr2=0x560c163c2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 shutdown_connections 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 1 -- 172.21.15.46:0/3099914439 wait complete. 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b799fd740 20 client.17100 trim_cache size 0 max 16384 2024-03-08T23:15:54.192 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:54.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.182+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 msgr2=0x7f4b540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.193 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 0x7f4b540a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4b60097b90 tx=0x7f4b600d5060 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 msgr2=0x7f4b7019e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000e9c0 tx=0x7f4b6000ee80 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 shutdown_connections 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b540a2970 0x7f4b540a4e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b7010d8a0 0x7f4b701a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4b70108250 0x7f4b7019e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 --2- 172.21.15.46:0/4152919200 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4b70107890 0x7f4b7019e280 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.194 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 >> 172.21.15.46:0/4152919200 conn(0x7f4b70075830 msgr2=0x7f4b70075ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 shutdown_connections 2024-03-08T23:15:54.195 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.183+0000 7f4b76e00640 1 -- 172.21.15.46:0/4152919200 wait complete. 2024-03-08T23:15:54.203 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:54.203 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:15:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 -- 172.21.15.46:0/1616632501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810107890 msgr2=0x7fd810107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 --2- 172.21.15.46:0/1616632501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810107890 0x7fd810107d10 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc009900 tx=0x7fd7fc02bc40 comp rx=0 tx=0).stop 2024-03-08T23:15:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 -- 172.21.15.46:0/1616632501 shutdown_connections 2024-03-08T23:15:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 --2- 172.21.15.46:0/1616632501 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810108250 0x7fd810114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 --2- 172.21.15.46:0/1616632501 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810107890 0x7fd810107d10 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 --2- 172.21.15.46:0/1616632501 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 0x7fd81010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 -- 172.21.15.46:0/1616632501 >> 172.21.15.46:0/1616632501 conn(0x7fd810075830 msgr2=0x7fd810077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 -- 172.21.15.46:0/1616632501 shutdown_connections 2024-03-08T23:15:54.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.430+0000 7fd817a00640 1 -- 172.21.15.46:0/1616632501 wait complete. 2024-03-08T23:15:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.431+0000 7fd817a00640 1 Processor -- start 2024-03-08T23:15:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 -- start start 2024-03-08T23:15:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810108250 0x7fd81019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 0x7fd8101a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810117d00 con 0x7fd81010d8a0 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810117b80 con 0x7fd810108250 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd817a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810117e80 con 0x7fd810107890 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd815000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd815000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56650/0 (socket says 172.21.15.46:56650) 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.432+0000 7fd815000640 1 -- 172.21.15.46:0/3124561971 learned_addr learned my addr 172.21.15.46:0/3124561971 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.437 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 0x7fd8101a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd80fe00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810108250 0x7fd81019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 -- 172.21.15.46:0/3124561971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810108250 msgr2=0x7fd81019e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810108250 0x7fd81019e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 -- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 msgr2=0x7fd8101a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 0x7fd8101a2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 -- 172.21.15.46:0/3124561971 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8101a3160 con 0x7fd810107890 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.433+0000 7fd815000640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e9f0 tx=0x7fd80400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.434+0000 7fd80d600640 1 -- 172.21.15.46:0/3124561971 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80400cd80 con 0x7fd810107890 2024-03-08T23:15:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.434+0000 7fd80d600640 1 -- 172.21.15.46:0/3124561971 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd804010910 con 0x7fd810107890 2024-03-08T23:15:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.434+0000 7fd80d600640 1 -- 172.21.15.46:0/3124561971 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd804018990 con 0x7fd810107890 2024-03-08T23:15:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.434+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8101a3450 con 0x7fd810107890 2024-03-08T23:15:54.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.434+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101aace0 con 0x7fd810107890 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd81a800740 1 Processor -- start 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd80d600640 1 -- 172.21.15.46:0/3124561971 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd804010ac0 con 0x7fd810107890 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd81a800740 1 -- start start 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd81a800740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 0x557c267ff730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd81a800740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c26808310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.441 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.437+0000 7fd81a800740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c2680ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd81a800740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c26763dd0 con 0x557c267ffc70 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd81a800740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c2670cc70 con 0x557c26808850 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd81a800740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c2675fe70 con 0x557c267ff350 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c26808310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 0x557c267ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c26808310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45004/0 (socket says 172.21.15.46:45004) 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815000640 1 -- 172.21.15.46:0/18365168 learned_addr learned my addr 172.21.15.46:0/18365168 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.442 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd80fe00640 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c2680ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 -- 172.21.15.46:0/18365168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 msgr2=0x557c2680ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c2680ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 -- 172.21.15.46:0/18365168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 msgr2=0x557c26808310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c26808310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.438+0000 7fd815a00640 1 -- 172.21.15.46:0/18365168 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c267c62b0 con 0x557c267ff350 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd80d600640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 0x7fd7f80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.443 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd815a00640 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 0x557c267ff730 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd808009f00 tx=0x7fd808011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f0ec8d18b23477 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd80d600640 1 -- 172.21.15.46:0/3124561971 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd8040cf490 con 0x7fd810107890 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd815000640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 0x7fd7f80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd802000640 1 -- 172.21.15.46:0/18365168 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808004d20 con 0x557c267ff350 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd802000640 1 -- 172.21.15.46:0/18365168 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd808004030 con 0x557c267ff350 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.439+0000 7fd802000640 1 -- 172.21.15.46:0/18365168 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd808005220 con 0x557c267ff350 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 -- 172.21.15.46:0/18365168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 msgr2=0x557c267ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 0x557c267ff730 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fd808009f00 tx=0x7fd808011810 comp rx=0 tx=0).stop 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd815000640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 0x7fd7f80a4eb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd8040a3940 tx=0x7fd8040d2f40 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 -- 172.21.15.46:0/18365168 shutdown_connections 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c2680ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c26808310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 --2- 172.21.15.46:0/18365168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ff350 0x557c267ff730 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 -- 172.21.15.46:0/18365168 >> 172.21.15.46:0/18365168 conn(0x557c267f9a80 msgr2=0x557c267f9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 -- 172.21.15.46:0/18365168 shutdown_connections 2024-03-08T23:15:54.450 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.440+0000 7fd81a800740 1 -- 172.21.15.46:0/18365168 wait complete. 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 -- start start 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 Processor -- start 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c26805570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c26805af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c2680b640 con 0x557c267ff350 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c267200c0 con 0x557c26808850 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c2680b7c0 con 0x557c267ffc70 2024-03-08T23:15:54.451 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.441+0000 7fd81a800740 10 client.0 _ll_register_callbacks cb 0x557c2675aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45020/0 (socket says 172.21.15.46:45020) 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 -- 172.21.15.46:0/370645529 learned_addr learned my addr 172.21.15.46:0/370645529 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd80fe00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c26805570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c26805af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 -- 172.21.15.46:0/370645529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 msgr2=0x557c26805570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c26805570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 -- 172.21.15.46:0/370645529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 msgr2=0x557c26805af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c26805af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 -- 172.21.15.46:0/370645529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c264783f0 con 0x557c267ff350 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd80fe00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c26805570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd815a00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd808009270 tx=0x7fd808009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4079f58b8be8a58f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.442+0000 7fd7f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80802d060 con 0x557c267ff350 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd808004470 con 0x557c267ff350 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd808004030 con 0x557c267ff350 2024-03-08T23:15:54.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd808034050 con 0x557c267ff350 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd81a800740 10 client.17112 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557c265427d0 con 0x557c267ff350 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd808004030 con 0x557c267ff350 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.443+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd808024070 con 0x557c267ff350 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 10 client.17112 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 10 client.17112 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 20 client.17112 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c26818800 0x557c2681abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 4 client.17112 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c267384f0 con 0x557c26818800 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.444+0000 7fd80fe00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c26818800 0x557c2681abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.445+0000 7fd80fe00640 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c26818800 0x557c2681abf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8b974a2044ac27fa server_cookie=dfd7fef5bbeb0b35 in_seq=0 out_seq=0 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.445+0000 7fd7f6a00640 10 client.17112 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.445+0000 7fd7f6a00640 1 -- 172.21.15.46:0/370645529 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557c267384f0 con 0x557c26818800 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.445+0000 7fd7f6a00640 10 client.17112 handle_command_reply: tid=0 2024-03-08T23:15:54.455 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.445+0000 7fd81a800740 1 client.17112 shutdown 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 msgr2=0x557c268081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd808009270 tx=0x7fd808009c20 comp rx=0 tx=0).stop 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 shutdown_connections 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c26808850 0x557c26805af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c26818800 0x557c2681abf0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c26805570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 --2- 172.21.15.46:0/370645529 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ff350 0x557c268081e0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 >> 172.21.15.46:0/370645529 conn(0x557c267f9a80 msgr2=0x557c267fb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.456 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd7f6a00640 0 client.17112 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 shutdown_connections 2024-03-08T23:15:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 1 -- 172.21.15.46:0/370645529 wait complete. 2024-03-08T23:15:54.457 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.446+0000 7fd81a800740 20 client.17112 trim_cache size 0 max 16384 2024-03-08T23:15:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.459+0000 7fd81a800740 1 Processor -- start 2024-03-08T23:15:54.461 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.459+0000 7fd81a800740 1 -- start start 2024-03-08T23:15:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c26804e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c2681f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c26808850 0x557c26807ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b70d0 con 0x557c26808850 2024-03-08T23:15:54.463 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b4c70 con 0x557c267ff350 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd81a800740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b2d30 con 0x557c267ffc70 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd815a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c26804e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd80fe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c2681f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd815a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c26804e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56740/0 (socket says 172.21.15.46:56740) 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd815a00640 1 -- 172.21.15.46:0/3874361816 learned_addr learned my addr 172.21.15.46:0/3874361816 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd815a00640 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c26808850 0x557c26807ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd80fe00640 1 -- 172.21.15.46:0/3874361816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 msgr2=0x557c26804e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.464 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.460+0000 7fd80fe00640 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c26804e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd80fe00640 1 -- 172.21.15.46:0/3874361816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c26808850 msgr2=0x557c26807ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd80fe00640 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c26808850 0x557c26807ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd80fe00640 1 -- 172.21.15.46:0/3874361816 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c2680b520 con 0x557c267ffc70 2024-03-08T23:15:54.465 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd80fe00640 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c2681f310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02c180 tx=0x7fd7fc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=674f6e5c8270ee74 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd801600640 1 -- 172.21.15.46:0/3874361816 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc004110 con 0x557c267ffc70 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd801600640 1 -- 172.21.15.46:0/3874361816 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7fc0042b0 con 0x557c267ffc70 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.461+0000 7fd801600640 1 -- 172.21.15.46:0/3874361816 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc004920 con 0x557c267ffc70 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 -- 172.21.15.46:0/3874361816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 msgr2=0x557c2681f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c2681f310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02c180 tx=0x7fd7fc000bc0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 -- 172.21.15.46:0/3874361816 shutdown_connections 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c26808850 0x557c26807ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.469 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c267ffc70 0x557c2681f310 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 --2- 172.21.15.46:0/3874361816 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c26804e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 -- 172.21.15.46:0/3874361816 >> 172.21.15.46:0/3874361816 conn(0x557c267f9a80 msgr2=0x557c267f9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 -- 172.21.15.46:0/3874361816 shutdown_connections 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.462+0000 7fd81a800740 1 -- 172.21.15.46:0/3874361816 wait complete. 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 -- start start 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 Processor -- start 2024-03-08T23:15:54.470 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c2680f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c26808850 0x557c2680de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b3330 con 0x557c267ffc70 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b4610 con 0x557c267ff350 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b3950 con 0x557c26808850 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.463+0000 7fd81a800740 10 client.0 _ll_register_callbacks cb 0x7fd808009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.471 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45030/0 (socket says 172.21.15.46:45030) 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 -- 172.21.15.46:0/550599248 learned_addr learned my addr 172.21.15.46:0/550599248 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd815a00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c26808850 0x557c2680de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd815a00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c2680f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 -- 172.21.15.46:0/550599248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c26808850 msgr2=0x557c2680de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c26808850 0x557c2680de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 -- 172.21.15.46:0/550599248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 msgr2=0x557c2680f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c2680f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 -- 172.21.15.46:0/550599248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c2680e5f0 con 0x557c267ffc70 2024-03-08T23:15:54.472 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd815a00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c2680f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd80fe00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02c860 tx=0x7fd7fc004240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ae408a79659ecce server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.464+0000 7fd7f6a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7fc0473c0 con 0x557c267ffc70 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd7fc047560 con 0x557c267ffc70 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7fc0042b0 con 0x557c267ffc70 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7fc02c8c0 con 0x557c267ffc70 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd81a800740 10 client.17118 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.473 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557c2686f610 con 0x557c267ffc70 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.465+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7fc0042b0 con 0x557c267ffc70 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd7fc005250 con 0x557c267ffc70 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 10 client.17118 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 10 client.17118 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 20 client.17118 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c2686fd40 0x557c26872160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 4 client.17118 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:54.474 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c26771310 con 0x557c2686fd40 2024-03-08T23:15:54.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.466+0000 7fd815a00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c2686fd40 0x557c26872160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd815a00640 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c2686fd40 0x557c26872160 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ba98e1d22bc2caf1 server_cookie=abf7fb0f9b2a3e91 in_seq=0 out_seq=0 2024-03-08T23:15:54.475 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd7f6a00640 10 client.17118 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:54.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd7f6a00640 1 -- 172.21.15.46:0/550599248 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557c26771310 con 0x557c2686fd40 2024-03-08T23:15:54.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd7f6a00640 10 client.17118 handle_command_reply: tid=0 2024-03-08T23:15:54.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 client.17118 shutdown 2024-03-08T23:15:54.476 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 msgr2=0x557c2680fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc02c860 tx=0x7fd7fc004240 comp rx=0 tx=0).stop 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 shutdown_connections 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x557c2686fd40 0x557c26872160 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557c26808850 0x557c2680de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557c267ffc70 0x557c2680fd60 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd7f6a00640 0 client.17118 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 --2- 172.21.15.46:0/550599248 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557c267ff350 0x557c2680f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 >> 172.21.15.46:0/550599248 conn(0x557c267f9a80 msgr2=0x557c2681f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 shutdown_connections 2024-03-08T23:15:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.467+0000 7fd81a800740 1 -- 172.21.15.46:0/550599248 wait complete. 2024-03-08T23:15:54.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd81a800740 20 client.17118 trim_cache size 0 max 16384 2024-03-08T23:15:54.478 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 msgr2=0x7fd7f80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 0x7fd7f80a4eb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fd8040a3940 tx=0x7fd8040d2f40 comp rx=0 tx=0).stop 2024-03-08T23:15:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 msgr2=0x7fd81019e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd80400e9f0 tx=0x7fd80400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 shutdown_connections 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd7f80a29f0 0x7fd7f80a4eb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd81010d8a0 0x7fd8101a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd810108250 0x7fd81019e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 --2- 172.21.15.46:0/3124561971 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd810107890 0x7fd81019e1c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.468+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 >> 172.21.15.46:0/3124561971 conn(0x7fd810075830 msgr2=0x7fd810075e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.469+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 shutdown_connections 2024-03-08T23:15:54.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.469+0000 7fd817a00640 1 -- 172.21.15.46:0/3124561971 wait complete. 2024-03-08T23:15:54.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:54 smithi046 ceph-mon[31859]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:54.481 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/790523825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:54.489 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:15:54.489 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:15:54.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:54 smithi062 ceph-mon[35761]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:54.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/790523825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:54.717 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 -- 172.21.15.46:0/998151665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0107890 msgr2=0x7ff4a0107d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 --2- 172.21.15.46:0/998151665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0107890 0x7ff4a0107d10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff488009930 tx=0x7ff48802bc30 comp rx=0 tx=0).stop 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 -- 172.21.15.46:0/998151665 shutdown_connections 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 --2- 172.21.15.46:0/998151665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0108250 0x7ff4a0114760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 --2- 172.21.15.46:0/998151665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0107890 0x7ff4a0107d10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 --2- 172.21.15.46:0/998151665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 0x7ff4a010dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 -- 172.21.15.46:0/998151665 >> 172.21.15.46:0/998151665 conn(0x7ff4a0075830 msgr2=0x7ff4a0077c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.718 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 -- 172.21.15.46:0/998151665 shutdown_connections 2024-03-08T23:15:54.719 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.715+0000 7ff4a5600640 1 -- 172.21.15.46:0/998151665 wait complete. 2024-03-08T23:15:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.716+0000 7ff4a5600640 1 Processor -- start 2024-03-08T23:15:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.716+0000 7ff4a5600640 1 -- start start 2024-03-08T23:15:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 0x7ff4a019e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 0x7ff4a01a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0117d00 con 0x7ff4a0108250 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0117b80 con 0x7ff4a0107890 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff4a5600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0117e80 con 0x7ff4a010d8a0 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45046/0 (socket says 172.21.15.46:45046) 2024-03-08T23:15:54.722 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff49ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 0x7ff4a019e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 -- 172.21.15.46:0/1977023324 learned_addr learned my addr 172.21.15.46:0/1977023324 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff49f400640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 0x7ff4a01a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 -- 172.21.15.46:0/1977023324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 msgr2=0x7ff4a01a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 0x7ff4a01a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 -- 172.21.15.46:0/1977023324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 msgr2=0x7ff4a019e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 0x7ff4a019e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff497e00640 1 -- 172.21.15.46:0/1977023324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a01a31d0 con 0x7ff4a0108250 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.717+0000 7ff49ea00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 0x7ff4a019e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:54.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff497e00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff488009900 tx=0x7ff488004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff497400640 1 -- 172.21.15.46:0/1977023324 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488043020 con 0x7ff4a0108250 2024-03-08T23:15:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff497400640 1 -- 172.21.15.46:0/1977023324 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4880054e0 con 0x7ff4a0108250 2024-03-08T23:15:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a01a3460 con 0x7ff4a0108250 2024-03-08T23:15:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff497400640 1 -- 172.21.15.46:0/1977023324 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488003bb0 con 0x7ff4a0108250 2024-03-08T23:15:54.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.718+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a01aad50 con 0x7ff4a0108250 2024-03-08T23:15:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.722+0000 7ff4a842b740 1 Processor -- start 2024-03-08T23:15:54.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 -- start start 2024-03-08T23:15:54.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd335310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff497400640 1 -- 172.21.15.46:0/1977023324 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff488046020 con 0x7ff4a0108250 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd335850 0x55e3bd337c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd290dd0 con 0x55e3bd32c350 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd239c70 con 0x55e3bd32cc70 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff4a842b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd28ce70 con 0x55e3bd335850 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.723+0000 7ff49f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45048/0 (socket says 172.21.15.46:45048) 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 -- 172.21.15.46:0/3522270159 learned_addr learned my addr 172.21.15.46:0/3522270159 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49ea00640 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd335310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff497e00640 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd335850 0x55e3bd337c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 -- 172.21.15.46:0/3522270159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd335850 msgr2=0x55e3bd337c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd335850 0x55e3bd337c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 -- 172.21.15.46:0/3522270159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 msgr2=0x55e3bd335310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd335310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.728 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 -- 172.21.15.46:0/3522270159 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3bd2f32b0 con 0x55e3bd32c350 2024-03-08T23:15:54.729 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.724+0000 7ff49f400640 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff498009fd0 tx=0x7ff4980118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=135153cc33643415 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff48f400640 1 -- 172.21.15.46:0/3522270159 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498004d20 con 0x55e3bd32c350 2024-03-08T23:15:54.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff48f400640 1 -- 172.21.15.46:0/3522270159 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff498004050 con 0x55e3bd32c350 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff48f400640 1 -- 172.21.15.46:0/3522270159 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498005220 con 0x55e3bd32c350 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff497400640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 0x7ff4840a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff497400640 1 -- 172.21.15.46:0/1977023324 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff4880f10f0 con 0x7ff4a0108250 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.725+0000 7ff49ea00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 0x7ff4840a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- 172.21.15.46:0/3522270159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 msgr2=0x55e3bd32c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff498009fd0 tx=0x7ff4980118d0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- 172.21.15.46:0/3522270159 shutdown_connections 2024-03-08T23:15:54.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd335850 0x55e3bd337c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd335310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 --2- 172.21.15.46:0/3522270159 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd32c730 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- 172.21.15.46:0/3522270159 >> 172.21.15.46:0/3522270159 conn(0x55e3bd326a80 msgr2=0x55e3bd326e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- 172.21.15.46:0/3522270159 shutdown_connections 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- 172.21.15.46:0/3522270159 wait complete. 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff49ea00640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 0x7ff4840a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55e3bd32be20 tx=0x7ff490005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.726+0000 7ff4a842b740 1 -- start start 2024-03-08T23:15:54.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 Processor -- start 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd332550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 0x55e3bd332ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd338640 con 0x55e3bd335850 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd24d0c0 con 0x55e3bd32cc70 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3bd3387c0 con 0x55e3bd32c350 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff4a842b740 10 client.0 _ll_register_callbacks cb 0x55e3bd287ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.727+0000 7ff49ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56740/0 (socket says 172.21.15.46:56740) 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49ea00640 1 -- 172.21.15.46:0/653486442 learned_addr learned my addr 172.21.15.46:0/653486442 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 0x55e3bd332ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff497e00640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd332550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 -- 172.21.15.46:0/653486442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 msgr2=0x55e3bd3351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 -- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 msgr2=0x55e3bd332550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd332550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 -- 172.21.15.46:0/653486442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3bcfa53f0 con 0x55e3bd335850 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff497e00640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd332550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff49f400640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 0x55e3bd332ab0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff498006010 tx=0x7ff498004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efa3525ed8603468 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.728+0000 7ff483e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498028020 con 0x55e3bd335850 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff498004410 con 0x55e3bd335850 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff498004050 con 0x55e3bd335850 2024-03-08T23:15:54.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff498033050 con 0x55e3bd335850 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff4a842b740 10 client.17130 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e3bd06f7d0 con 0x55e3bd335850 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.729+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff498004050 con 0x55e3bd335850 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff49800f4e0 con 0x55e3bd335850 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 10 client.17130 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 10 client.17130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 20 client.17130 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd3458d0 0x55e3bd347cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 4 client.17130 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3bd2654f0 con 0x55e3bd3458d0 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.730+0000 7ff49ea00640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd3458d0 0x55e3bd347cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.731+0000 7ff49ea00640 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd3458d0 0x55e3bd347cc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e294c79a21293b11 server_cookie=9e90bb9965d83510 in_seq=0 out_seq=0 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.731+0000 7ff483e00640 10 client.17130 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.731+0000 7ff483e00640 1 -- 172.21.15.46:0/653486442 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3bd2654f0 con 0x55e3bd3458d0 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff483e00640 10 client.17130 handle_command_reply: tid=0 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 client.17130 shutdown 2024-03-08T23:15:54.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 msgr2=0x55e3bd332ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 0x55e3bd332ab0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff498006010 tx=0x7ff498004cf0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 shutdown_connections 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd3458d0 0x55e3bd347cc0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd335850 0x55e3bd332ab0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd332550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 --2- 172.21.15.46:0/653486442 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff483e00640 0 client.17130 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:54.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 >> 172.21.15.46:0/653486442 conn(0x55e3bd326a80 msgr2=0x55e3bd3287d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 shutdown_connections 2024-03-08T23:15:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 1 -- 172.21.15.46:0/653486442 wait complete. 2024-03-08T23:15:54.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.732+0000 7ff4a842b740 20 client.17130 trim_cache size 0 max 16384 2024-03-08T23:15:54.748 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.745+0000 7ff4a842b740 1 Processor -- start 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.745+0000 7ff4a842b740 1 -- start start 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd334c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd32f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b7190 con 0x55e3bd32c350 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b4d30 con 0x55e3bd32cc70 2024-03-08T23:15:54.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff4a842b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b2df0 con 0x55e3bd326a80 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45068/0 (socket says 172.21.15.46:45068) 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff497e00640 1 -- 172.21.15.46:0/1701550175 learned_addr learned my addr 172.21.15.46:0/1701550175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff49ea00640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd334c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff49f400640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd32f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.746+0000 7ff497e00640 1 -- 172.21.15.46:0/1701550175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 msgr2=0x55e3bd334c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff497e00640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd334c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff497e00640 1 -- 172.21.15.46:0/1701550175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 msgr2=0x55e3bd32f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff497e00640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd32f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff497e00640 1 -- 172.21.15.46:0/1701550175 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3bd32caa0 con 0x55e3bd32c350 2024-03-08T23:15:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff49f400640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd32f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff49ea00640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd334c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff497e00640 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4880b98a0 tx=0x7ff4880b9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8874bda72ecd9ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff48ea00640 1 -- 172.21.15.46:0/1701550175 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880b96e0 con 0x55e3bd32c350 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff48ea00640 1 -- 172.21.15.46:0/1701550175 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff488004c80 con 0x55e3bd32c350 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.747+0000 7ff48ea00640 1 -- 172.21.15.46:0/1701550175 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880bcda0 con 0x55e3bd32c350 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- 172.21.15.46:0/1701550175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 msgr2=0x55e3bd3351c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7ff4880b98a0 tx=0x7ff4880b9040 comp rx=0 tx=0).stop 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- 172.21.15.46:0/1701550175 shutdown_connections 2024-03-08T23:15:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd32f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd3351c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 --2- 172.21.15.46:0/1701550175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd334c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- 172.21.15.46:0/1701550175 >> 172.21.15.46:0/1701550175 conn(0x55e3bd348030 msgr2=0x55e3bd330e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- 172.21.15.46:0/1701550175 shutdown_connections 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- 172.21.15.46:0/1701550175 wait complete. 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.748+0000 7ff4a842b740 1 -- start start 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 Processor -- start 2024-03-08T23:15:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd33b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd33bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b33f0 con 0x55e3bd32c350 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b46d0 con 0x55e3bd32cc70 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01b3a10 con 0x55e3bd326a80 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff4a842b740 10 client.0 _ll_register_callbacks cb 0x7ff498009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.749+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.758 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45074/0 (socket says 172.21.15.46:45074) 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 -- 172.21.15.46:0/2982899861 learned_addr learned my addr 172.21.15.46:0/2982899861 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff49ea00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd33b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff49f400640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd33bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 -- 172.21.15.46:0/2982899861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 msgr2=0x55e3bd33b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd33b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 -- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 msgr2=0x55e3bd33bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd33bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.759 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 -- 172.21.15.46:0/2982899861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3bd39bd00 con 0x55e3bd32c350 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff49f400640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd33bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff497e00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff4880bd140 tx=0x7ff488004ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba973660d08c27a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff483e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880bcf80 con 0x55e3bd32c350 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff48802cb90 con 0x55e3bd32c350 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.750+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff488004c80 con 0x55e3bd32c350 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4880bd210 con 0x55e3bd32c350 2024-03-08T23:15:54.760 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 10 client.17136 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e3bd39d000 con 0x55e3bd32c350 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff488004c80 con 0x55e3bd32c350 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff4880ee6a0 con 0x55e3bd32c350 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 10 client.17136 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 20 client.17136 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd335850 0x55e3bd39f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:54.761 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 4 client.17136 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.751+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e3bd29e310 con 0x55e3bd335850 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff49ea00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd335850 0x55e3bd39f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff49ea00640 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd335850 0x55e3bd39f630 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64f72eccac67dc6c server_cookie=1bc799d62b6b99a7 in_seq=0 out_seq=0 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff483e00640 10 client.17136 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff483e00640 1 -- 172.21.15.46:0/2982899861 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e3bd29e310 con 0x55e3bd335850 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff483e00640 10 client.17136 handle_command_reply: tid=0 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.752+0000 7ff4a842b740 1 client.17136 shutdown 2024-03-08T23:15:54.762 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 msgr2=0x55e3bd33bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ff4880bd140 tx=0x7ff488004ee0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 shutdown_connections 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55e3bd335850 0x55e3bd39f630 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55e3bd32cc70 0x55e3bd33bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55e3bd32c350 0x55e3bd33bab0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff483e00640 0 client.17136 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 --2- 172.21.15.46:0/2982899861 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55e3bd326a80 0x55e3bd33b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 >> 172.21.15.46:0/2982899861 conn(0x55e3bd348030 msgr2=0x55e3bd327d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 shutdown_connections 2024-03-08T23:15:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 1 -- 172.21.15.46:0/2982899861 wait complete. 2024-03-08T23:15:54.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.753+0000 7ff4a842b740 20 client.17136 trim_cache size 0 max 16384 2024-03-08T23:15:54.764 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:15:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 msgr2=0x7ff4840a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 0x7ff4840a4eb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x55e3bd32be20 tx=0x7ff490005f90 comp rx=0 tx=0).stop 2024-03-08T23:15:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 msgr2=0x7ff4a019e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7ff488009900 tx=0x7ff488004360 comp rx=0 tx=0).stop 2024-03-08T23:15:54.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 shutdown_connections 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff4840a29f0 0x7ff4840a4eb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff4a010d8a0 0x7ff4a01a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff4a0108250 0x7ff4a019e720 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 --2- 172.21.15.46:0/1977023324 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff4a0107890 0x7ff4a019e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 >> 172.21.15.46:0/1977023324 conn(0x7ff4a0075830 msgr2=0x7ff4a0075f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 shutdown_connections 2024-03-08T23:15:54.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:54.754+0000 7ff4a5600640 1 -- 172.21.15.46:0/1977023324 wait complete. 2024-03-08T23:15:54.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:54 smithi192 ceph-mon[36700]: pgmap v250: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:15:54.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/790523825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:15:55.775 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:15:55.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:15:55.776 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 -- 172.21.15.46:0/342907475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc103f90 msgr2=0x7fc3cc107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 --2- 172.21.15.46:0/342907475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc103f90 0x7fc3cc107fd0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009860 tx=0x7fc3b402bb70 comp rx=0 tx=0).stop 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 -- 172.21.15.46:0/342907475 shutdown_connections 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 --2- 172.21.15.46:0/342907475 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 --2- 172.21.15.46:0/342907475 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc103f90 0x7fc3cc107fd0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 --2- 172.21.15.46:0/342907475 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 -- 172.21.15.46:0/342907475 >> 172.21.15.46:0/342907475 conn(0x7fc3cc0fee20 msgr2=0x7fc3cc101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 -- 172.21.15.46:0/342907475 shutdown_connections 2024-03-08T23:15:56.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.014+0000 7fc3cb200640 1 -- 172.21.15.46:0/342907475 wait complete. 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.015+0000 7fc3cb200640 1 Processor -- start 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.015+0000 7fc3cb200640 1 -- start start 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 0x7fc3cc199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc113740 con 0x7fc3cc199b00 2024-03-08T23:15:56.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1135c0 con 0x7fc3cc1035e0 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3cb200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1138c0 con 0x7fc3cc1086b0 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3c9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3ca800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 0x7fc3cc199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3c9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56784/0 (socket says 172.21.15.46:56784) 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.016+0000 7fc3c9400640 1 -- 172.21.15.46:0/326898386 learned_addr learned my addr 172.21.15.46:0/326898386 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 -- 172.21.15.46:0/326898386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 msgr2=0x7fc3cc19f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc19f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 -- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 msgr2=0x7fc3cc199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 0x7fc3cc199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 -- 172.21.15.46:0/326898386 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3cc19a620 con 0x7fc3cc1086b0 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3ca800640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 0x7fc3cc199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c9400640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4037040 tx=0x7fc3b40045b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c2a00640 1 -- 172.21.15.46:0/326898386 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4041020 con 0x7fc3cc1086b0 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c2a00640 1 -- 172.21.15.46:0/326898386 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3b402c9f0 con 0x7fc3cc1086b0 2024-03-08T23:15:56.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.017+0000 7fc3c2a00640 1 -- 172.21.15.46:0/326898386 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4039040 con 0x7fc3cc1086b0 2024-03-08T23:15:56.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.018+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc19a8b0 con 0x7fc3cc1086b0 2024-03-08T23:15:56.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.018+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc1a6a50 con 0x7fc3cc1086b0 2024-03-08T23:15:56.024 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 Processor -- start 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 -- start start 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3c2a00640 1 -- 172.21.15.46:0/326898386 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc3b4044020 con 0x7fc3cc1086b0 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3848760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe384b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe37a3a50 con 0x559fe383f7a0 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe374cb00 con 0x559fe3848ca0 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.021+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe379faf0 con 0x559fe38400c0 2024-03-08T23:15:56.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45106/0 (socket says 172.21.15.46:45106) 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 -- 172.21.15.46:0/4208852768 learned_addr learned my addr 172.21.15.46:0/4208852768 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3c9400640 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe384b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3848760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 -- 172.21.15.46:0/4208852768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 msgr2=0x559fe3848760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3848760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 -- 172.21.15.46:0/4208852768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 msgr2=0x559fe384b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe384b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.022+0000 7fc3ca800640 1 -- 172.21.15.46:0/4208852768 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fe376a1e0 con 0x559fe383f7a0 2024-03-08T23:15:56.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3ca800640 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009fb0 tx=0x7fc3c40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a8e3a6fb539c76 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3bb000640 1 -- 172.21.15.46:0/4208852768 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c401e070 con 0x559fe383f7a0 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3c2a00640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 0x7fc3b00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3bb000640 1 -- 172.21.15.46:0/4208852768 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3c40035f0 con 0x559fe383f7a0 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3bb000640 1 -- 172.21.15.46:0/4208852768 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3c400f800 con 0x559fe383f7a0 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3c2a00640 1 -- 172.21.15.46:0/326898386 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc3b40f09c0 con 0x7fc3cc1086b0 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.023+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 0x7fc3b00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- 172.21.15.46:0/4208852768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 msgr2=0x559fe383fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4009fb0 tx=0x7fc3c40118d0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- 172.21.15.46:0/4208852768 shutdown_connections 2024-03-08T23:15:56.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe384b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3848760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/4208852768 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe383fb80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- 172.21.15.46:0/4208852768 >> 172.21.15.46:0/4208852768 conn(0x559fe3839ed0 msgr2=0x559fe383a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- 172.21.15.46:0/4208852768 shutdown_connections 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- 172.21.15.46:0/4208852768 wait complete. 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 0x7fc3b00a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x559fe383f270 tx=0x7fc3bc009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.024+0000 7fc3d37f1740 1 -- start start 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 Processor -- start 2024-03-08T23:15:56.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3846210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe3846750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe384ba90 con 0x559fe383f7a0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe375ff60 con 0x559fe3848ca0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fe384bc10 con 0x559fe38400c0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3d37f1740 10 client.0 _ll_register_callbacks cb 0x559fe374bb60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.025+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3846210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45110/0 (socket says 172.21.15.46:45110) 2024-03-08T23:15:56.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1699206876 learned_addr learned my addr 172.21.15.46:0/1699206876 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3ca800640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe3846750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1699206876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 msgr2=0x559fe3846210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3846210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 msgr2=0x559fe3846750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe3846750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1699206876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fe35807e0 con 0x559fe383f7a0 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9400640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3846210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:56.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00fb00 tx=0x7fc3bc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be52ad7552c0da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.026+0000 7fc3af400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc011c90 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc3bc01f420 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3c40035f0 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3bc011e30 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3d37f1740 10 client.17148 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559fe35e0f70 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.027+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3c40035f0 con 0x559fe383f7a0 2024-03-08T23:15:56.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc3bc027070 con 0x559fe383f7a0 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 10 client.17148 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 10 client.17148 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 20 client.17148 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe3859320 0x559fe385b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 4 client.17148 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fe37b0f90 con 0x559fe3859320 2024-03-08T23:15:56.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.028+0000 7fc3c9400640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe3859320 0x559fe385b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.029+0000 7fc3c9400640 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe3859320 0x559fe385b790 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aa5e1cc9681966f server_cookie=d53c6b7b0e7a6dc5 in_seq=0 out_seq=0 2024-03-08T23:15:56.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.029+0000 7fc3af400640 10 client.17148 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.029+0000 7fc3af400640 1 -- 172.21.15.46:0/1699206876 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559fe37b0f90 con 0x559fe3859320 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.029+0000 7fc3af400640 10 client.17148 handle_command_reply: tid=0 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.029+0000 7fc3d37f1740 1 client.17148 shutdown 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 msgr2=0x559fe3845cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00fb00 tx=0x7fc3bc004ab0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 shutdown_connections 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe3859320 0x559fe385b790 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe3848ca0 0x559fe3846750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe38400c0 0x559fe3846210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1699206876 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3845cd0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3af400640 0 client.17148 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 >> 172.21.15.46:0/1699206876 conn(0x559fe3839ed0 msgr2=0x559fe383bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 shutdown_connections 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1699206876 wait complete. 2024-03-08T23:15:56.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.030+0000 7fc3d37f1740 20 client.17148 trim_cache size 0 max 16384 2024-03-08T23:15:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 Processor -- start 2024-03-08T23:15:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 -- start start 2024-03-08T23:15:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe38434b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe3845c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1b2b30 con 0x559fe383f7a0 2024-03-08T23:15:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1b06d0 con 0x559fe38400c0 2024-03-08T23:15:56.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1ae790 con 0x559fe3848ca0 2024-03-08T23:15:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.047+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45114/0 (socket says 172.21.15.46:45114) 2024-03-08T23:15:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1512552901 learned_addr learned my addr 172.21.15.46:0/1512552901 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3ca800640 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe3845c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9400640 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe38434b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1512552901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 msgr2=0x559fe3845c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe3845c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1512552901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 msgr2=0x559fe38434b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe38434b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 -- 172.21.15.46:0/1512552901 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fe3478170 con 0x559fe383f7a0 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.048+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc013010 tx=0x7fc3bc005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0bebe14f26b17ae server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3ba600640 1 -- 172.21.15.46:0/1512552901 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc03c070 con 0x559fe383f7a0 2024-03-08T23:15:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3ba600640 1 -- 172.21.15.46:0/1512552901 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc011810 con 0x559fe383f7a0 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3ba600640 1 -- 172.21.15.46:0/1512552901 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc037bc0 con 0x559fe383f7a0 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1512552901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 msgr2=0x559fe3842f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc013010 tx=0x7fc3bc005f70 comp rx=0 tx=0).stop 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1512552901 shutdown_connections 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe3845c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe38434b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/1512552901 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe3842f70 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1512552901 >> 172.21.15.46:0/1512552901 conn(0x559fe3839ed0 msgr2=0x559fe38445c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1512552901 shutdown_connections 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.049+0000 7fc3d37f1740 1 -- 172.21.15.46:0/1512552901 wait complete. 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 -- start start 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 Processor -- start 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe384f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe38af880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1aed90 con 0x559fe383f7a0 2024-03-08T23:15:56.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1b0070 con 0x559fe38400c0 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.050+0000 7fc3d37f1740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc1af3b0 con 0x559fe3848ca0 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3d37f1740 10 client.0 _ll_register_callbacks cb 0x7fc3c4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45122/0 (socket says 172.21.15.46:45122) 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9e00640 1 -- 172.21.15.46:0/3357601999 learned_addr learned my addr 172.21.15.46:0/3357601999 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe384f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3ca800640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe38af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 -- 172.21.15.46:0/3357601999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 msgr2=0x559fe38af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe38af880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 -- 172.21.15.46:0/3357601999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 msgr2=0x559fe384ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9400640 1 -- 172.21.15.46:0/3357601999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fe376f050 con 0x559fe38400c0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.051+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3c9400640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe384f1d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40ede00 tx=0x7fc3b402bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4bad1ce776400f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3af400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4032640 con 0x559fe38400c0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc3b40f61a0 con 0x559fe38400c0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3bc011810 con 0x559fe38400c0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3b40bd410 con 0x559fe38400c0 2024-03-08T23:15:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3d37f1740 10 client.27317 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559fe38b0e20 con 0x559fe38400c0 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.052+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3bc011810 con 0x559fe38400c0 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc3b40d9070 con 0x559fe38400c0 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 10 client.27317 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 10 client.27317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 20 client.27317 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe38b1500 0x559fe38b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 4 client.27317 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x559fe37ab2e0 con 0x559fe38b1500 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.053+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe38b1500 0x559fe38b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.054+0000 7fc3c9e00640 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe38b1500 0x559fe38b3920 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d378f9e208f6801 server_cookie=d2106c6f203f8d0c in_seq=0 out_seq=0 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.054+0000 7fc3af400640 10 client.27317 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3af400640 1 -- 172.21.15.46:0/3357601999 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x559fe37ab2e0 con 0x559fe38b1500 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3af400640 10 client.27317 handle_command_reply: tid=0 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 client.27317 shutdown 2024-03-08T23:15:56.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 msgr2=0x559fe384f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe384f1d0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40ede00 tx=0x7fc3b402bb70 comp rx=0 tx=0).stop 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 shutdown_connections 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559fe38b1500 0x559fe38b3920 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559fe3848ca0 0x559fe38af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559fe38400c0 0x559fe384f1d0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 --2- 172.21.15.46:0/3357601999 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559fe383f7a0 0x559fe384ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3af400640 0 client.27317 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 >> 172.21.15.46:0/3357601999 conn(0x559fe3839ed0 msgr2=0x559fe38386e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 shutdown_connections 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 1 -- 172.21.15.46:0/3357601999 wait complete. 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.055+0000 7fc3d37f1740 20 client.27317 trim_cache size 0 max 16384 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "a7525ab6-e781-4d74-af0e-4427d46cc44b", 2024-03-08T23:15:56.065 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:15:56.066 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 msgr2=0x7fc3b00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 0x7fc3b00a4e30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x559fe383f270 tx=0x7fc3bc009500 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 msgr2=0x7fc3cc19fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4037040 tx=0x7fc3b40045b0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 shutdown_connections 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc3b00a2970 0x7fc3b00a4e30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc3cc199b00 0x7fc3cc199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc3cc1086b0 0x7fc3cc19fe90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 --2- 172.21.15.46:0/326898386 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc3cc1035e0 0x7fc3cc19f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 >> 172.21.15.46:0/326898386 conn(0x7fc3cc0fee20 msgr2=0x7fc3cc100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 shutdown_connections 2024-03-08T23:15:56.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.056+0000 7fc3cb200640 1 -- 172.21.15.46:0/326898386 wait complete. 2024-03-08T23:15:56.079 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:15:56.080 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:15:56.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 -- 172.21.15.46:0/3314247487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0109570 msgr2=0x7fcbf0109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 --2- 172.21.15.46:0/3314247487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0109570 0x7fcbf0109950 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4009860 tx=0x7fcbe402bb70 comp rx=0 tx=0).stop 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 -- 172.21.15.46:0/3314247487 shutdown_connections 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 --2- 172.21.15.46:0/3314247487 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0103e70 0x7fcbf0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 --2- 172.21.15.46:0/3314247487 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 --2- 172.21.15.46:0/3314247487 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0109570 0x7fcbf0109950 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 -- 172.21.15.46:0/3314247487 >> 172.21.15.46:0/3314247487 conn(0x7fcbf00fef40 msgr2=0x7fcbf0101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 -- 172.21.15.46:0/3314247487 shutdown_connections 2024-03-08T23:15:56.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.375+0000 7fcbf7800640 1 -- 172.21.15.46:0/3314247487 wait complete. 2024-03-08T23:15:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.376+0000 7fcbf7800640 1 Processor -- start 2024-03-08T23:15:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 -- start start 2024-03-08T23:15:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 0x7fcbf0075d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0109570 0x7fcbf0076280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0113830 con 0x7fcbf0103e70 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01136b0 con 0x7fcbf01034b0 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.377+0000 7fcbf7800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01139b0 con 0x7fcbf0109570 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf5800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0109570 0x7fcbf0076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56882/0 (socket says 172.21.15.46:56882) 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 -- 172.21.15.46:0/2585294619 learned_addr learned my addr 172.21.15.46:0/2585294619 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbefe00640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 0x7fcbf0075d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 -- 172.21.15.46:0/2585294619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0109570 msgr2=0x7fcbf0076280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0109570 0x7fcbf0076280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 -- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 msgr2=0x7fcbf0075d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 0x7fcbf0075d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.378+0000 7fcbf4e00640 1 -- 172.21.15.46:0/2585294619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf01aa440 con 0x7fcbf01034b0 2024-03-08T23:15:56.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbefe00640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 0x7fcbf0075d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4004590 tx=0x7fcbe40045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbed600640 1 -- 172.21.15.46:0/2585294619 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4041020 con 0x7fcbf01034b0 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbed600640 1 -- 172.21.15.46:0/2585294619 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbe402c9f0 con 0x7fcbf01034b0 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbed600640 1 -- 172.21.15.46:0/2585294619 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe4039040 con 0x7fcbf01034b0 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf01aa6d0 con 0x7fcbf01034b0 2024-03-08T23:15:56.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.379+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01aab70 con 0x7fcbf01034b0 2024-03-08T23:15:56.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.384+0000 7fcbfa426740 1 Processor -- start 2024-03-08T23:15:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.384+0000 7fcbfa426740 1 -- start start 2024-03-08T23:15:56.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b7f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b88540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b88a80 0x560f35b8ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35b41410 con 0x560f35b88a80 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35ae3d50 con 0x560f35b7f580 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbfa426740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35a8cb80 con 0x560f35b7fea0 2024-03-08T23:15:56.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbed600640 1 -- 172.21.15.46:0/2585294619 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcbe4044020 con 0x7fcbf01034b0 2024-03-08T23:15:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbf5800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b7f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbf5800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b7f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56890/0 (socket says 172.21.15.46:56890) 2024-03-08T23:15:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.385+0000 7fcbf5800640 1 -- 172.21.15.46:0/3025106123 learned_addr learned my addr 172.21.15.46:0/3025106123 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b88a80 0x560f35b8ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b88540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3025106123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 msgr2=0x560f35b7f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b7f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3025106123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b88a80 msgr2=0x560f35b8ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b88a80 0x560f35b8ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.386+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3025106123 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f358c27d0 con 0x560f35b7fea0 2024-03-08T23:15:56.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.387+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b88540 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40eddf0 tx=0x7fcbe40f6ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1364fb1d47b10493 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.387+0000 7fcbe2000640 1 -- 172.21.15.46:0/3025106123 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40ff070 con 0x560f35b7fea0 2024-03-08T23:15:56.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.387+0000 7fcbe2000640 1 -- 172.21.15.46:0/3025106123 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbe40fb420 con 0x560f35b7fea0 2024-03-08T23:15:56.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.387+0000 7fcbe2000640 1 -- 172.21.15.46:0/3025106123 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40fa5e0 con 0x560f35b7fea0 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.388+0000 7fcbfa426740 1 -- 172.21.15.46:0/3025106123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 msgr2=0x560f35b88540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.388+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b88540 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40eddf0 tx=0x7fcbe40f6ab0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.388+0000 7fcbed600640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 0x7fcbd80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbed600640 1 -- 172.21.15.46:0/2585294619 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fcbe40f09b0 con 0x7fcbf01034b0 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 -- 172.21.15.46:0/3025106123 shutdown_connections 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b88a80 0x560f35b8ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b88540 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3025106123 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b7f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 -- 172.21.15.46:0/3025106123 >> 172.21.15.46:0/3025106123 conn(0x560f35b79cb0 msgr2=0x560f35b7a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbf5800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 0x7fcbd80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 -- 172.21.15.46:0/3025106123 shutdown_connections 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.389+0000 7fcbfa426740 1 -- 172.21.15.46:0/3025106123 wait complete. 2024-03-08T23:15:56.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.390+0000 7fcbfa426740 1 -- start start 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.390+0000 7fcbfa426740 1 Processor -- start 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.390+0000 7fcbf5800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 0x7fcbd80a4eb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8005c10 tx=0x7fcbe8010040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35b85f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 0x560f35b864c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35a9fee0 con 0x560f35b7fea0 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35adfdf0 con 0x560f35b7f580 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f35b8b630 con 0x560f35b88a80 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.391+0000 7fcbfa426740 10 client.0 _ll_register_callbacks cb 0x560f35aa68d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:56902/0 (socket says 172.21.15.46:56902) 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 -- 172.21.15.46:0/1366102266 learned_addr learned my addr 172.21.15.46:0/1366102266 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbefe00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35b85f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf5800640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 0x560f35b864c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 -- 172.21.15.46:0/1366102266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 msgr2=0x560f35b864c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 0x560f35b864c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 -- 172.21.15.46:0/1366102266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 msgr2=0x560f35b85f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35b85f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 -- 172.21.15.46:0/1366102266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f357f83f0 con 0x560f35b7f580 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf5800640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 0x560f35b864c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40b9160 tx=0x7fcbe4032740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93aedc3c2b40c05f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.392+0000 7fcbd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40ff070 con 0x560f35b7f580 2024-03-08T23:15:56.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fcbe4034640 con 0x560f35b7f580 2024-03-08T23:15:56.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbe40fb420 con 0x560f35b7f580 2024-03-08T23:15:56.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbe40b8fa0 con 0x560f35b7f580 2024-03-08T23:15:56.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbfa426740 10 client.27329 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560f357b80f0 con 0x560f35b7f580 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.393+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbe40fb420 con 0x560f35b7f580 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcbe403c030 con 0x560f35b7f580 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 10 client.27329 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 10 client.27329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 20 client.27329 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35b98830 0x560f35b9ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 4 client.27329 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f35af1290 con 0x560f35b98830 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.394+0000 7fcbefe00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35b98830 0x560f35b9ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.395+0000 7fcbd6a00640 10 client.27329 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.395+0000 7fcbefe00640 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35b98830 0x560f35b9ac20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94cabb22c05f6549 server_cookie=6bfc4ff553db747a in_seq=0 out_seq=0 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.396+0000 7fcbd6a00640 1 -- 172.21.15.46:0/1366102266 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560f35af1290 con 0x560f35b98830 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.396+0000 7fcbd6a00640 10 client.27329 handle_command_reply: tid=0 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.396+0000 7fcbfa426740 1 client.27329 shutdown 2024-03-08T23:15:56.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 msgr2=0x560f35b85a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40b9160 tx=0x7fcbe4032740 comp rx=0 tx=0).stop 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 shutdown_connections 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35b98830 0x560f35b9ac20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b88a80 0x560f35b864c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35b85f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 --2- 172.21.15.46:0/1366102266 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b7f580 0x560f35b85a40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 >> 172.21.15.46:0/1366102266 conn(0x560f35b79cb0 msgr2=0x560f35b7ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbd6a00640 0 client.27329 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 shutdown_connections 2024-03-08T23:15:56.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 1 -- 172.21.15.46:0/1366102266 wait complete. 2024-03-08T23:15:56.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.397+0000 7fcbfa426740 20 client.27329 trim_cache size 0 max 16384 2024-03-08T23:15:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 Processor -- start 2024-03-08T23:15:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 -- start start 2024-03-08T23:15:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7f580 0x560f35b88170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35b8fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b7090 con 0x560f35b7f580 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b4c30 con 0x560f35b88a80 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbfa426740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b2cf0 con 0x560f35b7fea0 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7f580 0x560f35b88170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56836/0 (socket says 172.21.15.46:56836) 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbefe00640 1 -- 172.21.15.46:0/3047031772 learned_addr learned my addr 172.21.15.46:0/3047031772 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.418+0000 7fcbf5800640 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35b8fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 -- 172.21.15.46:0/3047031772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 msgr2=0x560f35b8fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35b8fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 -- 172.21.15.46:0/3047031772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7f580 msgr2=0x560f35b88170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7f580 0x560f35b88170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 -- 172.21.15.46:0/3047031772 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f35b9b1c0 con 0x560f35b7fea0 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbf5800640 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35b8fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:15:56.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0076fc0 tx=0x7fcbdc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3c4fd43c4a351c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbe1600640 1 -- 172.21.15.46:0/3047031772 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc023020 con 0x560f35b7fea0 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbe1600640 1 -- 172.21.15.46:0/3047031772 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbdc010670 con 0x560f35b7fea0 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.419+0000 7fcbe1600640 1 -- 172.21.15.46:0/3047031772 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc003830 con 0x560f35b7fea0 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 -- 172.21.15.46:0/3047031772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 msgr2=0x560f35b8f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0076fc0 tx=0x7fcbdc0046c0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 -- 172.21.15.46:0/3047031772 shutdown_connections 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35b8fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7fea0 0x560f35b8f870 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3047031772 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7f580 0x560f35b88170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 -- 172.21.15.46:0/3047031772 >> 172.21.15.46:0/3047031772 conn(0x560f35b79cb0 msgr2=0x560f35b80d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 -- 172.21.15.46:0/3047031772 shutdown_connections 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.420+0000 7fcbfa426740 1 -- 172.21.15.46:0/3047031772 wait complete. 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 -- start start 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 Processor -- start 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35be9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35be2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b32f0 con 0x560f35b7fea0 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b45d0 con 0x560f35b88a80 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf01b3910 con 0x560f35b7f580 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.421+0000 7fcbfa426740 10 client.0 _ll_register_callbacks cb 0x7fcbe40f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35be9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.430 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbefe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35be9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:45162/0 (socket says 172.21.15.46:45162) 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:56840/0 (socket says 172.21.15.46:56840) 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbefe00640 1 -- 172.21.15.46:0/3336646408 learned_addr learned my addr 172.21.15.46:0/3336646408 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf5800640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35be2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 msgr2=0x560f35be2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35be2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3336646408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 msgr2=0x560f35be9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35be9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.431 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 -- 172.21.15.46:0/3336646408 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f35be36e0 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbf4e00640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40fbed0 tx=0x7fcbe40bb210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cacbdd90f1529f6c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.422+0000 7fcbd6a00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40ff070 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fcbe40bd160 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbdc010670 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbe40bc660 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 10 client.27292 fetch_fsmap learned FSMap version 31 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560f35be36e0 con 0x560f35b7f580 2024-03-08T23:15:56.432 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbdc010670 con 0x560f35b7f580 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcbe40c5010 con 0x560f35b7f580 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 10 client.27292 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 10 client.27292 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 20 client.27292 populate_metadata read hostname 'smithi046' 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35bf0720 0x560f35bf2b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 4 client.27292 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:15:56.433 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.423+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560f35aeb5e0 con 0x560f35bf0720 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35bf0720 0x560f35bf2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbefe00640 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35bf0720 0x560f35bf2b40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c7607fca03524fb server_cookie=762b8d5971ed69d4 in_seq=0 out_seq=0 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbd6a00640 10 client.27292 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbd6a00640 1 -- 172.21.15.46:0/3336646408 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560f35aeb5e0 con 0x560f35bf0720 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbd6a00640 10 client.27292 handle_command_reply: tid=0 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbfa426740 1 client.27292 shutdown 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 msgr2=0x560f35be8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.434 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.424+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fcbe40fbed0 tx=0x7fcbe40bb210 comp rx=0 tx=0).stop 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 shutdown_connections 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x560f35bf0720 0x560f35bf2b40 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x560f35b88a80 0x560f35be2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x560f35b7fea0 0x560f35be9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 --2- 172.21.15.46:0/3336646408 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x560f35b7f580 0x560f35be8be0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbd6a00640 0 client.27292 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 >> 172.21.15.46:0/3336646408 conn(0x560f35b79cb0 msgr2=0x560f35b85a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 shutdown_connections 2024-03-08T23:15:56.435 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 1 -- 172.21.15.46:0/3336646408 wait complete. 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbfa426740 20 client.27292 trim_cache size 0 max 16384 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout: "a7525ab6-e781-4d74-af0e-4427d46cc44b": { 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:15:56.436 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "a7525ab6-e781-4d74-af0e-4427d46cc44b", 2024-03-08T23:15:56.437 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:15:56.437 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:15:56.437 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:15:56.437 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 msgr2=0x7fcbd80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 0x7fcbd80a4eb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8005c10 tx=0x7fcbe8010040 comp rx=0 tx=0).stop 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 msgr2=0x7fcbf0075800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.425+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcbe4004590 tx=0x7fcbe40045c0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 shutdown_connections 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcbd80a29f0 0x7fcbd80a4eb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.438 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcbf0109570 0x7fcbf0076280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcbf0103e70 0x7fcbf0075d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 --2- 172.21.15.46:0/2585294619 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcbf01034b0 0x7fcbf0075800 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:15:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 >> 172.21.15.46:0/2585294619 conn(0x7fcbf00fef40 msgr2=0x7fcbf0100750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:15:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 shutdown_connections 2024-03-08T23:15:56.439 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:15:56.426+0000 7fcbf7800640 1 -- 172.21.15.46:0/2585294619 wait complete. 2024-03-08T23:15:56.443 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:56 smithi046 ceph-mon[31859]: pgmap v251: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:56.445 INFO:tasks.cephfs.filesystem:scrub status for tag:a7525ab6-e781-4d74-af0e-4427d46cc44b - {'path': '/', 'tag': 'a7525ab6-e781-4d74-af0e-4427d46cc44b', 'options': 'recursive,force'} 2024-03-08T23:15:56.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:56 smithi062 ceph-mon[35761]: pgmap v251: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:56.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:56 smithi192 ceph-mon[36700]: pgmap v251: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:57.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:57 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:15:57.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:57 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:57.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:57 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:15:58.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:58 smithi062 ceph-mon[35761]: pgmap v252: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:58.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:58 smithi046 ceph-mon[31859]: pgmap v252: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:58.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:58 smithi192 ceph-mon[36700]: pgmap v252: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:15:59.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:15:59 smithi062 ceph-mon[35761]: pgmap v253: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:59.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:15:59 smithi046 ceph-mon[31859]: pgmap v253: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:15:59.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:15:59 smithi192 ceph-mon[36700]: pgmap v253: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:02.037 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:01 smithi046 ceph-mon[31859]: pgmap v254: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:02.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:01 smithi192 ceph-mon[36700]: pgmap v254: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:02.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:01 smithi062 ceph-mon[35761]: pgmap v254: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:04.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:03 smithi192 ceph-mon[36700]: pgmap v255: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:16:04.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:03 smithi062 ceph-mon[35761]: pgmap v255: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:16:04.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:03 smithi046 ceph-mon[31859]: pgmap v255: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:16:06.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:05 smithi192 ceph-mon[36700]: pgmap v256: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:06.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:05 smithi062 ceph-mon[35761]: pgmap v256: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:06.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:05 smithi046 ceph-mon[31859]: pgmap v256: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:08.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:07 smithi192 ceph-mon[36700]: pgmap v257: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:08.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:07 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:16:08.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:07 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:08.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:07 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:07 smithi062 ceph-mon[35761]: pgmap v257: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:07 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:16:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:07 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:07 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:08.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:07 smithi046 ceph-mon[31859]: pgmap v257: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:16:08.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:07 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:16:08.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:07 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:08.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:07 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:09.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:08 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:09.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:08 smithi192 ceph-mon[36700]: osdmap e99: 12 total, 12 up, 12 in 2024-03-08T23:16:09.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:08 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:16:09.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:08 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:16:09.065 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:08 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:16:09.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:08 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:09.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:08 smithi062 ceph-mon[35761]: osdmap e99: 12 total, 12 up, 12 in 2024-03-08T23:16:09.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:08 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:16:09.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:08 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:16:09.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:08 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:16:09.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:08 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:09.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:08 smithi046 ceph-mon[31859]: osdmap e99: 12 total, 12 up, 12 in 2024-03-08T23:16:09.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:08 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:16:09.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:08 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:16:09.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:08 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:16:10.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:10 smithi046 ceph-mon[31859]: pgmap v259: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:10.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:10 smithi192 ceph-mon[36700]: pgmap v259: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:10.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:10 smithi062 ceph-mon[35761]: pgmap v259: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:12.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:12 smithi046 ceph-mon[31859]: pgmap v260: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:12.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:12 smithi192 ceph-mon[36700]: pgmap v260: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:12 smithi062 ceph-mon[35761]: pgmap v260: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:16:14.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:14 smithi046 ceph-mon[31859]: pgmap v261: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:16:14.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:14 smithi192 ceph-mon[36700]: pgmap v261: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:16:14.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:14 smithi062 ceph-mon[35761]: pgmap v261: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:16:15.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:15 smithi062 ceph-mon[35761]: osdmap e100: 12 total, 12 up, 12 in 2024-03-08T23:16:15.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:15 smithi046 ceph-mon[31859]: osdmap e100: 12 total, 12 up, 12 in 2024-03-08T23:16:15.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:15 smithi192 ceph-mon[36700]: osdmap e100: 12 total, 12 up, 12 in 2024-03-08T23:16:16.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:16 smithi062 ceph-mon[35761]: pgmap v263: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:16:16.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:16 smithi046 ceph-mon[31859]: pgmap v263: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:16:16.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:16 smithi192 ceph-mon[36700]: pgmap v263: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:16:18.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:18 smithi062 ceph-mon[35761]: pgmap v264: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-08T23:16:18.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:18 smithi046 ceph-mon[31859]: pgmap v264: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-08T23:16:18.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:18 smithi192 ceph-mon[36700]: pgmap v264: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 114 B/s wr, 3 op/s 2024-03-08T23:16:20.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:20 smithi062 ceph-mon[35761]: pgmap v265: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:20.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:20 smithi046 ceph-mon[31859]: pgmap v265: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:20.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:20 smithi192 ceph-mon[36700]: pgmap v265: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:22.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:22 smithi062 ceph-mon[35761]: pgmap v266: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:22.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:22 smithi046 ceph-mon[31859]: pgmap v266: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:22.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:22 smithi192 ceph-mon[36700]: pgmap v266: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:16:24.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:24 smithi062 ceph-mon[35761]: pgmap v267: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:16:24.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:24 smithi046 ceph-mon[31859]: pgmap v267: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:16:24.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:24 smithi192 ceph-mon[36700]: pgmap v267: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:16:26.446 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:16:26.447 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:16:26.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:26 smithi062 ceph-mon[35761]: pgmap v268: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:16:26.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.684+0000 7f4061800640 1 -- 172.21.15.46:0/2203100492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 msgr2=0x7f405c114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.684+0000 7f4061800640 1 --2- 172.21.15.46:0/2203100492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c114520 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4054009930 tx=0x7f405402bc20 comp rx=0 tx=0).stop 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 -- 172.21.15.46:0/2203100492 shutdown_connections 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 --2- 172.21.15.46:0/2203100492 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c114520 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 --2- 172.21.15.46:0/2203100492 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 0x7f405c111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 --2- 172.21.15.46:0/2203100492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 -- 172.21.15.46:0/2203100492 >> 172.21.15.46:0/2203100492 conn(0x7f405c0fefa0 msgr2=0x7f405c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 -- 172.21.15.46:0/2203100492 shutdown_connections 2024-03-08T23:16:26.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.685+0000 7f4061800640 1 -- 172.21.15.46:0/2203100492 wait complete. 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.686+0000 7f4061800640 1 Processor -- start 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.686+0000 7f4061800640 1 -- start start 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c19e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 0x7f405c19e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c117ce0 con 0x7f405c076140 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c117b60 con 0x7f405c112130 2024-03-08T23:16:26.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f4061800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c117e60 con 0x7f405c075820 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 0x7f405c19e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c19e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44644/0 (socket says 172.21.15.46:44644) 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 -- 172.21.15.46:0/3809716920 learned_addr learned my addr 172.21.15.46:0/3809716920 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 -- 172.21.15.46:0/3809716920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 msgr2=0x7f405c19e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c19e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 -- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 msgr2=0x7f405c19e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 0x7f405c19e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405b400640 1 -- 172.21.15.46:0/3809716920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c1a3210 con 0x7f405c112130 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.687+0000 7f405aa00640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c19e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f405b400640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4054005ab0 tx=0x7f40540042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f4053400640 1 -- 172.21.15.46:0/3809716920 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054043020 con 0x7f405c112130 2024-03-08T23:16:26.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f4053400640 1 -- 172.21.15.46:0/3809716920 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4054005000 con 0x7f405c112130 2024-03-08T23:16:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f4053400640 1 -- 172.21.15.46:0/3809716920 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4054003af0 con 0x7f405c112130 2024-03-08T23:16:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c1a34a0 con 0x7f405c112130 2024-03-08T23:16:26.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.688+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c1aad90 con 0x7f405c112130 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.691+0000 7f4053400640 1 -- 172.21.15.46:0/3809716920 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4054046020 con 0x7f405c112130 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.691+0000 7f4064626740 1 Processor -- start 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 -- start start 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d98fd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9906540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9908e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d98bf410 con 0x5570d98fd580 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9861d50 con 0x5570d98fdea0 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f4064626740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d980ab80 con 0x5570d9906a80 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f405a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d98fd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f405aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9906540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f405a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d98fd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38886/0 (socket says 172.21.15.46:38886) 2024-03-08T23:16:26.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f405a000640 1 -- 172.21.15.46:0/3501090505 learned_addr learned my addr 172.21.15.46:0/3501090505 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.692+0000 7f405a000640 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9908e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 -- 172.21.15.46:0/3501090505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 msgr2=0x5570d9908e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9908e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 -- 172.21.15.46:0/3501090505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 msgr2=0x5570d98fd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d98fd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 -- 172.21.15.46:0/3501090505 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d96407d0 con 0x5570d98fdea0 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f4053400640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 0x7f40400a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405aa00640 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9906540 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f404c005fd0 tx=0x7f404c011c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=714e3eb4cab67bca server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f4053400640 1 -- 172.21.15.46:0/3809716920 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f40540f0f60 con 0x7f405c112130 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f405a000640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 0x7f40400a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.693+0000 7f404be00640 1 -- 172.21.15.46:0/3501090505 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c004d20 con 0x5570d98fdea0 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f404be00640 1 -- 172.21.15.46:0/3501090505 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f404c00f9a0 con 0x5570d98fdea0 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f404be00640 1 -- 172.21.15.46:0/3501090505 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c0044a0 con 0x5570d98fdea0 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 -- 172.21.15.46:0/3501090505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 msgr2=0x5570d9906540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9906540 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f404c005fd0 tx=0x7f404c011c70 comp rx=0 tx=0).stop 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f405a000640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 0x7f40400a4eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x5570d98f8e40 tx=0x7f4044015040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 -- 172.21.15.46:0/3501090505 shutdown_connections 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9908e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9906540 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.694+0000 7f4064626740 1 --2- 172.21.15.46:0/3501090505 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d98fd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.695+0000 7f4064626740 1 -- 172.21.15.46:0/3501090505 >> 172.21.15.46:0/3501090505 conn(0x5570d98f7cb0 msgr2=0x5570d98f8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.695+0000 7f4064626740 1 -- 172.21.15.46:0/3501090505 shutdown_connections 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.695+0000 7f4064626740 1 -- 172.21.15.46:0/3501090505 wait complete. 2024-03-08T23:16:26.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.695+0000 7f4064626740 1 -- start start 2024-03-08T23:16:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.695+0000 7f4064626740 1 Processor -- start 2024-03-08T23:16:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d9903a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9904500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d981dee0 con 0x5570d98fd580 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d985ddf0 con 0x5570d98fdea0 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5570d9909870 con 0x5570d9906a80 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f4064626740 10 client.0 _ll_register_callbacks cb 0x5570d98248d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d9903a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44654/0 (socket says 172.21.15.46:44654) 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d9903a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38890/0 (socket says 172.21.15.46:38890) 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405a000640 1 -- 172.21.15.46:0/1043724469 learned_addr learned my addr 172.21.15.46:0/1043724469 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.696+0000 7f405b400640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9904500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 -- 172.21.15.46:0/1043724469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 msgr2=0x5570d9904500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9904500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 -- 172.21.15.46:0/1043724469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 msgr2=0x5570d9903a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d9903a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 -- 172.21.15.46:0/1043724469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570d95763f0 con 0x5570d98fdea0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405b400640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9904500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f405a000640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f404400f9d0 tx=0x7f4044005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cb6447291b70159 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f4048c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4044021b90 con 0x5570d98fdea0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4044021d30 con 0x5570d98fdea0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.697+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f404c00f9a0 con 0x5570d98fdea0 2024-03-08T23:16:26.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4044036050 con 0x5570d98fdea0 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 10 client.27347 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5570d95360f0 con 0x5570d98fdea0 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f404c00f9a0 con 0x5570d98fdea0 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f404402a450 con 0x5570d98fdea0 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 10 client.27347 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 10 client.27347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 20 client.27347 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:26.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d9916c70 0x5570d9919080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 4 client.27347 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.698+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5570d986f290 con 0x5570d9916c70 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.699+0000 7f405aa00640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d9916c70 0x5570d9919080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.699+0000 7f405aa00640 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d9916c70 0x5570d9919080 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37e77e881eaae7f4 server_cookie=53f2ded60c3d4891 in_seq=0 out_seq=0 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.699+0000 7f4048c00640 10 client.27347 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4048c00640 1 -- 172.21.15.46:0/1043724469 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5570d986f290 con 0x5570d9916c70 2024-03-08T23:16:26.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4048c00640 10 client.27347 handle_command_reply: tid=0 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 client.27347 shutdown 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 msgr2=0x5570d9903fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f404400f9d0 tx=0x7f4044005f90 comp rx=0 tx=0).stop 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 shutdown_connections 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d9916c70 0x5570d9919080 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d9906a80 0x5570d9904500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d98fdea0 0x5570d9903fc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4048c00640 0 client.27347 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:26.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 --2- 172.21.15.46:0/1043724469 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d9903a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.700+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 >> 172.21.15.46:0/1043724469 conn(0x5570d98f7cb0 msgr2=0x5570d98f6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.701+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 shutdown_connections 2024-03-08T23:16:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.701+0000 7f4064626740 1 -- 172.21.15.46:0/1043724469 wait complete. 2024-03-08T23:16:26.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.701+0000 7f4064626740 20 client.27347 trim_cache size 0 max 16384 2024-03-08T23:16:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 Processor -- start 2024-03-08T23:16:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 -- start start 2024-03-08T23:16:26.720 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 0x5570d9919060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d990dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b7130 con 0x5570d98fd580 2024-03-08T23:16:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b4cd0 con 0x5570d9906a80 2024-03-08T23:16:26.721 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f4064626740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b2d90 con 0x5570d98fdea0 2024-03-08T23:16:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f405aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.718+0000 7f405b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d990dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.723 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d990dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44670/0 (socket says 172.21.15.46:44670) 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38892/0 (socket says 172.21.15.46:38892) 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405b400640 1 -- 172.21.15.46:0/365876082 learned_addr learned my addr 172.21.15.46:0/365876082 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405a000640 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 0x5570d9919060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 -- 172.21.15.46:0/365876082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 msgr2=0x5570d9919060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 0x5570d9919060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 -- 172.21.15.46:0/365876082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 msgr2=0x5570d990dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.724 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d990dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 -- 172.21.15.46:0/365876082 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5570d9908e70 con 0x5570d98fd580 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405a000640 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 0x5570d9919060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.719+0000 7f405aa00640 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f404c005c10 tx=0x7f404c004020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0f271ad1657579 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f404b400640 1 -- 172.21.15.46:0/365876082 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00fcd0 con 0x5570d98fd580 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f404b400640 1 -- 172.21.15.46:0/365876082 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f404c02e030 con 0x5570d98fd580 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f404b400640 1 -- 172.21.15.46:0/365876082 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c005490 con 0x5570d98fd580 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f4064626740 1 -- 172.21.15.46:0/365876082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 msgr2=0x5570d99061b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.725 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f4064626740 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f404c005c10 tx=0x7f404c004020 comp rx=0 tx=0).stop 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.720+0000 7f4064626740 1 -- 172.21.15.46:0/365876082 shutdown_connections 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d990dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fdea0 0x5570d9919060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 --2- 172.21.15.46:0/365876082 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fd580 0x5570d99061b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 -- 172.21.15.46:0/365876082 >> 172.21.15.46:0/365876082 conn(0x5570d98f7cb0 msgr2=0x5570d98fed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 -- 172.21.15.46:0/365876082 shutdown_connections 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 -- 172.21.15.46:0/365876082 wait complete. 2024-03-08T23:16:26.726 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 -- start start 2024-03-08T23:16:26.727 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.721+0000 7f4064626740 1 Processor -- start 2024-03-08T23:16:26.727 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:26 smithi046 ceph-mon[31859]: pgmap v268: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:16:26.730 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fd580 0x5570d9966d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d9961150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b3390 con 0x5570d98fdea0 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b4670 con 0x5570d9906a80 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c1b39b0 con 0x5570d98fd580 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f4064626740 10 client.0 _ll_register_callbacks cb 0x7f404c009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f405a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f405a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38908/0 (socket says 172.21.15.46:38908) 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f405a000640 1 -- 172.21.15.46:0/2244205865 learned_addr learned my addr 172.21.15.46:0/2244205865 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f405b400640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d9961150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.722+0000 7f405aa00640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fd580 0x5570d9966d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 -- 172.21.15.46:0/2244205865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fd580 msgr2=0x5570d9966d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fd580 0x5570d9966d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 -- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 msgr2=0x5570d9961150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d9961150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 -- 172.21.15.46:0/2244205865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5570d9961920 con 0x5570d98fdea0 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405b400640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d9961150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f405a000640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f40440058c0 tx=0x7f404402a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=810f5066a494010e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f4048c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404403e070 con 0x5570d98fdea0 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f404402ac70 con 0x5570d98fdea0 2024-03-08T23:16:26.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.723+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f404c02e030 con 0x5570d98fdea0 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f404402ae70 con 0x5570d98fdea0 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 10 client.17190 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5570d996e540 con 0x5570d98fdea0 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f404c02e030 con 0x5570d98fdea0 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4044004730 con 0x5570d98fdea0 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 10 client.17190 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:26.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 20 client.17190 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d996eca0 0x5570d99710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 4 client.17190 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.724+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5570d98695e0 con 0x5570d996eca0 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f405aa00640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d996eca0 0x5570d99710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f405aa00640 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d996eca0 0x5570d99710c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b5541f9876bb37 server_cookie=a26ea2cc914c686b in_seq=0 out_seq=0 2024-03-08T23:16:26.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f4048c00640 10 client.17190 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f4048c00640 1 -- 172.21.15.46:0/2244205865 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5570d98695e0 con 0x5570d996eca0 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f4048c00640 10 client.17190 handle_command_reply: tid=0 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.725+0000 7f4064626740 1 client.17190 shutdown 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 msgr2=0x5570d99672b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f40440058c0 tx=0x7f404402a650 comp rx=0 tx=0).stop 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 shutdown_connections 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5570d996eca0 0x5570d99710c0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.736 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5570d9906a80 0x5570d9961150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5570d98fdea0 0x5570d99672b0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 --2- 172.21.15.46:0/2244205865 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5570d98fd580 0x5570d9966d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4048c00640 0 client.17190 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 >> 172.21.15.46:0/2244205865 conn(0x5570d98f7cb0 msgr2=0x5570d98f6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 shutdown_connections 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 1 -- 172.21.15.46:0/2244205865 wait complete. 2024-03-08T23:16:26.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.726+0000 7f4064626740 20 client.17190 trim_cache size 0 max 16384 2024-03-08T23:16:26.738 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:16:26.738 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:16:26.738 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:16:26.738 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 msgr2=0x7f40400a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 0x7f40400a4eb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x5570d98f8e40 tx=0x7f4044015040 comp rx=0 tx=0).stop 2024-03-08T23:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 msgr2=0x7f405c1a2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f4054005ab0 tx=0x7f40540042e0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 shutdown_connections 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40400a29f0 0x7f40400a4eb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f405c112130 0x7f405c1a2ae0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f405c076140 0x7f405c19e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 --2- 172.21.15.46:0/3809716920 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f405c075820 0x7f405c19e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 >> 172.21.15.46:0/3809716920 conn(0x7f405c0fefa0 msgr2=0x7f405c101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 shutdown_connections 2024-03-08T23:16:26.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.727+0000 7f4061800640 1 -- 172.21.15.46:0/3809716920 wait complete. 2024-03-08T23:16:26.748 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:16:26.748 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:16:26.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:26 smithi192 ceph-mon[36700]: pgmap v268: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:16:26.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 -- 172.21.15.46:0/2429947114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae8101320 msgr2=0x7fcae8101700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 --2- 172.21.15.46:0/2429947114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae8101320 0x7fcae8101700 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009860 tx=0x7fcad802bb70 comp rx=0 tx=0).stop 2024-03-08T23:16:26.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 -- 172.21.15.46:0/2429947114 shutdown_connections 2024-03-08T23:16:26.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 --2- 172.21.15.46:0/2429947114 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae81063f0 0x7fcae810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 --2- 172.21.15.46:0/2429947114 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101cd0 0x7fcae8105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 --2- 172.21.15.46:0/2429947114 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae8101320 0x7fcae8101700 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 -- 172.21.15.46:0/2429947114 >> 172.21.15.46:0/2429947114 conn(0x7fcae80fcbc0 msgr2=0x7fcae80fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:26.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 -- 172.21.15.46:0/2429947114 shutdown_connections 2024-03-08T23:16:26.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.968+0000 7fcae7200640 1 -- 172.21.15.46:0/2429947114 wait complete. 2024-03-08T23:16:26.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.969+0000 7fcae7200640 1 Processor -- start 2024-03-08T23:16:26.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.969+0000 7fcae7200640 1 -- start start 2024-03-08T23:16:26.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 0x7fcae819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae8199a60 0x7fcae8199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae81114a0 con 0x7fcae8199a60 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae8111320 con 0x7fcae8101320 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae7200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae8111620 con 0x7fcae81063f0 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae8199a60 0x7fcae8199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 0x7fcae819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44696/0 (socket says 172.21.15.46:44696) 2024-03-08T23:16:26.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.970+0000 7fcae5e00640 1 -- 172.21.15.46:0/1458576460 learned_addr learned my addr 172.21.15.46:0/1458576460 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:26.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 -- 172.21.15.46:0/1458576460 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 msgr2=0x7fcae819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 0x7fcae819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 -- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae8199a60 msgr2=0x7fcae8199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae8199a60 0x7fcae8199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 -- 172.21.15.46:0/1458576460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae819a580 con 0x7fcae8101320 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5400640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 0x7fcae819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae5e00640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcad8005e00 tx=0x7fcad80045d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8041020 con 0x7fcae8101320 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae819a810 con 0x7fcae8101320 2024-03-08T23:16:26.977 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcad802c9f0 con 0x7fcae8101320 2024-03-08T23:16:26.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8039040 con 0x7fcae8101320 2024-03-08T23:16:26.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.971+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae81a6a60 con 0x7fcae8101320 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.972+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae8068450 con 0x7fcae8101320 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.975+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcad8044020 con 0x7fcae8101320 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.976+0000 7fcadea00640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 0x7fcacc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.976+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fcad80f0cc0 con 0x7fcae8101320 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.976+0000 7fcae5400640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 0x7fcacc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:26.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.977+0000 7fcae5400640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 0x7fcacc0a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcae8100700 tx=0x7fcad4005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:26.980 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:26.978+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcad804b020 con 0x7fcae8101320 2024-03-08T23:16:27.226 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.224+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcae8101cd0 con 0x7fcae8101320 2024-03-08T23:16:27.227 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.225+0000 7fcadea00640 1 -- 172.21.15.46:0/1458576460 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fcad80bd710 con 0x7fcae8101320 2024-03-08T23:16:27.227 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:16:27.228 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:16:27.228 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:16:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 msgr2=0x7fcacc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 0x7fcacc0a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fcae8100700 tx=0x7fcad4005f90 comp rx=0 tx=0).stop 2024-03-08T23:16:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 msgr2=0x7fcae819f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fcad8005e00 tx=0x7fcad80045d0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.230 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 shutdown_connections 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fcacc0a2970 0x7fcacc0a4e30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcae8199a60 0x7fcae8199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcae81063f0 0x7fcae819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 --2- 172.21.15.46:0/1458576460 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcae8101320 0x7fcae819f8b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 >> 172.21.15.46:0/1458576460 conn(0x7fcae80fcbc0 msgr2=0x7fcae80feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 shutdown_connections 2024-03-08T23:16:27.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.228+0000 7fcae7200640 1 -- 172.21.15.46:0/1458576460 wait complete. 2024-03-08T23:16:27.251 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:16:27.251 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:16:27.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 -- 172.21.15.46:0/3944811628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624102c40 msgr2=0x7f262410b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 --2- 172.21.15.46:0/3944811628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624102c40 0x7f262410b120 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2610009860 tx=0x7f261002bc10 comp rx=0 tx=0).stop 2024-03-08T23:16:27.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 -- 172.21.15.46:0/3944811628 shutdown_connections 2024-03-08T23:16:27.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 --2- 172.21.15.46:0/3944811628 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f262410b740 0x7f262410dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.477 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 --2- 172.21.15.46:0/3944811628 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2624102c40 0x7f262410b120 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 --2- 172.21.15.46:0/3944811628 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102320 0x7f2624102700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.474+0000 7f2623200640 1 -- 172.21.15.46:0/3944811628 >> 172.21.15.46:0/3944811628 conn(0x7f26240fbb40 msgr2=0x7f26240fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.475+0000 7f2623200640 1 -- 172.21.15.46:0/3944811628 shutdown_connections 2024-03-08T23:16:27.478 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.475+0000 7f2623200640 1 -- 172.21.15.46:0/3944811628 wait complete. 2024-03-08T23:16:27.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.476+0000 7f2623200640 1 Processor -- start 2024-03-08T23:16:27.479 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.476+0000 7f2623200640 1 -- start start 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2624102320 0x7f262419d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f262410b740 0x7f2624197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2624111500 con 0x7f2624102c40 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2624111380 con 0x7f262410b740 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2623200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2624111680 con 0x7f2624102320 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2621400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2621400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38946/0 (socket says 172.21.15.46:38946) 2024-03-08T23:16:27.480 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2621400640 1 -- 172.21.15.46:0/3521919627 learned_addr learned my addr 172.21.15.46:0/3521919627 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2622800640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f262410b740 0x7f2624197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.481 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.477+0000 7f2621e00640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2624102320 0x7f262419d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 -- 172.21.15.46:0/3521919627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2624102320 msgr2=0x7f262419d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2624102320 0x7f262419d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 -- 172.21.15.46:0/3521919627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f262410b740 msgr2=0x7f2624197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f262410b740 0x7f2624197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 -- 172.21.15.46:0/3521919627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624197fe0 con 0x7f2624102c40 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2621400640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2610037040 tx=0x7f26100344c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.482 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f261aa00640 1 -- 172.21.15.46:0/3521919627 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610041020 con 0x7f2624102c40 2024-03-08T23:16:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2624198270 con 0x7f2624102c40 2024-03-08T23:16:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f261aa00640 1 -- 172.21.15.46:0/3521919627 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f261002ca90 con 0x7f2624102c40 2024-03-08T23:16:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.478+0000 7f261aa00640 1 -- 172.21.15.46:0/3521919627 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2610039040 con 0x7f2624102c40 2024-03-08T23:16:27.483 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.479+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26241a4570 con 0x7f2624102c40 2024-03-08T23:16:27.484 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.482+0000 7f262bb6e740 1 Processor -- start 2024-03-08T23:16:27.485 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 -- start start 2024-03-08T23:16:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b73900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b73e40 0x561115b76230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.486 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115acadd0 con 0x561115b6a940 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115a73c70 con 0x561115b73e40 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f262bb6e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115ac6e70 con 0x561115b6b260 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f2622800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f2622800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38950/0 (socket says 172.21.15.46:38950) 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f2622800640 1 -- 172.21.15.46:0/1193498220 learned_addr learned my addr 172.21.15.46:0/1193498220 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.483+0000 7f2621e00640 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b73900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 -- 172.21.15.46:0/1193498220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 msgr2=0x561115b73900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.487 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b73900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 -- 172.21.15.46:0/1193498220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b73e40 msgr2=0x561115b76230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-08T23:16:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b73e40 0x561115b76230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 -- 172.21.15.46:0/1193498220 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561115b2d2b0 con 0x561115b6a940 2024-03-08T23:16:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f261aa00640 1 -- 172.21.15.46:0/3521919627 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2610044020 con 0x7f2624102c40 2024-03-08T23:16:27.488 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f2622800640 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f261c009f50 tx=0x7f261c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b83e17bd082b441 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.490 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.484+0000 7f260f400640 1 -- 172.21.15.46:0/1193498220 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c004d20 con 0x561115b6a940 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.485+0000 7f260f400640 1 -- 172.21.15.46:0/1193498220 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f261c004050 con 0x561115b6a940 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.485+0000 7f260f400640 1 -- 172.21.15.46:0/1193498220 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c005280 con 0x561115b6a940 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.485+0000 7f262bb6e740 1 -- 172.21.15.46:0/1193498220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 msgr2=0x561115b6ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.485+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f261c009f50 tx=0x7f261c0118d0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.485+0000 7f261aa00640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 0x7f26080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f261aa00640 1 -- 172.21.15.46:0/3521919627 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f26100f0750 con 0x7f2624102c40 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 -- 172.21.15.46:0/1193498220 shutdown_connections 2024-03-08T23:16:27.491 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b73e40 0x561115b76230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b73900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1193498220 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6a940 0x561115b6ad20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 -- 172.21.15.46:0/1193498220 >> 172.21.15.46:0/1193498220 conn(0x561115b65070 msgr2=0x561115b65450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f2622800640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 0x7f26080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 -- 172.21.15.46:0/1193498220 shutdown_connections 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.486+0000 7f262bb6e740 1 -- 172.21.15.46:0/1193498220 wait complete. 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 -- start start 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f2622800640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 0x7f26080a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f261c005fd0 tx=0x7f261c004700 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.492 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 Processor -- start 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b70bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b711c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115b76c30 con 0x561115b73e40 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115a870c0 con 0x561115b6a940 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.487+0000 7f262bb6e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x561115b76db0 con 0x561115b6b260 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f262bb6e740 10 client.0 _ll_register_callbacks cb 0x561115ac1ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:27.493 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44718/0 (socket says 172.21.15.46:44718) 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 -- 172.21.15.46:0/1135303142 learned_addr learned my addr 172.21.15.46:0/1135303142 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621400640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b70bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2622800640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b711c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 -- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 msgr2=0x561115b70bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b70bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 -- 172.21.15.46:0/1135303142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 msgr2=0x561115b711c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b711c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.494 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2621e00640 1 -- 172.21.15.46:0/1135303142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5611157df3f0 con 0x561115b6a940 2024-03-08T23:16:27.495 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.488+0000 7f2622800640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b711c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f2621e00640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f26140116d0 tx=0x7f2614011ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28123797e7b78657 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f2603e00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:16:27.497 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261401e060 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f261c004050 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f2614004020 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2614025050 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f262bb6e740 10 client.27368 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.489+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5611158a97d0 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f261c004050 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2614019070 con 0x561115b6a940 2024-03-08T23:16:27.498 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 10 client.27368 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 10 client.27368 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 20 client.27368 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115b83f40 0x561115b86330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 4 client.27368 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.490+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561115a9f4f0 con 0x561115b83f40 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.491+0000 7f2621400640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115b83f40 0x561115b86330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.491+0000 7f2621400640 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115b83f40 0x561115b86330 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88ebab56accf227e server_cookie=1cb0732695eedf1 in_seq=0 out_seq=0 2024-03-08T23:16:27.499 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.491+0000 7f2603e00640 10 client.27368 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f2603e00640 1 -- 172.21.15.46:0/1135303142 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561115a9f4f0 con 0x561115b83f40 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f2603e00640 10 client.27368 handle_command_reply: tid=0 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 client.27368 shutdown 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 msgr2=0x561115b73820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f26140116d0 tx=0x7f2614011ad0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 shutdown_connections 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115b83f40 0x561115b86330 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.500 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b711c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6b260 0x561115b70bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f2603e00640 0 client.27368 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 --2- 172.21.15.46:0/1135303142 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b73820 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 >> 172.21.15.46:0/1135303142 conn(0x561115b65070 msgr2=0x561115b66dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 shutdown_connections 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 1 -- 172.21.15.46:0/1135303142 wait complete. 2024-03-08T23:16:27.501 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.492+0000 7f262bb6e740 20 client.27368 trim_cache size 0 max 16384 2024-03-08T23:16:27.508 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 Processor -- start 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 -- start start 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b6fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 0x561115b73820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241ae7b0 con 0x561115b6b260 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241ac350 con 0x561115b6a940 2024-03-08T23:16:27.509 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.506+0000 7f262bb6e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241aa410 con 0x561115b73e40 2024-03-08T23:16:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38966/0 (socket says 172.21.15.46:38966) 2024-03-08T23:16:27.512 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 -- 172.21.15.46:0/3911575007 learned_addr learned my addr 172.21.15.46:0/3911575007 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621e00640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b6fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2622800640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 0x561115b73820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 -- 172.21.15.46:0/3911575007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 msgr2=0x561115b73820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 0x561115b73820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 -- 172.21.15.46:0/3911575007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 msgr2=0x561115b6fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b6fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2621400640 1 -- 172.21.15.46:0/3911575007 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561115b866a0 con 0x561115b6b260 2024-03-08T23:16:27.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.507+0000 7f2622800640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 0x561115b73820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f2621400640 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26100bc8b0 tx=0x7f2610005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79327dea5f0929d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f260ea00640 1 -- 172.21.15.46:0/3911575007 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100bcbb0 con 0x561115b6b260 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f260ea00640 1 -- 172.21.15.46:0/3911575007 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26100c0c50 con 0x561115b6b260 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f260ea00640 1 -- 172.21.15.46:0/3911575007 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100b93b0 con 0x561115b6b260 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f262bb6e740 1 -- 172.21.15.46:0/3911575007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 msgr2=0x561115b732e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.508+0000 7f262bb6e740 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26100bc8b0 tx=0x7f2610005dc0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 -- 172.21.15.46:0/3911575007 shutdown_connections 2024-03-08T23:16:27.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b73e40 0x561115b73820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b6b260 0x561115b732e0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 --2- 172.21.15.46:0/3911575007 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6a940 0x561115b6fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 -- 172.21.15.46:0/3911575007 >> 172.21.15.46:0/3911575007 conn(0x561115b65070 msgr2=0x561115b6f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 -- 172.21.15.46:0/3911575007 shutdown_connections 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 -- 172.21.15.46:0/3911575007 wait complete. 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 -- start start 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.509+0000 7f262bb6e740 1 Processor -- start 2024-03-08T23:16:27.515 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6b260 0x561115b795e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b79b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241aaa10 con 0x561115b73e40 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241abcf0 con 0x561115b6b260 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26241ab030 con 0x561115b6a940 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f262bb6e740 10 client.0 _ll_register_callbacks cb 0x7f261c009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2622800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b79b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2621e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.516 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2621e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52204/0 (socket says 172.21.15.46:52204) 2024-03-08T23:16:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2621e00640 1 -- 172.21.15.46:0/306265127 learned_addr learned my addr 172.21.15.46:0/306265127 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2622800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b79b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38970/0 (socket says 172.21.15.46:38970) 2024-03-08T23:16:27.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.510+0000 7f2621400640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6b260 0x561115b795e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 -- 172.21.15.46:0/306265127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6b260 msgr2=0x561115b795e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6b260 0x561115b795e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 -- 172.21.15.46:0/306265127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 msgr2=0x561115b79b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b79b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 -- 172.21.15.46:0/306265127 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561115b7a0e0 con 0x561115b6a940 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2621e00640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2614000c00 tx=0x7f2614009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fc062c0a025b103 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.520 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2603e00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400f910 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f261400fab0 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.511+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26100c0c50 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f261403a050 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 10 client.27331 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x561115bdb110 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26100c0c50 con 0x561115b6a940 2024-03-08T23:16:27.521 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2614019040 con 0x561115b6a940 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 10 client.27331 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 10 client.27331 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 20 client.27331 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115bdb780 0x561115bddba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 4 client.27331 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561115ad8310 con 0x561115bdb780 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.512+0000 7f2621400640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115bdb780 0x561115bddba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f2621400640 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115bdb780 0x561115bddba0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e8c5bc31f879bd5b server_cookie=84635fe01ecbaa17 in_seq=0 out_seq=0 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f2603e00640 10 client.27331 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f2603e00640 1 -- 172.21.15.46:0/306265127 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561115ad8310 con 0x561115bdb780 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f2603e00640 10 client.27331 handle_command_reply: tid=0 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 client.27331 shutdown 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 msgr2=0x561115b7b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f2614000c00 tx=0x7f2614009fd0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 shutdown_connections 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x561115bdb780 0x561115bddba0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x561115b73e40 0x561115b79b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x561115b6b260 0x561115b795e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 --2- 172.21.15.46:0/306265127 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x561115b6a940 0x561115b7b130 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f2603e00640 0 client.27331 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.513+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 >> 172.21.15.46:0/306265127 conn(0x561115b65070 msgr2=0x561115b5f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 shutdown_connections 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f262bb6e740 1 -- 172.21.15.46:0/306265127 wait complete. 2024-03-08T23:16:27.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f262bb6e740 20 client.27331 trim_cache size 0 max 16384 2024-03-08T23:16:27.525 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:16:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 msgr2=0x7f26080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 0x7f26080a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f261c005fd0 tx=0x7f261c004700 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 msgr2=0x7f262419dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2610037040 tx=0x7f26100344c0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 shutdown_connections 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f26080a2970 0x7f26080a4e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f262410b740 0x7f2624197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2624102c40 0x7f262419dbf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 --2- 172.21.15.46:0/3521919627 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2624102320 0x7f262419d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.514+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 >> 172.21.15.46:0/3521919627 conn(0x7f26240fbb40 msgr2=0x7f26240fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.515+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 shutdown_connections 2024-03-08T23:16:27.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.515+0000 7f2623200640 1 -- 172.21.15.46:0/3521919627 wait complete. 2024-03-08T23:16:27.532 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:27 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1458576460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:16:27.536 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:16:27.536 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:16:27.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:27 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1458576460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:16:27.764 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 -- 172.21.15.46:0/3690774331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0106490 msgr2=0x7ff1e010e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 --2- 172.21.15.46:0/3690774331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0106490 0x7ff1e010e060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8009900 tx=0x7ff1d802bc30 comp rx=0 tx=0).stop 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 -- 172.21.15.46:0/3690774331 shutdown_connections 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 --2- 172.21.15.46:0/3690774331 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0106490 0x7ff1e010e060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 --2- 172.21.15.46:0/3690774331 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e0101d70 0x7ff1e0105db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 --2- 172.21.15.46:0/3690774331 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e01013c0 0x7ff1e01017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 -- 172.21.15.46:0/3690774331 >> 172.21.15.46:0/3690774331 conn(0x7ff1e00fcc60 msgr2=0x7ff1e00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.765 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 -- 172.21.15.46:0/3690774331 shutdown_connections 2024-03-08T23:16:27.766 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.762+0000 7ff1e6000640 1 -- 172.21.15.46:0/3690774331 wait complete. 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.763+0000 7ff1e6000640 1 Processor -- start 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.763+0000 7ff1e6000640 1 -- start start 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 0x7ff1e019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 0x7ff1e0199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0111580 con 0x7ff1e0106490 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0111400 con 0x7ff1e01013c0 2024-03-08T23:16:27.768 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e6000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e0111700 con 0x7ff1e0199ab0 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 0x7ff1e019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:38996/0 (socket says 172.21.15.46:38996) 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1dfe00640 1 -- 172.21.15.46:0/505036532 learned_addr learned my addr 172.21.15.46:0/505036532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e5600640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 0x7ff1e0199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 -- 172.21.15.46:0/505036532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 msgr2=0x7ff1e0199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 0x7ff1e0199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.769 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 -- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 msgr2=0x7ff1e019fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e4c00640 1 -- 172.21.15.46:0/505036532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e019a5d0 con 0x7ff1e01013c0 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1e5600640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 0x7ff1e0199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.764+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 0x7ff1e019f900 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000eb00 tx=0x7ff1d000c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1dd600640 1 -- 172.21.15.46:0/505036532 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0010430 con 0x7ff1e01013c0 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1dd600640 1 -- 172.21.15.46:0/505036532 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d0010ab0 con 0x7ff1e01013c0 2024-03-08T23:16:27.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e019a8c0 con 0x7ff1e01013c0 2024-03-08T23:16:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1dd600640 1 -- 172.21.15.46:0/505036532 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d0017bb0 con 0x7ff1e01013c0 2024-03-08T23:16:27.771 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.765+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e01a6b60 con 0x7ff1e01013c0 2024-03-08T23:16:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.768+0000 7ff1dd600640 1 -- 172.21.15.46:0/505036532 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff1d0010c60 con 0x7ff1e01013c0 2024-03-08T23:16:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.768+0000 7ff1ea405740 1 Processor -- start 2024-03-08T23:16:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 -- start start 2024-03-08T23:16:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 0x5599a4868910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.773 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4871a30 0x5599a4873e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a47ccdd0 con 0x5599a4868e50 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a4775c70 con 0x5599a4868530 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1ea405740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a47c8e70 con 0x5599a4871a30 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.769+0000 7ff1e4c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e4c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39012/0 (socket says 172.21.15.46:39012) 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 0x5599a4868910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.774 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e4c00640 1 -- 172.21.15.46:0/2344045522 learned_addr learned my addr 172.21.15.46:0/2344045522 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4871a30 0x5599a4873e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 -- 172.21.15.46:0/2344045522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4871a30 msgr2=0x5599a4873e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4871a30 0x5599a4873e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 -- 172.21.15.46:0/2344045522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 msgr2=0x5599a48714f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e5600640 1 -- 172.21.15.46:0/2344045522 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599a482f2b0 con 0x5599a4868530 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:27.775 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1dd600640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 0x7ff1c40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.770+0000 7ff1dd600640 1 -- 172.21.15.46:0/505036532 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff1d00cfc10 con 0x7ff1e01013c0 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.771+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 0x7ff1c40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.771+0000 7ff1e5600640 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 0x5599a4868910 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8036040 tx=0x7ff1d8005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba2c6aa98b6e5da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.771+0000 7ff1d5c00640 1 -- 172.21.15.46:0/2344045522 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8044020 con 0x5599a4868530 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.771+0000 7ff1d5c00640 1 -- 172.21.15.46:0/2344045522 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8004240 con 0x5599a4868530 2024-03-08T23:16:27.776 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.771+0000 7ff1d5c00640 1 -- 172.21.15.46:0/2344045522 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d80046c0 con 0x5599a4868530 2024-03-08T23:16:27.780 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- 172.21.15.46:0/2344045522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 msgr2=0x5599a4868910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1e4c00640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 0x7ff1c40a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00982b0 tx=0x7ff1d00d36c0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 0x5599a4868910 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8036040 tx=0x7ff1d8005e80 comp rx=0 tx=0).stop 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- 172.21.15.46:0/2344045522 shutdown_connections 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4871a30 0x5599a4873e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a48714f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 --2- 172.21.15.46:0/2344045522 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868530 0x5599a4868910 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- 172.21.15.46:0/2344045522 >> 172.21.15.46:0/2344045522 conn(0x5599a4862c60 msgr2=0x5599a4863040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- 172.21.15.46:0/2344045522 shutdown_connections 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- 172.21.15.46:0/2344045522 wait complete. 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.772+0000 7ff1ea405740 1 -- start start 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 Processor -- start 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a48713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a486ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a48747c0 con 0x5599a4868e50 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a47890c0 con 0x5599a4871a30 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599a4874940 con 0x5599a4868530 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1ea405740 10 client.0 _ll_register_callbacks cb 0x5599a47c3ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39024/0 (socket says 172.21.15.46:39024) 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a486ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1dfe00640 1 -- 172.21.15.46:0/3487124503 learned_addr learned my addr 172.21.15.46:0/3487124503 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.773+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a48713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.783 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 -- 172.21.15.46:0/3487124503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 msgr2=0x5599a48713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a48713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 -- 172.21.15.46:0/3487124503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 msgr2=0x5599a486ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a486ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 -- 172.21.15.46:0/3487124503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599a44e13f0 con 0x5599a4868e50 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc012cf0 tx=0x7ff1cc00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c70439aeb35a4e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1ca000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc012110 con 0x5599a4868e50 2024-03-08T23:16:27.784 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff1cc004050 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.774+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1d8004240 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1cc0122b0 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 10 client.17226 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5599a45ab7d0 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8004240 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff1cc0106d0 con 0x5599a4868e50 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 10 client.17226 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 10 client.17226 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:16:27.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 20 client.17226 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a4881aa0 0x5599a4883ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 4 client.17226 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.775+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599a47a14f0 con 0x5599a4881aa0 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.776+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a4881aa0 0x5599a4883ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.776+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a4881aa0 0x5599a4883ec0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9185396a3ce67919 server_cookie=2ec62e976f21b192 in_seq=0 out_seq=0 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.776+0000 7ff1ca000640 10 client.17226 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ca000640 1 -- 172.21.15.46:0/3487124503 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5599a47a14f0 con 0x5599a4881aa0 2024-03-08T23:16:27.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ca000640 10 client.17226 handle_command_reply: tid=0 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 client.17226 shutdown 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 msgr2=0x5599a486e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc012cf0 tx=0x7ff1cc00f790 comp rx=0 tx=0).stop 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 shutdown_connections 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a486ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868e50 0x5599a486e740 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a4881aa0 0x5599a4883ec0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3487124503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a48713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 >> 172.21.15.46:0/3487124503 conn(0x5599a4862c60 msgr2=0x5599a48649b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ca000640 0 client.17226 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:16:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 shutdown_connections 2024-03-08T23:16:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.777+0000 7ff1ea405740 1 -- 172.21.15.46:0/3487124503 wait complete. 2024-03-08T23:16:27.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.778+0000 7ff1ea405740 20 client.17226 trim_cache size 0 max 16384 2024-03-08T23:16:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.790+0000 7ff1ea405740 1 Processor -- start 2024-03-08T23:16:27.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 -- start start 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a4888840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868e50 0x5599a4870e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01b2c10 con 0x5599a4871a30 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01b07b0 con 0x5599a4868e50 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1ea405740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01ae870 con 0x5599a4868530 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.794 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39026/0 (socket says 172.21.15.46:39026) 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 -- 172.21.15.46:0/3342013365 learned_addr learned my addr 172.21.15.46:0/3342013365 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868e50 0x5599a4870e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 -- 172.21.15.46:0/3342013365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 msgr2=0x5599a4888840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a4888840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 -- 172.21.15.46:0/3342013365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868e50 msgr2=0x5599a4870e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868e50 0x5599a4870e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.795 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.791+0000 7ff1e5600640 1 -- 172.21.15.46:0/3342013365 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599a486eff0 con 0x5599a4871a30 2024-03-08T23:16:27.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.792+0000 7ff1e5600640 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1d802c240 tx=0x7ff1d8005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a36d797442ce2c5b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.792+0000 7ff1d5200640 1 -- 172.21.15.46:0/3342013365 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8044040 con 0x5599a4871a30 2024-03-08T23:16:27.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.792+0000 7ff1d5200640 1 -- 172.21.15.46:0/3342013365 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8004220 con 0x5599a4871a30 2024-03-08T23:16:27.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.792+0000 7ff1d5200640 1 -- 172.21.15.46:0/3342013365 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1d8056870 con 0x5599a4871a30 2024-03-08T23:16:27.799 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- 172.21.15.46:0/3342013365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 msgr2=0x5599a48713b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7ff1d802c240 tx=0x7ff1d8005340 comp rx=0 tx=0).stop 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- 172.21.15.46:0/3342013365 shutdown_connections 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4871a30 0x5599a48713b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4868e50 0x5599a4870e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 --2- 172.21.15.46:0/3342013365 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868530 0x5599a4888840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- 172.21.15.46:0/3342013365 >> 172.21.15.46:0/3342013365 conn(0x5599a4862c60 msgr2=0x5599a4863040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- 172.21.15.46:0/3342013365 shutdown_connections 2024-03-08T23:16:27.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- 172.21.15.46:0/3342013365 wait complete. 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 -- start start 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.793+0000 7ff1ea405740 1 Processor -- start 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 0x5599a4877780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a4878200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01aee70 con 0x5599a4868530 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01b0150 con 0x5599a4871a30 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e01af490 con 0x5599a4868e50 2024-03-08T23:16:27.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1ea405740 10 client.0 _ll_register_callbacks cb 0x7ff1d00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 0x5599a4877780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1dfe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52246/0 (socket says 172.21.15.46:52246) 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1dfe00640 1 -- 172.21.15.46:0/373911735 learned_addr learned my addr 172.21.15.46:0/373911735 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.794+0000 7ff1e5600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 0x5599a4877780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39032/0 (socket says 172.21.15.46:39032) 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1e5600640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a4878200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 -- 172.21.15.46:0/373911735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 msgr2=0x5599a4878200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a4878200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 -- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 msgr2=0x5599a4877780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 0x5599a4877780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 -- 172.21.15.46:0/373911735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599a48e0a30 con 0x5599a4868e50 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1dfe00640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5599a486fd10 tx=0x7ff1cc009c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1cea43083c2faaf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.795+0000 7ff1ca000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc02d070 con 0x5599a4868e50 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1d8004220 con 0x5599a4868e50 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff1cc010430 con 0x5599a4868e50 2024-03-08T23:16:27.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1cc010630 con 0x5599a4868e50 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 10 client.27346 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5599a48e1d30 con 0x5599a4868e50 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1d8004220 con 0x5599a4868e50 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff1cc01c420 con 0x5599a4868e50 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 10 client.27346 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 10 client.27346 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 20 client.27346 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:27.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a48e23a0 0x5599a48e47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 4 client.27346 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:16:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.796+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5599a47da310 con 0x5599a48e23a0 2024-03-08T23:16:27.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.797+0000 7ff1e5600640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a48e23a0 0x5599a48e47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.797+0000 7ff1e5600640 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a48e23a0 0x5599a48e47c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=701c2161cd388792 server_cookie=efa736afd755a6d6 in_seq=0 out_seq=0 2024-03-08T23:16:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.797+0000 7ff1ca000640 10 client.27346 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:16:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ca000640 1 -- 172.21.15.46:0/373911735 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5599a47da310 con 0x5599a48e23a0 2024-03-08T23:16:27.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ca000640 10 client.27346 handle_command_reply: tid=0 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 client.27346 shutdown 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 msgr2=0x5599a4877cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5599a486fd10 tx=0x7ff1cc009c10 comp rx=0 tx=0).stop 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 shutdown_connections 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5599a4871a30 0x5599a4878200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x5599a48e23a0 0x5599a48e47c0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5599a4868e50 0x5599a4877cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 --2- 172.21.15.46:0/373911735 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5599a4868530 0x5599a4877780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ca000640 0 client.27346 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:16:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 >> 172.21.15.46:0/373911735 conn(0x5599a4862c60 msgr2=0x5599a4888840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 shutdown_connections 2024-03-08T23:16:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 1 -- 172.21.15.46:0/373911735 wait complete. 2024-03-08T23:16:27.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.798+0000 7ff1ea405740 20 client.27346 trim_cache size 0 max 16384 2024-03-08T23:16:27.808 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:16:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 msgr2=0x7ff1c40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 0x7ff1c40a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1d00982b0 tx=0x7ff1d00d36c0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 msgr2=0x7ff1e019f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:27.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 0x7ff1e019f900 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000eb00 tx=0x7ff1d000c2d0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 shutdown_connections 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff1c40a29f0 0x7ff1c40a4eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1e0199ab0 0x7ff1e0199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff1e0106490 0x7ff1e019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 --2- 172.21.15.46:0/505036532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff1e01013c0 0x7ff1e019f900 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 >> 172.21.15.46:0/505036532 conn(0x7ff1e00fcc60 msgr2=0x7ff1e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 shutdown_connections 2024-03-08T23:16:27.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:27.799+0000 7ff1e6000640 1 -- 172.21.15.46:0/505036532 wait complete. 2024-03-08T23:16:27.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:27 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1458576460' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:16:27.820 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:16:27.820 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:16:28.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 -- 172.21.15.46:0/1706851423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1000a0 msgr2=0x7f40fc10c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 --2- 172.21.15.46:0/1706851423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1000a0 0x7f40fc10c300 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009930 tx=0x7f40f402bc30 comp rx=0 tx=0).stop 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 -- 172.21.15.46:0/1706851423 shutdown_connections 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 --2- 172.21.15.46:0/1706851423 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1000a0 0x7f40fc10c300 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 --2- 172.21.15.46:0/1706851423 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc0ff6e0 0x7f40fc0ffb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 --2- 172.21.15.46:0/1706851423 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc1054d0 0x7f40fc1058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.041+0000 7f4103000640 1 -- 172.21.15.46:0/1706851423 >> 172.21.15.46:0/1706851423 conn(0x7f40fc074e90 msgr2=0x7f40fc0752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.045 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.042+0000 7f4103000640 1 -- 172.21.15.46:0/1706851423 shutdown_connections 2024-03-08T23:16:28.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.042+0000 7f4103000640 1 -- 172.21.15.46:0/1706851423 wait complete. 2024-03-08T23:16:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 Processor -- start 2024-03-08T23:16:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 -- start start 2024-03-08T23:16:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc0ff6e0 0x7f40fc19d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.048 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 0x7f40fc197920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc111660 con 0x7f40fc1054d0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1114e0 con 0x7f40fc1000a0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4103000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1117e0 con 0x7f40fc0ff6e0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.043+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 0x7f40fc197920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc0ff6e0 0x7f40fc19d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.049 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44788/0 (socket says 172.21.15.46:44788) 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 -- 172.21.15.46:0/3383079623 learned_addr learned my addr 172.21.15.46:0/3383079623 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 0x7f40fc197920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39062/0 (socket says 172.21.15.46:39062) 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 -- 172.21.15.46:0/3383079623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc0ff6e0 msgr2=0x7f40fc19d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc0ff6e0 0x7f40fc19d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 -- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 msgr2=0x7f40fc197920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 0x7f40fc197920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.050 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 -- 172.21.15.46:0/3383079623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40fc1980a0 con 0x7f40fc1000a0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.044+0000 7f4101200640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f40f000ebc0 tx=0x7f40f000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.045+0000 7f40fa000640 1 -- 172.21.15.46:0/3383079623 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0010630 con 0x7f40fc1000a0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.045+0000 7f40fa000640 1 -- 172.21.15.46:0/3383079623 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40f0010cb0 con 0x7f40fc1000a0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.045+0000 7f40fa000640 1 -- 172.21.15.46:0/3383079623 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0018bd0 con 0x7f40fc1000a0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.045+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40fc198390 con 0x7f40fc1000a0 2024-03-08T23:16:28.051 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.045+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40fc10e8d0 con 0x7f40fc1000a0 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.048+0000 7f41071ec740 1 Processor -- start 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.048+0000 7f41071ec740 1 -- start start 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.048+0000 7f40fa000640 1 -- 172.21.15.46:0/3383079623 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f40f00107d0 con 0x7f40fc1000a0 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.048+0000 7f41071ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977461d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.048+0000 7f41071ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f41071ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 0x559774628e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f41071ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559774581dd0 con 0x55977461de50 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f41071ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55977452ac70 con 0x559774626a30 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f41071ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55977457de70 con 0x55977461d530 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39074/0 (socket says 172.21.15.46:39074) 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 -- 172.21.15.46:0/3705238282 learned_addr learned my addr 172.21.15.46:0/3705238282 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101200640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 0x559774628e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4102600640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977461d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 -- 172.21.15.46:0/3705238282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 msgr2=0x55977461d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977461d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 -- 172.21.15.46:0/3705238282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 msgr2=0x559774628e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 0x559774628e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101c00640 1 -- 172.21.15.46:0/3705238282 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597745e42b0 con 0x55977461de50 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.049+0000 7f4101200640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 0x559774628e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:28.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f4101c00640 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009c10 tx=0x7f40ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ea4e2d85be47ff server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f40eaa00640 1 -- 172.21.15.46:0/3705238282 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec004230 con 0x55977461de50 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f40eaa00640 1 -- 172.21.15.46:0/3705238282 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40ec0043d0 con 0x55977461de50 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f40eaa00640 1 -- 172.21.15.46:0/3705238282 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ec004db0 con 0x55977461de50 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f40fa000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 0x7f40e40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f40fa000640 1 -- 172.21.15.46:0/3383079623 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f40f00cf7f0 con 0x7f40fc1000a0 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.050+0000 7f4102600640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 0x7f40e40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 -- 172.21.15.46:0/3705238282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 msgr2=0x5597746264f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f40ec009c10 tx=0x7f40ec011810 comp rx=0 tx=0).stop 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 -- 172.21.15.46:0/3705238282 shutdown_connections 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774626a30 0x559774628e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x5597746264f0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 --2- 172.21.15.46:0/3705238282 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977461d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 -- 172.21.15.46:0/3705238282 >> 172.21.15.46:0/3705238282 conn(0x559774617c60 msgr2=0x559774618040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 -- 172.21.15.46:0/3705238282 shutdown_connections 2024-03-08T23:16:28.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f41071ec740 1 -- 172.21.15.46:0/3705238282 wait complete. 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.051+0000 7f4102600640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 0x7f40e40a4eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036040 tx=0x7f40f403e040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 -- start start 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 Processor -- start 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x5597746262f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774623bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x559774629820 con 0x55977461de50 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55977453e0c0 con 0x55977461d530 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597746299a0 con 0x559774626a30 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.052+0000 7f41071ec740 10 client.0 _ll_register_callbacks cb 0x559774578ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39090/0 (socket says 172.21.15.46:39090) 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 -- 172.21.15.46:0/1268613357 learned_addr learned my addr 172.21.15.46:0/1268613357 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:28.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4102600640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774623bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101c00640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x5597746262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 -- 172.21.15.46:0/1268613357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 msgr2=0x559774623bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774623bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 -- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 msgr2=0x5597746262f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x5597746262f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 -- 172.21.15.46:0/1268613357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5597742963f0 con 0x55977461de50 2024-03-08T23:16:28.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f4101200640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f40f000eb90 tx=0x7f40f009b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530281bec71f99f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.053+0000 7f40df400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0010e60 con 0x55977461de50 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f40f00cf370 con 0x55977461de50 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40ec0043d0 con 0x55977461de50 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f00ccfa0 con 0x55977461de50 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f41071ec740 10 client.17244 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:28.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5597743607d0 con 0x55977461de50 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40ec0043d0 con 0x55977461de50 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f40f00cfbe0 con 0x55977461de50 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f41071ec740 10 client.17244 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.054+0000 7f41071ec740 10 client.17244 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f41071ec740 20 client.17244 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559774636a90 0x559774638e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f41071ec740 4 client.17244 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5597745564f0 con 0x559774636a90 2024-03-08T23:16:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f4101c00640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559774636a90 0x559774638e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f4101c00640 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559774636a90 0x559774638e80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffd4264982eec7f2 server_cookie=d2f3a85829971a4f in_seq=0 out_seq=0 2024-03-08T23:16:28.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.055+0000 7f40df400640 10 client.17244 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.056+0000 7f40df400640 1 -- 172.21.15.46:0/1268613357 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5597745564f0 con 0x559774636a90 2024-03-08T23:16:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.056+0000 7f40df400640 10 client.17244 handle_command_reply: tid=0 2024-03-08T23:16:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.056+0000 7f41071ec740 1 client.17244 shutdown 2024-03-08T23:16:28.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.056+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 msgr2=0x559774623680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.056+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f40f000eb90 tx=0x7f40f009b8b0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 shutdown_connections 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559774636a90 0x559774638e80 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774623bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55977461de50 0x559774623680 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 --2- 172.21.15.46:0/1268613357 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x5597746262f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.070 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 >> 172.21.15.46:0/1268613357 conn(0x559774617c60 msgr2=0x5597746199b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f40df400640 0 client.17244 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 shutdown_connections 2024-03-08T23:16:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 1 -- 172.21.15.46:0/1268613357 wait complete. 2024-03-08T23:16:28.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.057+0000 7f41071ec740 20 client.17244 trim_cache size 0 max 16384 2024-03-08T23:16:28.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 Processor -- start 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 -- start start 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x559774616b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774625db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 0x5597746262f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1ae800 con 0x559774623cb0 2024-03-08T23:16:28.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1ac3a0 con 0x55977461d530 2024-03-08T23:16:28.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.070+0000 7f41071ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1aa460 con 0x559774626a30 2024-03-08T23:16:28.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 0x5597746262f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 0x5597746262f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39094/0 (socket says 172.21.15.46:39094) 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4102600640 1 -- 172.21.15.46:0/1816120624 learned_addr learned my addr 172.21.15.46:0/1816120624 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774625db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101c00640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x559774616b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 -- 172.21.15.46:0/1816120624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 msgr2=0x559774616b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x559774616b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 -- 172.21.15.46:0/1816120624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 msgr2=0x5597746262f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 0x5597746262f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 -- 172.21.15.46:0/1816120624 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55977461dc80 con 0x559774626a30 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101c00640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x559774616b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.071+0000 7f4101200640 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774625db0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x559774624710 tx=0x7f40f0099ff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=130bbea283f297a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f40ea000640 1 -- 172.21.15.46:0/1816120624 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00ccfa0 con 0x559774626a30 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f40ea000640 1 -- 172.21.15.46:0/1816120624 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40f009bae0 con 0x559774626a30 2024-03-08T23:16:28.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f40ea000640 1 -- 172.21.15.46:0/1816120624 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f00cf370 con 0x559774626a30 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 -- 172.21.15.46:0/1816120624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 msgr2=0x559774625db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774625db0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x559774624710 tx=0x7f40f0099ff0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 -- 172.21.15.46:0/1816120624 shutdown_connections 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774623cb0 0x5597746262f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559774626a30 0x559774625db0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 --2- 172.21.15.46:0/1816120624 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55977461d530 0x559774616b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.072+0000 7f41071ec740 1 -- 172.21.15.46:0/1816120624 >> 172.21.15.46:0/1816120624 conn(0x559774617c60 msgr2=0x559774616450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.073+0000 7f41071ec740 1 -- 172.21.15.46:0/1816120624 shutdown_connections 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.073+0000 7f41071ec740 1 -- 172.21.15.46:0/1816120624 wait complete. 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.073+0000 7f41071ec740 1 -- start start 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.073+0000 7f41071ec740 1 Processor -- start 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977462c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 0x55977462cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1aaa60 con 0x559774626a30 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1abd40 con 0x559774623cb0 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40fc1ab080 con 0x55977461d530 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f41071ec740 10 client.0 _ll_register_callbacks cb 0x7f40ec0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39110/0 (socket says 172.21.15.46:39110) 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 -- 172.21.15.46:0/123358334 learned_addr learned my addr 172.21.15.46:0/123358334 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:28.088 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4101200640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 0x55977462cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4101c00640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977462c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 -- 172.21.15.46:0/123358334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 msgr2=0x55977462c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977462c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 -- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 msgr2=0x55977462cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 0x55977462cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4102600640 1 -- 172.21.15.46:0/123358334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55977468d410 con 0x559774626a30 2024-03-08T23:16:28.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4101c00640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977462c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.074+0000 7f4101200640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 0x55977462cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f4102600640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009230 tx=0x7f40f402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac51ecc3bbf00bb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f40df400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f405b060 con 0x559774626a30 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f40f4005160 con 0x559774626a30 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40f009bae0 con 0x559774626a30 2024-03-08T23:16:28.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f4005360 con 0x559774626a30 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f41071ec740 10 client.17250 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.075+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55977468e540 con 0x559774626a30 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40f009bae0 con 0x559774626a30 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f40f403c5d0 con 0x559774626a30 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 10 client.17250 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:28.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 20 client.17250 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55977468ec70 0x559774691090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 4 client.17250 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.076+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55977458f310 con 0x55977468ec70 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f4101c00640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55977468ec70 0x559774691090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f4101c00640 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55977468ec70 0x559774691090 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=163ebcb408b30080 server_cookie=2b465b6003193970 in_seq=0 out_seq=0 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f40df400640 10 client.17250 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f40df400640 1 -- 172.21.15.46:0/123358334 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55977458f310 con 0x55977468ec70 2024-03-08T23:16:28.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f40df400640 10 client.17250 handle_command_reply: tid=0 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f41071ec740 1 client.17250 shutdown 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 msgr2=0x55977468ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.077+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009230 tx=0x7f40f402bc30 comp rx=0 tx=0).stop 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 shutdown_connections 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55977468ec70 0x559774691090 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x559774626a30 0x55977468ced0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x559774623cb0 0x55977462cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f40df400640 0 client.17250 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 --2- 172.21.15.46:0/123358334 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55977461d530 0x55977462c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 >> 172.21.15.46:0/123358334 conn(0x559774617c60 msgr2=0x55977461e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 shutdown_connections 2024-03-08T23:16:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 1 -- 172.21.15.46:0/123358334 wait complete. 2024-03-08T23:16:28.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f41071ec740 20 client.17250 trim_cache size 0 max 16384 2024-03-08T23:16:28.095 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:16:28.095 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 msgr2=0x7f40e40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.078+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 0x7f40e40a4eb0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f40f4036040 tx=0x7f40f403e040 comp rx=0 tx=0).stop 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 msgr2=0x7f40fc19dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f40f000ebc0 tx=0x7f40f000c6a0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 shutdown_connections 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f40e40a29f0 0x7f40e40a4eb0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f40fc1054d0 0x7f40fc197920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.096 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f40fc1000a0 0x7f40fc19dcb0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 --2- 172.21.15.46:0/3383079623 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f40fc0ff6e0 0x7f40fc19d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 >> 172.21.15.46:0/3383079623 conn(0x7f40fc074e90 msgr2=0x7f40fc0fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 shutdown_connections 2024-03-08T23:16:28.097 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:28.079+0000 7f4103000640 1 -- 172.21.15.46:0/3383079623 wait complete. 2024-03-08T23:16:28.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:28 smithi062 ceph-mon[35761]: pgmap v269: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:28.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:28 smithi046 ceph-mon[31859]: pgmap v269: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:28.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:28 smithi192 ceph-mon[36700]: pgmap v269: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:29.099 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:16:29.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:16:29.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:16:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 -- 172.21.15.46:0/4084071769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8075820 msgr2=0x7f44a8075c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 --2- 172.21.15.46:0/4084071769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8075820 0x7f44a8075c00 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4498009930 tx=0x7f449802bc30 comp rx=0 tx=0).stop 2024-03-08T23:16:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 -- 172.21.15.46:0/4084071769 shutdown_connections 2024-03-08T23:16:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 --2- 172.21.15.46:0/4084071769 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a8114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 --2- 172.21.15.46:0/4084071769 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8076140 0x7f44a8111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 --2- 172.21.15.46:0/4084071769 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8075820 0x7f44a8075c00 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 -- 172.21.15.46:0/4084071769 >> 172.21.15.46:0/4084071769 conn(0x7f44a80fefa0 msgr2=0x7f44a81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 -- 172.21.15.46:0/4084071769 shutdown_connections 2024-03-08T23:16:29.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.333+0000 7f44aca00640 1 -- 172.21.15.46:0/4084071769 wait complete. 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.334+0000 7f44aca00640 1 Processor -- start 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.334+0000 7f44aca00640 1 -- start start 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 0x7f44a819e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8076140 0x7f44a819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8117c90 con 0x7f44a8112130 2024-03-08T23:16:29.340 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8117b10 con 0x7f44a8075820 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44aca00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a8117e10 con 0x7f44a8076140 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44a6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44a6800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39126/0 (socket says 172.21.15.46:39126) 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44a6800640 1 -- 172.21.15.46:0/3470480358 learned_addr learned my addr 172.21.15.46:0/3470480358 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44a5e00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 0x7f44a819e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.335+0000 7f44a5400640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8076140 0x7f44a819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 -- 172.21.15.46:0/3470480358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8076140 msgr2=0x7f44a819e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8076140 0x7f44a819e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 -- 172.21.15.46:0/3470480358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 msgr2=0x7f44a819e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 0x7f44a819e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 -- 172.21.15.46:0/3470480358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a81a31f0 con 0x7f44a8112130 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a5e00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 0x7f44a819e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44a6800640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44a000eb40 tx=0x7f44a000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f449ea00640 1 -- 172.21.15.46:0/3470480358 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0004b20 con 0x7f44a8112130 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a81a34e0 con 0x7f44a8112130 2024-03-08T23:16:29.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f449ea00640 1 -- 172.21.15.46:0/3470480358 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44a0004cc0 con 0x7f44a8112130 2024-03-08T23:16:29.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.336+0000 7f449ea00640 1 -- 172.21.15.46:0/3470480358 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0010850 con 0x7f44a8112130 2024-03-08T23:16:29.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.337+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a81aad70 con 0x7f44a8112130 2024-03-08T23:16:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.340+0000 7f449ea00640 1 -- 172.21.15.46:0/3470480358 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f44a0005020 con 0x7f44a8112130 2024-03-08T23:16:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.340+0000 7f44af7f6740 1 Processor -- start 2024-03-08T23:16:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 -- start start 2024-03-08T23:16:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56d819c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d8ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56ce5a50 con 0x562b56d815e0 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56c8eb00 con 0x562b56d81f00 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44af7f6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56ce1af0 con 0x562b56d8aae0 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f449ea00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 0x7f448c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44a5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44a5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56d819c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44a5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56d819c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39136/0 (socket says 172.21.15.46:39136) 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44a5400640 1 -- 172.21.15.46:0/829486094 learned_addr learned my addr 172.21.15.46:0/829486094 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f44a5e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44856/0 (socket says 172.21.15.46:44856) 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.341+0000 7f449ea00640 1 -- 172.21.15.46:0/3470480358 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f44a00cf9e0 con 0x7f44a8112130 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5400640 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d8ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a6800640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 0x7f448c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 -- 172.21.15.46:0/829486094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 msgr2=0x562b56d8ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d8ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 -- 172.21.15.46:0/829486094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 msgr2=0x562b56d819c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56d819c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 -- 172.21.15.46:0/829486094 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b56cac1e0 con 0x562b56d81f00 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5400640 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d8ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a5e00640 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4498009230 tx=0x7f4498009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90bb6181b9135de server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.342+0000 7f44a6800640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 0x7f448c0a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44a00a3e90 tx=0x7f44a0004a40 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.343+0000 7f4493000640 1 -- 172.21.15.46:0/829486094 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498042070 con 0x562b56d81f00 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.343+0000 7f4493000640 1 -- 172.21.15.46:0/829486094 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4498004a70 con 0x562b56d81f00 2024-03-08T23:16:29.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.343+0000 7f4493000640 1 -- 172.21.15.46:0/829486094 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498005190 con 0x562b56d81f00 2024-03-08T23:16:29.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.343+0000 7f44af7f6740 1 -- 172.21.15.46:0/829486094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 msgr2=0x562b56d8a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.354 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.343+0000 7f44af7f6740 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4498009230 tx=0x7f4498009630 comp rx=0 tx=0).stop 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 -- 172.21.15.46:0/829486094 shutdown_connections 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d8ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56d8a5a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 --2- 172.21.15.46:0/829486094 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56d819c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 -- 172.21.15.46:0/829486094 >> 172.21.15.46:0/829486094 conn(0x562b56d7bd10 msgr2=0x562b56d7c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 -- 172.21.15.46:0/829486094 shutdown_connections 2024-03-08T23:16:29.355 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 -- 172.21.15.46:0/829486094 wait complete. 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 -- start start 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.344+0000 7f44af7f6740 1 Processor -- start 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d87aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d88520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56d8d870 con 0x562b56d81f00 2024-03-08T23:16:29.356 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56ca1f60 con 0x562b56d8aae0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b56d8d9f0 con 0x562b56d815e0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44af7f6740 10 client.0 _ll_register_callbacks cb 0x562b56c8db60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44a5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44a5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d88520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44a5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39144/0 (socket says 172.21.15.46:39144) 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44a5e00640 1 -- 172.21.15.46:0/3773378729 learned_addr learned my addr 172.21.15.46:0/3773378729 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.345+0000 7f44a5400640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d87aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 -- 172.21.15.46:0/3773378729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 msgr2=0x562b56d87aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d87aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 -- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 msgr2=0x562b56d88520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d88520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 -- 172.21.15.46:0/3773378729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b56ac27e0 con 0x562b56d81f00 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5400640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d88520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44a5e00640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4498036040 tx=0x7f449802c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec52bf80a80a99d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f4487400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4498042030 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4498004f30 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.346+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4498004a70 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4498057050 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 10 client.17262 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562b56b22f70 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4498004a70 con 0x562b56d81f00 2024-03-08T23:16:29.359 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4498003cd0 con 0x562b56d81f00 2024-03-08T23:16:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 10 client.17262 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 10 client.17262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 20 client.17262 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56d9ac60 0x562b56d9d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 4 client.17262 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.347+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b56cf2f90 con 0x562b56d9ac60 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.348+0000 7f44a5400640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56d9ac60 0x562b56d9d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.348+0000 7f44a5400640 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56d9ac60 0x562b56d9d050 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98e1766310c349fa server_cookie=c7127ec2023946c4 in_seq=0 out_seq=0 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.348+0000 7f4487400640 10 client.17262 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f4487400640 1 -- 172.21.15.46:0/3773378729 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b56cf2f90 con 0x562b56d9ac60 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f4487400640 10 client.17262 handle_command_reply: tid=0 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 client.17262 shutdown 2024-03-08T23:16:29.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 msgr2=0x562b56d87fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f4498036040 tx=0x7f449802c5d0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 shutdown_connections 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56d9ac60 0x562b56d9d050 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d88520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d87fe0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 --2- 172.21.15.46:0/3773378729 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d87aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 >> 172.21.15.46:0/3773378729 conn(0x562b56d7bd10 msgr2=0x562b56d7da60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.349+0000 7f4487400640 0 client.17262 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.350+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 shutdown_connections 2024-03-08T23:16:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.350+0000 7f44af7f6740 1 -- 172.21.15.46:0/3773378729 wait complete. 2024-03-08T23:16:29.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.350+0000 7f44af7f6740 20 client.17262 trim_cache size 0 max 16384 2024-03-08T23:16:29.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.369+0000 7f44af7f6740 1 Processor -- start 2024-03-08T23:16:29.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.369+0000 7f44af7f6740 1 -- start start 2024-03-08T23:16:29.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d7ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d91c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b7070 con 0x562b56d81f00 2024-03-08T23:16:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b4c10 con 0x562b56d8aae0 2024-03-08T23:16:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44af7f6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b2cd0 con 0x562b56d815e0 2024-03-08T23:16:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d7ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52326/0 (socket says 172.21.15.46:52326) 2024-03-08T23:16:29.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 -- 172.21.15.46:0/49636943 learned_addr learned my addr 172.21.15.46:0/49636943 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d91c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 -- 172.21.15.46:0/49636943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 msgr2=0x562b56d91c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d91c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 -- 172.21.15.46:0/49636943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 msgr2=0x562b56d7ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d7ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.370+0000 7f44a5400640 1 -- 172.21.15.46:0/49636943 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b569ba170 con 0x562b56d815e0 2024-03-08T23:16:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.371+0000 7f44a5400640 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x562b56d9d670 tx=0x7f4494015040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d569f0cce6e0081 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.371+0000 7f4492600640 1 -- 172.21.15.46:0/49636943 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449401c040 con 0x562b56d815e0 2024-03-08T23:16:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.371+0000 7f4492600640 1 -- 172.21.15.46:0/49636943 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4494004030 con 0x562b56d815e0 2024-03-08T23:16:29.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.371+0000 7f4492600640 1 -- 172.21.15.46:0/49636943 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494003a80 con 0x562b56d815e0 2024-03-08T23:16:29.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- 172.21.15.46:0/49636943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 msgr2=0x562b56d8a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x562b56d9d670 tx=0x7f4494015040 comp rx=0 tx=0).stop 2024-03-08T23:16:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- 172.21.15.46:0/49636943 shutdown_connections 2024-03-08T23:16:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d8aae0 0x562b56d91c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d81f00 0x562b56d7ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 --2- 172.21.15.46:0/49636943 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d815e0 0x562b56d8a160 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- 172.21.15.46:0/49636943 >> 172.21.15.46:0/49636943 conn(0x562b56d7bd10 msgr2=0x562b56d7a580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- 172.21.15.46:0/49636943 shutdown_connections 2024-03-08T23:16:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- 172.21.15.46:0/49636943 wait complete. 2024-03-08T23:16:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.372+0000 7f44af7f6740 1 -- start start 2024-03-08T23:16:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 Processor -- start 2024-03-08T23:16:29.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56dea7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d910a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b32d0 con 0x562b56d815e0 2024-03-08T23:16:29.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b45b0 con 0x562b56d81f00 2024-03-08T23:16:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a81b38f0 con 0x562b56d8aae0 2024-03-08T23:16:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44af7f6740 10 client.0 _ll_register_callbacks cb 0x7f4494005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:29.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44a5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44a5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:39158/0 (socket says 172.21.15.46:39158) 2024-03-08T23:16:29.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.373+0000 7f44a5400640 1 -- 172.21.15.46:0/1186884537 learned_addr learned my addr 172.21.15.46:0/1186884537 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.391 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5e00640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56dea7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d910a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 -- 172.21.15.46:0/1186884537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 msgr2=0x562b56d910a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d910a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 -- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 msgr2=0x562b56dea7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56dea7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 -- 172.21.15.46:0/1186884537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b56cb1050 con 0x562b56d815e0 2024-03-08T23:16:29.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44a5400640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f44940107c0 tx=0x7f4494010750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9db4684ce7bf1d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f4487400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:16:29.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494010b00 con 0x562b56d815e0 2024-03-08T23:16:29.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4494010ca0 con 0x562b56d815e0 2024-03-08T23:16:29.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.374+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4494004030 con 0x562b56d815e0 2024-03-08T23:16:29.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4494025450 con 0x562b56d815e0 2024-03-08T23:16:29.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 10 client.17268 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:29.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562b56df29e0 con 0x562b56d815e0 2024-03-08T23:16:29.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4494004030 con 0x562b56d815e0 2024-03-08T23:16:29.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4494022040 con 0x562b56d815e0 2024-03-08T23:16:29.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 10 client.17268 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:29.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 10 client.17268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:29.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 20 client.17268 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56df2fd0 0x562b56df53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 4 client.17268 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.375+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562b56ced2e0 con 0x562b56df2fd0 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.376+0000 7f44a5e00640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56df2fd0 0x562b56df53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.376+0000 7f44a5e00640 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56df2fd0 0x562b56df53f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a5f00e1bcea4408 server_cookie=8287b304abd75e3c in_seq=0 out_seq=0 2024-03-08T23:16:29.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.376+0000 7f4487400640 10 client.17268 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.377+0000 7f4487400640 1 -- 172.21.15.46:0/1186884537 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562b56ced2e0 con 0x562b56df2fd0 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.377+0000 7f4487400640 10 client.17268 handle_command_reply: tid=0 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.377+0000 7f44af7f6740 1 client.17268 shutdown 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 msgr2=0x562b56dea290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f44940107c0 tx=0x7f4494010750 comp rx=0 tx=0).stop 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 shutdown_connections 2024-03-08T23:16:29.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x562b56df2fd0 0x562b56df53f0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x562b56d8aae0 0x562b56d910a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x562b56d81f00 0x562b56dea7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f4487400640 0 client.17268 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 --2- 172.21.15.46:0/1186884537 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x562b56d815e0 0x562b56dea290 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 >> 172.21.15.46:0/1186884537 conn(0x562b56d7bd10 msgr2=0x562b56d7d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 shutdown_connections 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 1 -- 172.21.15.46:0/1186884537 wait complete. 2024-03-08T23:16:29.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.378+0000 7f44af7f6740 20 client.17268 trim_cache size 0 max 16384 2024-03-08T23:16:29.400 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:16:29.400 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:16:29.400 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "6fae20a3-fc2c-4182-84f3-b36637893a67", 2024-03-08T23:16:29.400 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:16:29.400 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:16:29.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 msgr2=0x7f448c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 0x7f448c0a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f44a00a3e90 tx=0x7f44a0004a40 comp rx=0 tx=0).stop 2024-03-08T23:16:29.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 msgr2=0x7f44a81a2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f44a000eb40 tx=0x7f44a000c2d0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 shutdown_connections 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f448c0a2970 0x7f448c0a4e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f44a8112130 0x7f44a81a2ac0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f44a8076140 0x7f44a819e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 --2- 172.21.15.46:0/3470480358 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f44a8075820 0x7f44a819e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 >> 172.21.15.46:0/3470480358 conn(0x7f44a80fefa0 msgr2=0x7f44a8101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 shutdown_connections 2024-03-08T23:16:29.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.379+0000 7f44aca00640 1 -- 172.21.15.46:0/3470480358 wait complete. 2024-03-08T23:16:29.408 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:16:29.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 -- 172.21.15.46:0/1299187565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc10be00 msgr2=0x7f21cc10c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 --2- 172.21.15.46:0/1299187565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc10be00 0x7f21cc10c1e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009900 tx=0x7f21bc02bc30 comp rx=0 tx=0).stop 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 -- 172.21.15.46:0/1299187565 shutdown_connections 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 --2- 172.21.15.46:0/1299187565 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc110e80 0x7f21cc118a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 --2- 172.21.15.46:0/1299187565 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10c7b0 0x7f21cc1107a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 --2- 172.21.15.46:0/1299187565 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc10be00 0x7f21cc10c1e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.685+0000 7f21cb200640 1 -- 172.21.15.46:0/1299187565 >> 172.21.15.46:0/1299187565 conn(0x7f21cc06c120 msgr2=0x7f21cc10a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.686+0000 7f21cb200640 1 -- 172.21.15.46:0/1299187565 shutdown_connections 2024-03-08T23:16:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.686+0000 7f21cb200640 1 -- 172.21.15.46:0/1299187565 wait complete. 2024-03-08T23:16:29.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.687+0000 7f21cb200640 1 Processor -- start 2024-03-08T23:16:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.687+0000 7f21cb200640 1 -- start start 2024-03-08T23:16:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10be00 0x7f21cc1a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc110e80 0x7f21cc1a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc11bf80 con 0x7f21cc10be00 2024-03-08T23:16:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc11be00 con 0x7f21cc110e80 2024-03-08T23:16:29.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21cb200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc11c100 con 0x7f21cc1a6d50 2024-03-08T23:16:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.688+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10be00 0x7f21cc1a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52346/0 (socket says 172.21.15.46:52346) 2024-03-08T23:16:29.692 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 -- 172.21.15.46:0/809149736 learned_addr learned my addr 172.21.15.46:0/809149736 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21c9400640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc110e80 0x7f21cc1a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 -- 172.21.15.46:0/809149736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc110e80 msgr2=0x7f21cc1a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc110e80 0x7f21cc1a2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 -- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10be00 msgr2=0x7f21cc1a24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10be00 0x7f21cc1a24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 -- 172.21.15.46:0/809149736 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc1a7840 con 0x7f21cc1a6d50 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.689+0000 7f21ca800640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009fd0 tx=0x7f21c400b260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.690+0000 7f21c2a00640 1 -- 172.21.15.46:0/809149736 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c4017070 con 0x7f21cc1a6d50 2024-03-08T23:16:29.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.690+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc1a7b30 con 0x7f21cc1a6d50 2024-03-08T23:16:29.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.690+0000 7f21c2a00640 1 -- 172.21.15.46:0/809149736 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21c4003e50 con 0x7f21cc1a6d50 2024-03-08T23:16:29.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.690+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc1af2d0 con 0x7f21cc1a6d50 2024-03-08T23:16:29.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.690+0000 7f21c2a00640 1 -- 172.21.15.46:0/809149736 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21c40106c0 con 0x7f21cc1a6d50 2024-03-08T23:16:29.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.695+0000 7f21d3bd8740 1 Processor -- start 2024-03-08T23:16:29.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.695+0000 7f21c2a00640 1 -- 172.21.15.46:0/809149736 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f21c4003970 con 0x7f21cc1a6d50 2024-03-08T23:16:29.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 -- start start 2024-03-08T23:16:29.698 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b567fa3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b567faf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567f65410 con 0x55b567facc90 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567f07d50 con 0x55b567fa40b0 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567eb0b80 con 0x55b567fa3790 2024-03-08T23:16:29.699 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.696+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44894/0 (socket says 172.21.15.46:44894) 2024-03-08T23:16:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 -- 172.21.15.46:0/2782558315 learned_addr learned my addr 172.21.15.46:0/2782558315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9400640 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b567fa3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.700 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9400640 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b567faf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 -- 172.21.15.46:0/2782558315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 msgr2=0x55b567fa3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.701 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b567fa3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 -- 172.21.15.46:0/2782558315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 msgr2=0x55b567faf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b567faf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 -- 172.21.15.46:0/2782558315 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b567ce67d0 con 0x55b567fa40b0 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.697+0000 7f21c9e00640 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009900 tx=0x7f21bc02c5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d100324583cb7f76 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.698+0000 7f21c2a00640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 0x7f21b00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.698+0000 7f21b7400640 1 -- 172.21.15.46:0/2782558315 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc045c00 con 0x55b567fa40b0 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.698+0000 7f21b7400640 1 -- 172.21.15.46:0/2782558315 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21bc045da0 con 0x55b567fa40b0 2024-03-08T23:16:29.702 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.698+0000 7f21b7400640 1 -- 172.21.15.46:0/2782558315 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc004ae0 con 0x55b567fa40b0 2024-03-08T23:16:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.698+0000 7f21c2a00640 1 -- 172.21.15.46:0/809149736 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f21c40cf230 con 0x7f21cc1a6d50 2024-03-08T23:16:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.699+0000 7f21d3bd8740 1 -- 172.21.15.46:0/2782558315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 msgr2=0x55b567fac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.699+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009900 tx=0x7f21bc02c5d0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.703 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.699+0000 7f21c9400640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 0x7f21b00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 -- 172.21.15.46:0/2782558315 shutdown_connections 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b567faf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fac750 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/2782558315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b567fa3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 -- 172.21.15.46:0/2782558315 >> 172.21.15.46:0/2782558315 conn(0x55b567f9dec0 msgr2=0x55b567f9e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 -- 172.21.15.46:0/2782558315 shutdown_connections 2024-03-08T23:16:29.704 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21c9400640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 0x7f21b00a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55b567f9f050 tx=0x7f21b8004730 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.700+0000 7f21d3bd8740 1 -- 172.21.15.46:0/2782558315 wait complete. 2024-03-08T23:16:29.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.701+0000 7f21d3bd8740 1 -- start start 2024-03-08T23:16:29.705 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.701+0000 7f21d3bd8740 1 Processor -- start 2024-03-08T23:16:29.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 0x55b567faa250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567faa790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567ec3ee0 con 0x55b567fa40b0 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567f03df0 con 0x55b567fa3790 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b567fafa20 con 0x55b567facc90 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21d3bd8740 10 client.0 _ll_register_callbacks cb 0x55b567eca8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:29.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:44906/0 (socket says 172.21.15.46:44906) 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21c9e00640 1 -- 172.21.15.46:0/1858062543 learned_addr learned my addr 172.21.15.46:0/1858062543 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 -- 172.21.15.46:0/1858062543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 msgr2=0x55b567faa790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567faa790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 -- 172.21.15.46:0/1858062543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 msgr2=0x55b567faa250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.702+0000 7f21c9400640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 0x55b567faa250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 0x55b567faa250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 -- 172.21.15.46:0/1858062543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b567c1c3f0 con 0x55b567fa3790 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9400640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 0x55b567faa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005c40 tx=0x7f21bc02cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b47cf76c9ebed130 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.703+0000 7f21abe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc045c00 con 0x55b567fa3790 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21bc045da0 con 0x55b567fa3790 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f21bc004b90 con 0x55b567fa3790 2024-03-08T23:16:29.709 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21bc065050 con 0x55b567fa3790 2024-03-08T23:16:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21d3bd8740 10 client.27440 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b567bdc0f0 con 0x55b567fa3790 2024-03-08T23:16:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.704+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21bc045da0 con 0x55b567fa3790 2024-03-08T23:16:29.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f21bc050750 con 0x55b567fa3790 2024-03-08T23:16:29.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 10 client.27440 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 10 client.27440 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 20 client.27440 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b567fbcd10 0x55b567fbf100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 4 client.27440 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b567f15290 con 0x55b567fbcd10 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.705+0000 7f21c9400640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b567fbcd10 0x55b567fbf100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.706+0000 7f21c9400640 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b567fbcd10 0x55b567fbf100 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=648041802e991e55 server_cookie=aea9364f13b6e02f in_seq=0 out_seq=0 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.706+0000 7f21abe00640 10 client.27440 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21abe00640 1 -- 172.21.15.46:0/1858062543 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b567f15290 con 0x55b567fbcd10 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21abe00640 10 client.27440 handle_command_reply: tid=0 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 client.27440 shutdown 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 msgr2=0x55b567fa9d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f21bc005c40 tx=0x7f21bc02cde0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 shutdown_connections 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b567fbcd10 0x55b567fbf100 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567faa790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa40b0 0x55b567faa250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1858062543 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa3790 0x55b567fa9d10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 >> 172.21.15.46:0/1858062543 conn(0x55b567f9dec0 msgr2=0x55b567f9fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.707+0000 7f21abe00640 0 client.27440 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.708+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 shutdown_connections 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.708+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1858062543 wait complete. 2024-03-08T23:16:29.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.708+0000 7f21d3bd8740 20 client.27440 trim_cache size 0 max 16384 2024-03-08T23:16:29.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.728+0000 7f21d3bd8740 1 Processor -- start 2024-03-08T23:16:29.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.728+0000 7f21d3bd8740 1 -- start start 2024-03-08T23:16:29.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 0x55b567fac440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fc3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.731 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b6ec0 con 0x55b567fa3790 2024-03-08T23:16:29.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b4a60 con 0x55b567fa40b0 2024-03-08T23:16:29.732 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b2b20 con 0x55b567facc90 2024-03-08T23:16:29.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21ca800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21ca800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52382/0 (socket says 172.21.15.46:52382) 2024-03-08T23:16:29.733 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21ca800640 1 -- 172.21.15.46:0/1648195781 learned_addr learned my addr 172.21.15.46:0/1648195781 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 0x55b567fac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.729+0000 7f21c9400640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fc3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9400640 1 -- 172.21.15.46:0/1648195781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 msgr2=0x55b567fb3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9400640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9400640 1 -- 172.21.15.46:0/1648195781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 msgr2=0x55b567fac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9400640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 0x55b567fac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9400640 1 -- 172.21.15.46:0/1648195781 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b567f9d9f0 con 0x55b567fa40b0 2024-03-08T23:16:29.734 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21ca800640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.730+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 0x55b567fac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:16:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.731+0000 7f21c9400640 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fc3750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55b567fbf6d0 tx=0x7f21b8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a10cbf0e1376166f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.731+0000 7f21b6a00640 1 -- 172.21.15.46:0/1648195781 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b8020d50 con 0x55b567fa40b0 2024-03-08T23:16:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.731+0000 7f21b6a00640 1 -- 172.21.15.46:0/1648195781 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21b8012e50 con 0x55b567fa40b0 2024-03-08T23:16:29.735 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.731+0000 7f21b6a00640 1 -- 172.21.15.46:0/1648195781 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80035f0 con 0x55b567fa40b0 2024-03-08T23:16:29.737 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1648195781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 msgr2=0x55b567fc3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fc3750 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x55b567fbf6d0 tx=0x7f21b8015000 comp rx=0 tx=0).stop 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1648195781 shutdown_connections 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567facc90 0x55b567fb3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b567fc3750 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1648195781 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567fa3790 0x55b567fac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1648195781 >> 172.21.15.46:0/1648195781 conn(0x55b567f9dec0 msgr2=0x55b567fa5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.738 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1648195781 shutdown_connections 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.732+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1648195781 wait complete. 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 -- start start 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 Processor -- start 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b568007590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b568011ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b3120 con 0x55b567facc90 2024-03-08T23:16:29.739 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b4400 con 0x55b567fa40b0 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1b3740 con 0x55b567fa3790 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.733+0000 7f21d3bd8740 10 client.0 _ll_register_callbacks cb 0x7f21bc005dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21ca800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b568011ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b568007590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:52390/0 (socket says 172.21.15.46:52390) 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 -- 172.21.15.46:0/1737141607 learned_addr learned my addr 172.21.15.46:0/1737141607 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 -- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 msgr2=0x55b568007590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.740 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b568007590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 -- 172.21.15.46:0/1737141607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 msgr2=0x55b568011ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b568011ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 -- 172.21.15.46:0/1737141607 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b568012420 con 0x55b567fa3790 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9400640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b568007590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21c9e00640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f21bc02c800 tx=0x7f21bc005e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae7f1d59fcc64a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.734+0000 7f21abe00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc045d50 con 0x55b567fa3790 2024-03-08T23:16:29.741 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f21bc050480 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f21b8012e50 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21bc050680 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 10 client.27397 fetch_fsmap learned FSMap version 31 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55b5680136e0 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21b8012e50 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f21bc034440 con 0x55b567fa3790 2024-03-08T23:16:29.742 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 10 client.27397 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 10 client.27397 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 20 client.27397 populate_metadata read hostname 'smithi046' 2024-03-08T23:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b568013bb0 0x55b568015fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 4 client.27397 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:16:29.743 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.735+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b567fbf4f0 con 0x55b568013bb0 2024-03-08T23:16:29.744 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21c9400640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b568013bb0 0x55b568015fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21c9400640 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b568013bb0 0x55b568015fa0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=425a03b83ded85d8 server_cookie=b6b3a72b3e7c5dac in_seq=0 out_seq=0 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21abe00640 10 client.27397 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21abe00640 1 -- 172.21.15.46:0/1737141607 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b567fbf4f0 con 0x55b568013bb0 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21abe00640 10 client.27397 handle_command_reply: tid=0 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.736+0000 7f21d3bd8740 1 client.27397 shutdown 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 msgr2=0x55b568007050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f21bc02c800 tx=0x7f21bc005e80 comp rx=0 tx=0).stop 2024-03-08T23:16:29.745 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 shutdown_connections 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55b568013bb0 0x55b568015fa0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55b567facc90 0x55b568011ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55b567fa40b0 0x55b568007590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 --2- 172.21.15.46:0/1737141607 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55b567fa3790 0x55b568007050 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21abe00640 0 client.27397 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 >> 172.21.15.46:0/1737141607 conn(0x55b567f9dec0 msgr2=0x55b567fc3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 shutdown_connections 2024-03-08T23:16:29.746 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 1 -- 172.21.15.46:0/1737141607 wait complete. 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.737+0000 7f21d3bd8740 20 client.27397 trim_cache size 0 max 16384 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stdout: "6fae20a3-fc2c-4182-84f3-b36637893a67": { 2024-03-08T23:16:29.747 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:16:29.748 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "6fae20a3-fc2c-4182-84f3-b36637893a67", 2024-03-08T23:16:29.748 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:16:29.748 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:16:29.748 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:16:29.748 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 msgr2=0x7f21b00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 0x7f21b00a4eb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55b567f9f050 tx=0x7f21b8004730 comp rx=0 tx=0).stop 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 msgr2=0x7f21cc1a7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f21c4009fd0 tx=0x7f21c400b260 comp rx=0 tx=0).stop 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 shutdown_connections 2024-03-08T23:16:29.749 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f21b00a29f0 0x7f21b00a4eb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f21cc1a6d50 0x7f21cc1a7200 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f21cc110e80 0x7f21cc1a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 --2- 172.21.15.46:0/809149736 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f21cc10be00 0x7f21cc1a24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 >> 172.21.15.46:0/809149736 conn(0x7f21cc06c120 msgr2=0x7f21cc10f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 shutdown_connections 2024-03-08T23:16:29.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:29.738+0000 7f21cb200640 1 -- 172.21.15.46:0/809149736 wait complete. 2024-03-08T23:16:29.758 INFO:tasks.cephfs.filesystem:scrub status for tag:6fae20a3-fc2c-4182-84f3-b36637893a67 - {'path': '/', 'tag': '6fae20a3-fc2c-4182-84f3-b36637893a67', 'options': 'recursive,force'} 2024-03-08T23:16:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: pgmap v270: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:16:30.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:16:30.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:16:30.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:16:30.643 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.644 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:30 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: pgmap v270: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:30.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:16:30.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:16:30.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:16:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:16:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:30 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: pgmap v270: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:30.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:16:30.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:30.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:30 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:16:32.615 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:32 smithi192 ceph-mon[36700]: pgmap v271: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:32.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:32 smithi062 ceph-mon[35761]: pgmap v271: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:32.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:32 smithi046 ceph-mon[31859]: pgmap v271: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:34.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:34 smithi062 ceph-mon[35761]: pgmap v272: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:34.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:34 smithi046 ceph-mon[31859]: pgmap v272: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:34.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:34 smithi192 ceph-mon[36700]: pgmap v272: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:35.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:35 smithi062 ceph-mon[35761]: pgmap v273: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:35.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:35 smithi046 ceph-mon[31859]: pgmap v273: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:35.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:35 smithi192 ceph-mon[36700]: pgmap v273: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:38.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:37 smithi192 ceph-mon[36700]: pgmap v274: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:38.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:37 smithi062 ceph-mon[35761]: pgmap v274: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:38.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:37 smithi046 ceph-mon[31859]: pgmap v274: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:40.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:39 smithi192 ceph-mon[36700]: pgmap v275: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:40.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:39 smithi062 ceph-mon[35761]: pgmap v275: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:40.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:39 smithi046 ceph-mon[31859]: pgmap v275: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:42.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:41 smithi192 ceph-mon[36700]: pgmap v276: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:42.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:41 smithi062 ceph-mon[35761]: pgmap v276: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:42.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:41 smithi046 ceph-mon[31859]: pgmap v276: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:16:43.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:42 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:43.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:42 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:43.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:42 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:43.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:42 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:43.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:42 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:43.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:42 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:44.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:43 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:16:44.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:43 smithi192 ceph-mon[36700]: pgmap v277: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:44.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:43 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:16:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:43 smithi062 ceph-mon[35761]: pgmap v277: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:44.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:43 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:16:44.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:43 smithi046 ceph-mon[31859]: pgmap v277: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:16:46.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:45 smithi192 ceph-mon[36700]: pgmap v278: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:46.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:45 smithi062 ceph-mon[35761]: pgmap v278: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:46.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:45 smithi046 ceph-mon[31859]: pgmap v278: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:48.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:47 smithi192 ceph-mon[36700]: pgmap v279: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:48.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:47 smithi062 ceph-mon[35761]: pgmap v279: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:48.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:47 smithi046 ceph-mon[31859]: pgmap v279: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:50.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:49 smithi192 ceph-mon[36700]: pgmap v280: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:50.142 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:49 smithi062 ceph-mon[35761]: pgmap v280: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:50.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:49 smithi046 ceph-mon[31859]: pgmap v280: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:52.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:51 smithi192 ceph-mon[36700]: pgmap v281: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:52.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:51 smithi062 ceph-mon[35761]: pgmap v281: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:52.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:51 smithi046 ceph-mon[31859]: pgmap v281: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:54.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:53 smithi062 ceph-mon[35761]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:54.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:53 smithi046 ceph-mon[31859]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:54.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:53 smithi192 ceph-mon[36700]: pgmap v282: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:16:56.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:55 smithi062 ceph-mon[35761]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:56.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:55 smithi046 ceph-mon[31859]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:56.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:55 smithi192 ceph-mon[36700]: pgmap v283: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:58.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:57 smithi062 ceph-mon[35761]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:58.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:57 smithi046 ceph-mon[31859]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:58.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:57 smithi192 ceph-mon[36700]: pgmap v284: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:16:59.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:16:59.759 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:16:59.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 -- 172.21.15.46:0/3441062595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 msgr2=0x7f5968102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:16:59.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 --2- 172.21.15.46:0/3441062595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f5968102700 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5958009900 tx=0x7f595802bcc0 comp rx=0 tx=0).stop 2024-03-08T23:16:59.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 -- 172.21.15.46:0/3441062595 shutdown_connections 2024-03-08T23:16:59.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 --2- 172.21.15.46:0/3441062595 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 0x7f596810dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:59.996 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 --2- 172.21.15.46:0/3441062595 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596810b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:59.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 --2- 172.21.15.46:0/3441062595 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f5968102700 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:16:59.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 -- 172.21.15.46:0/3441062595 >> 172.21.15.46:0/3441062595 conn(0x7f59680fbb40 msgr2=0x7f59680fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:16:59.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 -- 172.21.15.46:0/3441062595 shutdown_connections 2024-03-08T23:16:59.997 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.992+0000 7f596d800640 1 -- 172.21.15.46:0/3441062595 wait complete. 2024-03-08T23:16:59.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.993+0000 7f596d800640 1 Processor -- start 2024-03-08T23:16:59.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 -- start start 2024-03-08T23:16:59.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f596819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:16:59.999 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 0x7f5968199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5968111550 con 0x7f5968102320 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681113d0 con 0x7f596810b740 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f596d800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681116d0 con 0x7f5968102c40 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.994+0000 7f5967400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39680/0 (socket says 172.21.15.46:39680) 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 -- 172.21.15.46:0/1023476945 learned_addr learned my addr 172.21.15.46:0/1023476945 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.000 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f596ce00640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 0x7f5968199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967e00640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f596819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 -- 172.21.15.46:0/1023476945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 msgr2=0x7f5968199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 0x7f5968199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 -- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 msgr2=0x7f596819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f596819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 -- 172.21.15.46:0/1023476945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596819a1e0 con 0x7f5968102c40 2024-03-08T23:17:00.001 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.995+0000 7f5967400640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f595000ed00 tx=0x7f595000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.996+0000 7f5964c00640 1 -- 172.21.15.46:0/1023476945 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950003b70 con 0x7f5968102c40 2024-03-08T23:17:00.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.996+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596819a4d0 con 0x7f5968102c40 2024-03-08T23:17:00.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.996+0000 7f5964c00640 1 -- 172.21.15.46:0/1023476945 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5950003d10 con 0x7f5968102c40 2024-03-08T23:17:00.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.996+0000 7f5964c00640 1 -- 172.21.15.46:0/1023476945 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59500108f0 con 0x7f5968102c40 2024-03-08T23:17:00.002 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.996+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59681a6770 con 0x7f5968102c40 2024-03-08T23:17:00.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.999+0000 7f5964c00640 1 -- 172.21.15.46:0/1023476945 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5950004000 con 0x7f5968102c40 2024-03-08T23:17:00.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.999+0000 7f5971bcf740 1 Processor -- start 2024-03-08T23:17:00.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:16:59.999+0000 7f5971bcf740 1 -- start start 2024-03-08T23:17:00.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a7790 0x55efee6a7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.005 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6b0c90 0x55efee6b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee669410 con 0x55efee6a80b0 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee60bd50 con 0x55efee6a7790 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5971bcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee5b4b80 con 0x55efee6b0c90 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5967e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5967e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60038/0 (socket says 172.21.15.46:60038) 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5967e00640 1 -- 172.21.15.46:0/170772264 learned_addr learned my addr 172.21.15.46:0/170772264 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.006 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f596ce00640 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a7790 0x55efee6a7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5967400640 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6b0c90 0x55efee6b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.000+0000 7f5964c00640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 0x7f59480a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 -- 172.21.15.46:0/170772264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6b0c90 msgr2=0x55efee6b3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6b0c90 0x55efee6b3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 -- 172.21.15.46:0/170772264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a7790 msgr2=0x55efee6a7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a7790 0x55efee6a7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 -- 172.21.15.46:0/170772264 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efee3ea7d0 con 0x55efee6a80b0 2024-03-08T23:17:00.007 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5964c00640 1 -- 172.21.15.46:0/1023476945 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f59500cf7a0 con 0x7f5968102c40 2024-03-08T23:17:00.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f596ce00640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 0x7f59480a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f5967e00640 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5958000c00 tx=0x7f5958009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e62a01ea07e009 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f595d600640 1 -- 172.21.15.46:0/170772264 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958043070 con 0x55efee6a80b0 2024-03-08T23:17:00.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f595d600640 1 -- 172.21.15.46:0/170772264 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f595802c6e0 con 0x55efee6a80b0 2024-03-08T23:17:00.008 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.001+0000 7f595d600640 1 -- 172.21.15.46:0/170772264 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958004a40 con 0x55efee6a80b0 2024-03-08T23:17:00.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 -- 172.21.15.46:0/170772264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 msgr2=0x55efee6b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f5958000c00 tx=0x7f5958009560 comp rx=0 tx=0).stop 2024-03-08T23:17:00.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f596ce00640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 0x7f59480a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59600058c0 tx=0x7f59600046c0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 -- 172.21.15.46:0/170772264 shutdown_connections 2024-03-08T23:17:00.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6b0c90 0x55efee6b3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6a80b0 0x55efee6b0750 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 --2- 172.21.15.46:0/170772264 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a7790 0x55efee6a7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 -- 172.21.15.46:0/170772264 >> 172.21.15.46:0/170772264 conn(0x55efee6a1ec0 msgr2=0x55efee6a22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 -- 172.21.15.46:0/170772264 shutdown_connections 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.002+0000 7f5971bcf740 1 -- 172.21.15.46:0/170772264 wait complete. 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.003+0000 7f5971bcf740 1 -- start start 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.003+0000 7f5971bcf740 1 Processor -- start 2024-03-08T23:17:00.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.003+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.003+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5971bcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee5c7ee0 con 0x55efee6b0c90 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5971bcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee607df0 con 0x55efee6a80b0 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5971bcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efee6b3a20 con 0x55efee6a7790 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5971bcf740 10 client.0 _ll_register_callbacks cb 0x55efee5ce8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:00.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f596ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39704/0 (socket says 172.21.15.46:39704) 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 -- 172.21.15.46:0/2207749226 learned_addr learned my addr 172.21.15.46:0/2207749226 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967400640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 -- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 msgr2=0x55efee6ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 -- 172.21.15.46:0/2207749226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 msgr2=0x55efee6ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6ae6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.016 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967e00640 1 -- 172.21.15.46:0/2207749226 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efee3203f0 con 0x55efee6a7790 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.004+0000 7f5967400640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5967e00640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5958036040 tx=0x7f59580047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e5e57e46e40ac89 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5956000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958043050 con 0x55efee6a7790 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f595802c6e0 con 0x55efee6a7790 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f59580050e0 con 0x55efee6a7790 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5958058050 con 0x55efee6a7790 2024-03-08T23:17:00.017 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5971bcf740 10 client.27409 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.005+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55efee2e00f0 con 0x55efee6a7790 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f595802c6e0 con 0x55efee6a7790 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5958048b90 con 0x55efee6a7790 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 10 client.27409 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 10 client.27409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 20 client.27409 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee6c0c60 0x55efee6c3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 4 client.27409 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efee619290 con 0x55efee6c0c60 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.006+0000 7f5967400640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee6c0c60 0x55efee6c3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.007+0000 7f5967400640 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee6c0c60 0x55efee6c3050 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=81e6067441cd8a51 server_cookie=2f9e0a3722e9cd0a in_seq=0 out_seq=0 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.007+0000 7f5956000640 10 client.27409 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.007+0000 7f5956000640 1 -- 172.21.15.46:0/2207749226 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55efee619290 con 0x55efee6c0c60 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.007+0000 7f5956000640 10 client.27409 handle_command_reply: tid=0 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.007+0000 7f5971bcf740 1 client.27409 shutdown 2024-03-08T23:17:00.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 msgr2=0x55efee6adc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5958036040 tx=0x7f59580047a0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 shutdown_connections 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee6c0c60 0x55efee6c3050 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2207749226 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6adc50 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5956000640 0 client.27409 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:00.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 >> 172.21.15.46:0/2207749226 conn(0x55efee6a1ec0 msgr2=0x55efee6a3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 shutdown_connections 2024-03-08T23:17:00.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 1 -- 172.21.15.46:0/2207749226 wait complete. 2024-03-08T23:17:00.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.008+0000 7f5971bcf740 20 client.27409 trim_cache size 0 max 16384 2024-03-08T23:17:00.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 Processor -- start 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 -- start start 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6b7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681b2b50 con 0x55efee6b0c90 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681b06f0 con 0x55efee6a80b0 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5971bcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681ae7b0 con 0x55efee6a7790 2024-03-08T23:17:00.029 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5967e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f596ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f596ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60058/0 (socket says 172.21.15.46:60058) 2024-03-08T23:17:00.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5967e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39708/0 (socket says 172.21.15.46:39708) 2024-03-08T23:17:00.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f596ce00640 1 -- 172.21.15.46:0/1415386540 learned_addr learned my addr 172.21.15.46:0/1415386540 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.025+0000 7f5967400640 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6b7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 -- 172.21.15.46:0/1415386540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 msgr2=0x55efee6b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.035 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 -- 172.21.15.46:0/1415386540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 msgr2=0x55efee6b7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6b7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 -- 172.21.15.46:0/1415386540 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efee6b3080 con 0x55efee6b0c90 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f5967400640 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f596ce00640 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f596001f570 tx=0x7f5960013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6047910277b8f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.026+0000 7f595cc00640 1 -- 172.21.15.46:0/1415386540 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960005020 con 0x55efee6b0c90 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f595cc00640 1 -- 172.21.15.46:0/1415386540 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59600047a0 con 0x55efee6b0c90 2024-03-08T23:17:00.036 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f595cc00640 1 -- 172.21.15.46:0/1415386540 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f596001c750 con 0x55efee6b0c90 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 -- 172.21.15.46:0/1415386540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 msgr2=0x55efee6b81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f596001f570 tx=0x7f5960013030 comp rx=0 tx=0).stop 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 -- 172.21.15.46:0/1415386540 shutdown_connections 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee6b81a0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee6b7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 --2- 172.21.15.46:0/1415386540 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee6b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 -- 172.21.15.46:0/1415386540 >> 172.21.15.46:0/1415386540 conn(0x55efee6a1ec0 msgr2=0x55efee6c7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.037 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 -- 172.21.15.46:0/1415386540 shutdown_connections 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.027+0000 7f5971bcf740 1 -- 172.21.15.46:0/1415386540 wait complete. 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 -- start start 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 Processor -- start 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee70af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee70b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681aedb0 con 0x55efee6b0c90 2024-03-08T23:17:00.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681b0090 con 0x55efee6a80b0 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681af3d0 con 0x55efee6a7790 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.028+0000 7f5971bcf740 10 client.0 _ll_register_callbacks cb 0x7f59500093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f596ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee70af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f596ce00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60070/0 (socket says 172.21.15.46:60070) 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f596ce00640 1 -- 172.21.15.46:0/2310698085 learned_addr learned my addr 172.21.15.46:0/2310698085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967400640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee70b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 -- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 msgr2=0x55efee70b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee70b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 -- 172.21.15.46:0/2310698085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 msgr2=0x55efee70fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 -- 172.21.15.46:0/2310698085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efee6ada20 con 0x55efee6a7790 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967400640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee70b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f596ce00640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5967e00640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee70af10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5958009800 tx=0x7f5958004900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16920025bc47d5ca server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.029+0000 7f5956000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:00.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958061060 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f59600047a0 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f59580050e0 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5958068050 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5971bcf740 10 client.27415 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55efee717600 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.030+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59600047a0 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5958048a50 con 0x55efee6a7790 2024-03-08T23:17:00.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 10 client.27415 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 10 client.27415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 20 client.27415 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee717be0 0x55efee71a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 4 client.27415 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55efee6135e0 con 0x55efee717be0 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5967400640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee717be0 0x55efee71a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5967400640 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee717be0 0x55efee71a000 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5400c7d02e7a8ee3 server_cookie=fdd7255a7851f22c in_seq=0 out_seq=0 2024-03-08T23:17:00.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.031+0000 7f5956000640 10 client.27415 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5956000640 1 -- 172.21.15.46:0/2310698085 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55efee6135e0 con 0x55efee717be0 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5956000640 10 client.27415 handle_command_reply: tid=0 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 client.27415 shutdown 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 msgr2=0x55efee70af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee70af10 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f5958009800 tx=0x7f5958004900 comp rx=0 tx=0).stop 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 shutdown_connections 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55efee717be0 0x55efee71a000 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55efee6b0c90 0x55efee70fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55efee6a80b0 0x55efee70b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5956000640 0 client.27415 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 --2- 172.21.15.46:0/2310698085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55efee6a7790 0x55efee70af10 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 >> 172.21.15.46:0/2310698085 conn(0x55efee6a1ec0 msgr2=0x55efee6a8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 shutdown_connections 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 1 -- 172.21.15.46:0/2310698085 wait complete. 2024-03-08T23:17:00.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.032+0000 7f5971bcf740 20 client.27415 trim_cache size 0 max 16384 2024-03-08T23:17:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:00.045 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:17:00.045 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:17:00.045 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 msgr2=0x7f59480a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 0x7f59480a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f59600058c0 tx=0x7f59600046c0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 msgr2=0x7f596819fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f595000ed00 tx=0x7f595000c6a0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.046 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 shutdown_connections 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f59480a2970 0x7f59480a4e30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f596810b740 0x7f5968199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5968102c40 0x7f596819fdf0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 --2- 172.21.15.46:0/1023476945 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5968102320 0x7f596819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 >> 172.21.15.46:0/1023476945 conn(0x7f59680fbb40 msgr2=0x7f596810ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 shutdown_connections 2024-03-08T23:17:00.047 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.033+0000 7f596d800640 1 -- 172.21.15.46:0/1023476945 wait complete. 2024-03-08T23:17:00.054 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:17:00.055 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:17:00.123 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:16:59 smithi046 ceph-mon[31859]: pgmap v285: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:00.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:16:59 smithi062 ceph-mon[35761]: pgmap v285: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 -- 172.21.15.46:0/2317780268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07641120f0 msgr2=0x7f07641144e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 --2- 172.21.15.46:0/2317780268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07641120f0 0x7f07641144e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f075c009ca0 tx=0x7f075c02c180 comp rx=0 tx=0).stop 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 -- 172.21.15.46:0/2317780268 shutdown_connections 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 --2- 172.21.15.46:0/2317780268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f07641120f0 0x7f07641144e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 --2- 172.21.15.46:0/2317780268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0764073fb0 0x7f0764111bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.259 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 --2- 172.21.15.46:0/2317780268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 0x7f0764073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 -- 172.21.15.46:0/2317780268 >> 172.21.15.46:0/2317780268 conn(0x7f07640fcbd0 msgr2=0x7f07640feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 -- 172.21.15.46:0/2317780268 shutdown_connections 2024-03-08T23:17:00.260 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.255+0000 7f0763200640 1 -- 172.21.15.46:0/2317780268 wait complete. 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.256+0000 7f0763200640 1 Processor -- start 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 -- start start 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 0x7f076419dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0764073fb0 0x7f076419e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.262 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764117af0 con 0x7f0764073fb0 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764117970 con 0x7f07641120f0 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0763200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0764117c70 con 0x7f0764073690 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0759400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0764073fb0 0x7f076419e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0761e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 0x7f076419dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0762800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0762800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45750/0 (socket says 172.21.15.46:45750) 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.257+0000 7f0762800640 1 -- 172.21.15.46:0/1434991140 learned_addr learned my addr 172.21.15.46:0/1434991140 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 -- 172.21.15.46:0/1434991140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 msgr2=0x7f076419dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.263 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 0x7f076419dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 -- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0764073fb0 msgr2=0x7f076419e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0764073fb0 0x7f076419e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 -- 172.21.15.46:0/1434991140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07641a2f90 con 0x7f07641120f0 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f0762800640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f075c005e30 tx=0x7f075c035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.258+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c039040 con 0x7f07641120f0 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.259+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f075c044070 con 0x7f07641120f0 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.259+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07641a3220 con 0x7f07641120f0 2024-03-08T23:17:00.264 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.259+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c0328d0 con 0x7f07641120f0 2024-03-08T23:17:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.259+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07641aab10 con 0x7f07641120f0 2024-03-08T23:17:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.260+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0764068000 con 0x7f07641120f0 2024-03-08T23:17:00.265 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.262+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f075c071bc0 con 0x7f07641120f0 2024-03-08T23:17:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.263+0000 7f075b400640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 0x7f07480a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.266 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.263+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f075c0f1470 con 0x7f07641120f0 2024-03-08T23:17:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.264+0000 7f0761e00640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 0x7f07480a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.264+0000 7f0761e00640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 0x7f07480a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0754009bd0 tx=0x7f0754005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.266+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f075c0bdec0 con 0x7f07641120f0 2024-03-08T23:17:00.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:16:59 smithi192 ceph-mon[36700]: pgmap v285: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:00.513 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.510+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0764104570 con 0x7f07641120f0 2024-03-08T23:17:00.514 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.511+0000 7f075b400640 1 -- 172.21.15.46:0/1434991140 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f075c0bd7c0 con 0x7f07641120f0 2024-03-08T23:17:00.515 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:17:00.515 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:17:00.515 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:17:00.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 msgr2=0x7f07480a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 0x7f07480a4eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0754009bd0 tx=0x7f0754005f90 comp rx=0 tx=0).stop 2024-03-08T23:17:00.517 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 msgr2=0x7f07641a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f075c005e30 tx=0x7f075c035d80 comp rx=0 tx=0).stop 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 shutdown_connections 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f07480a29f0 0x7f07480a4eb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f07641120f0 0x7f07641a2860 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0764073fb0 0x7f076419e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 --2- 172.21.15.46:0/1434991140 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0764073690 0x7f076419dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 >> 172.21.15.46:0/1434991140 conn(0x7f07640fcbd0 msgr2=0x7f07640fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 shutdown_connections 2024-03-08T23:17:00.518 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.514+0000 7f0763200640 1 -- 172.21.15.46:0/1434991140 wait complete. 2024-03-08T23:17:00.537 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:00.537 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:17:00.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 -- 172.21.15.46:0/3076390112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c8109570 msgr2=0x7ff2c8109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 --2- 172.21.15.46:0/3076390112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c8109570 0x7ff2c8109950 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8009900 tx=0x7ff2b802bc30 comp rx=0 tx=0).stop 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 -- 172.21.15.46:0/3076390112 shutdown_connections 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 --2- 172.21.15.46:0/3076390112 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 0x7ff2c8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 --2- 172.21.15.46:0/3076390112 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c81034b0 0x7ff2c8103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 --2- 172.21.15.46:0/3076390112 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c8109570 0x7ff2c8109950 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 -- 172.21.15.46:0/3076390112 >> 172.21.15.46:0/3076390112 conn(0x7ff2c80fef60 msgr2=0x7ff2c8101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.787 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 -- 172.21.15.46:0/3076390112 shutdown_connections 2024-03-08T23:17:00.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.782+0000 7ff2cd200640 1 -- 172.21.15.46:0/3076390112 wait complete. 2024-03-08T23:17:00.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.783+0000 7ff2cd200640 1 Processor -- start 2024-03-08T23:17:00.788 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 -- start start 2024-03-08T23:17:00.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 0x7ff2c819fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c8109570 0x7ff2c8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.790 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8113880 con 0x7ff2c81034b0 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8113700 con 0x7ff2c8103e70 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2cd200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c8113a00 con 0x7ff2c8109570 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60086/0 (socket says 172.21.15.46:60086) 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1092120838 learned_addr learned my addr 172.21.15.46:0/1092120838 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2c5400640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 0x7ff2c819fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.784+0000 7ff2c6800640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c8109570 0x7ff2c8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.791 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1092120838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c8109570 msgr2=0x7ff2c8199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c8109570 0x7ff2c8199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1092120838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 msgr2=0x7ff2c819fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 0x7ff2c819fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1092120838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c819a240 con 0x7ff2c81034b0 2024-03-08T23:17:00.792 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff2b802efd0 tx=0x7ff2b80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2bea00640 1 -- 172.21.15.46:0/1092120838 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8041070 con 0x7ff2c81034b0 2024-03-08T23:17:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2bea00640 1 -- 172.21.15.46:0/1092120838 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2b802ccc0 con 0x7ff2c81034b0 2024-03-08T23:17:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c819a4d0 con 0x7ff2c81034b0 2024-03-08T23:17:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2bea00640 1 -- 172.21.15.46:0/1092120838 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8003d10 con 0x7ff2c81034b0 2024-03-08T23:17:00.793 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.785+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c81a67d0 con 0x7ff2c81034b0 2024-03-08T23:17:00.800 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.788+0000 7ff2bea00640 1 -- 172.21.15.46:0/1092120838 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff2b8044070 con 0x7ff2c81034b0 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.788+0000 7ff2d0018740 1 Processor -- start 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 -- start start 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919b350 0x55801919b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580190ffdd0 con 0x55801919bc70 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580190a8c70 con 0x55801919b350 2024-03-08T23:17:00.801 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2d0018740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580190fbe70 con 0x5580191a4850 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2c6800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919b350 0x55801919b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39764/0 (socket says 172.21.15.46:39764) 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.789+0000 7ff2c5400640 1 -- 172.21.15.46:0/2518983982 learned_addr learned my addr 172.21.15.46:0/2518983982 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 -- 172.21.15.46:0/2518983982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919b350 msgr2=0x55801919b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.802 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919b350 0x55801919b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 -- 172.21.15.46:0/2518983982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 msgr2=0x5580191a4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 -- 172.21.15.46:0/2518983982 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580191622b0 con 0x5580191a4850 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2bea00640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 0x7ff2ac0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c5400640 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009c60 tx=0x7ff2b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e53fb54fde3e8cd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2bea00640 1 -- 172.21.15.46:0/1092120838 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff2b80f0af0 con 0x7ff2c81034b0 2024-03-08T23:17:00.803 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2c6800640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 0x7ff2ac0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.790+0000 7ff2b3400640 1 -- 172.21.15.46:0/2518983982 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b4004230 con 0x5580191a4850 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2b3400640 1 -- 172.21.15.46:0/2518983982 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4009e10 con 0x5580191a4850 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2b3400640 1 -- 172.21.15.46:0/2518983982 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40198a0 con 0x5580191a4850 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 -- 172.21.15.46:0/2518983982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 msgr2=0x5580191a6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2c6800640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 0x7ff2ac0a4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0005fd0 tx=0x7ff2c0009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009c60 tx=0x7ff2b4011810 comp rx=0 tx=0).stop 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 -- 172.21.15.46:0/2518983982 shutdown_connections 2024-03-08T23:17:00.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a6c40 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 --2- 172.21.15.46:0/2518983982 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919b350 0x55801919b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.791+0000 7ff2d0018740 1 -- 172.21.15.46:0/2518983982 >> 172.21.15.46:0/2518983982 conn(0x558019195a80 msgr2=0x558019195e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.792+0000 7ff2d0018740 1 -- 172.21.15.46:0/2518983982 shutdown_connections 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.792+0000 7ff2d0018740 1 -- 172.21.15.46:0/2518983982 wait complete. 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.792+0000 7ff2d0018740 1 -- start start 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.792+0000 7ff2d0018740 1 Processor -- start 2024-03-08T23:17:00.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919bc70 0x5580191a1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580191a7640 con 0x55801919b350 2024-03-08T23:17:00.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580190bc0c0 con 0x55801919bc70 2024-03-08T23:17:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5580191a77c0 con 0x5580191a4850 2024-03-08T23:17:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2d0018740 10 client.0 _ll_register_callbacks cb 0x5580190f6ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.807 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c6800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919bc70 0x5580191a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60092/0 (socket says 172.21.15.46:60092) 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1961742418 learned_addr learned my addr 172.21.15.46:0/1961742418 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 msgr2=0x5580191a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1961742418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919bc70 msgr2=0x5580191a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919bc70 0x5580191a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.793+0000 7ff2c5e00640 1 -- 172.21.15.46:0/1961742418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558018e143f0 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8006010 tx=0x7ff2b8005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70672f6ccb1a380d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80bf020 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff2b80baf90 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2b4009e10 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2b804b070 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2d0018740 10 client.17319 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.794+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558018ede7d0 con 0x55801919b350 2024-03-08T23:17:00.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4009e10 con 0x55801919b350 2024-03-08T23:17:00.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff2b80ee0a0 con 0x55801919b350 2024-03-08T23:17:00.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 10 client.17319 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:00.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 10 client.17319 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:17:00.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 20 client.17319 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:00.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5580191b4730 0x5580191b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 4 client.17319 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5580190d44f0 con 0x5580191b4730 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.795+0000 7ff2c5400640 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5580191b4730 0x5580191b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.796+0000 7ff2c5400640 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5580191b4730 0x5580191b6b60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8689e235f29f5b28 server_cookie=7c5222987773d159 in_seq=0 out_seq=0 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.796+0000 7ff2a7e00640 10 client.17319 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2a7e00640 1 -- 172.21.15.46:0/1961742418 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5580190d44f0 con 0x5580191b4730 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2a7e00640 10 client.17319 handle_command_reply: tid=0 2024-03-08T23:17:00.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 client.17319 shutdown 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 msgr2=0x5580191a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8006010 tx=0x7ff2b8005670 comp rx=0 tx=0).stop 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 shutdown_connections 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5580191b4730 0x5580191b6b60 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5580191a4850 0x5580191a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55801919bc70 0x5580191a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.812 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 --2- 172.21.15.46:0/1961742418 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919b350 0x5580191a40f0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2a7e00640 0 client.17319 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:00.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 >> 172.21.15.46:0/1961742418 conn(0x558019195a80 msgr2=0x5580191977d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 shutdown_connections 2024-03-08T23:17:00.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 1 -- 172.21.15.46:0/1961742418 wait complete. 2024-03-08T23:17:00.813 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.797+0000 7ff2d0018740 20 client.17319 trim_cache size 0 max 16384 2024-03-08T23:17:00.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.810+0000 7ff2d0018740 1 Processor -- start 2024-03-08T23:17:00.816 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.810+0000 7ff2d0018740 1 -- start start 2024-03-08T23:17:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191a0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81b2b40 con 0x55801919bc70 2024-03-08T23:17:00.817 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81b06e0 con 0x5580191a4850 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2d0018740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81ae7a0 con 0x55801919b350 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60094/0 (socket says 172.21.15.46:60094) 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 -- 172.21.15.46:0/655718907 learned_addr learned my addr 172.21.15.46:0/655718907 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c6800640 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.818 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 -- 172.21.15.46:0/655718907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 msgr2=0x5580191a0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.819 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191a0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 -- 172.21.15.46:0/655718907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 msgr2=0x5580191a40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191a40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.811+0000 7ff2c5400640 1 -- 172.21.15.46:0/655718907 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5580191958b0 con 0x55801919bc70 2024-03-08T23:17:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.812+0000 7ff2c5400640 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009c60 tx=0x7ff2b4019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f875bbae111970c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.820 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.812+0000 7ff2b2a00640 1 -- 172.21.15.46:0/655718907 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b401f040 con 0x55801919bc70 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.812+0000 7ff2b2a00640 1 -- 172.21.15.46:0/655718907 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4009e10 con 0x55801919bc70 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.812+0000 7ff2b2a00640 1 -- 172.21.15.46:0/655718907 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b40194f0 con 0x55801919bc70 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- 172.21.15.46:0/655718907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 msgr2=0x5580191a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff2b4009c60 tx=0x7ff2b4019480 comp rx=0 tx=0).stop 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- 172.21.15.46:0/655718907 shutdown_connections 2024-03-08T23:17:00.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191a40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191a3bb0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 --2- 172.21.15.46:0/655718907 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191a0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- 172.21.15.46:0/655718907 >> 172.21.15.46:0/655718907 conn(0x558019195a80 msgr2=0x55801919fd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- 172.21.15.46:0/655718907 shutdown_connections 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- 172.21.15.46:0/655718907 wait complete. 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 -- start start 2024-03-08T23:17:00.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.813+0000 7ff2d0018740 1 Processor -- start 2024-03-08T23:17:00.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191abd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191aa660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81aeda0 con 0x55801919bc70 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81b0080 con 0x5580191a4850 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c81af3c0 con 0x55801919b350 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2d0018740 10 client.0 _ll_register_callbacks cb 0x7ff2b4009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:00.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2c5e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191abd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2c5400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60100/0 (socket says 172.21.15.46:60100) 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2c5400640 1 -- 172.21.15.46:0/3879240175 learned_addr learned my addr 172.21.15.46:0/3879240175 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.814+0000 7ff2c6800640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191aa660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 -- 172.21.15.46:0/3879240175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 msgr2=0x5580191aa660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191aa660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.829 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 -- 172.21.15.46:0/3879240175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 msgr2=0x5580191aa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:00.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 -- 172.21.15.46:0/3879240175 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5580191aae60 con 0x55801919b350 2024-03-08T23:17:00.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5400640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:00.830 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2c5e00640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191abd50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8003fd0 tx=0x7ff2b80bb2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64e843ef85e8e17b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:00.831 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2a7e00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80bf070 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff2b80f4230 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff2b4009e10 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2b80bca30 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2d0018740 10 client.27439 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.815+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55801920bbb0 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff2b4009e10 con 0x55801919b350 2024-03-08T23:17:00.832 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff2b80bc3d0 con 0x55801919b350 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 10 client.27439 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 10 client.27439 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 20 client.27439 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55801920c090 0x55801920e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 4 client.27439 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55801910d310 con 0x55801920c090 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.816+0000 7ff2c5400640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55801920c090 0x55801920e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2c5400640 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55801920c090 0x55801920e480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d529a1c26bd2b4dd server_cookie=4b9c78a3ecf81f90 in_seq=0 out_seq=0 2024-03-08T23:17:00.833 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2a7e00640 10 client.27439 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2a7e00640 1 -- 172.21.15.46:0/3879240175 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55801910d310 con 0x55801920c090 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2a7e00640 10 client.27439 handle_command_reply: tid=0 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2d0018740 1 client.27439 shutdown 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 msgr2=0x5580191abd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191abd50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff2b8003fd0 tx=0x7ff2b80bb2c0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.817+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 shutdown_connections 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55801920c090 0x55801920e480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.834 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5580191a4850 0x5580191aa660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55801919bc70 0x5580191aa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2a7e00640 0 client.27439 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 --2- 172.21.15.46:0/3879240175 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55801919b350 0x5580191abd50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 >> 172.21.15.46:0/3879240175 conn(0x558019195a80 msgr2=0x5580191a05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 shutdown_connections 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 1 -- 172.21.15.46:0/3879240175 wait complete. 2024-03-08T23:17:00.835 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2d0018740 20 client.27439 trim_cache size 0 max 16384 2024-03-08T23:17:00.836 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:00.836 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 msgr2=0x7ff2ac0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 0x7ff2ac0a4eb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7ff2c0005fd0 tx=0x7ff2c0009500 comp rx=0 tx=0).stop 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 msgr2=0x7ff2c819f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.818+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7ff2b802efd0 tx=0x7ff2b80042f0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 shutdown_connections 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff2ac0a29f0 0x7ff2ac0a4eb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff2c8109570 0x7ff2c8199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.837 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff2c8103e70 0x7ff2c819fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 --2- 172.21.15.46:0/1092120838 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff2c81034b0 0x7ff2c819f910 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 >> 172.21.15.46:0/1092120838 conn(0x7ff2c80fef60 msgr2=0x7ff2c8100580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 shutdown_connections 2024-03-08T23:17:00.838 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:00.819+0000 7ff2cd200640 1 -- 172.21.15.46:0/1092120838 wait complete. 2024-03-08T23:17:00.844 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:00.844 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:17:01.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 -- 172.21.15.46:0/3490174367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8103f90 msgr2=0x7ff6f8107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/3490174367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8103f90 0x7ff6f8107fd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff6e4009860 tx=0x7ff6e402bb70 comp rx=0 tx=0).stop 2024-03-08T23:17:01.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 -- 172.21.15.46:0/3490174367 shutdown_connections 2024-03-08T23:17:01.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/3490174367 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f81086b0 0x7ff6f8110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/3490174367 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f8103f90 0x7ff6f8107fd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/3490174367 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f81039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.057+0000 7ff6f7e00640 1 -- 172.21.15.46:0/3490174367 >> 172.21.15.46:0/3490174367 conn(0x7ff6f80fee20 msgr2=0x7ff6f8101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.058+0000 7ff6f7e00640 1 -- 172.21.15.46:0/3490174367 shutdown_connections 2024-03-08T23:17:01.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.058+0000 7ff6f7e00640 1 -- 172.21.15.46:0/3490174367 wait complete. 2024-03-08T23:17:01.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.059+0000 7ff6f7e00640 1 Processor -- start 2024-03-08T23:17:01.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.059+0000 7ff6f7e00640 1 -- start start 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f81086b0 0x7ff6f819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8199aa0 0x7ff6f8199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f8113720 con 0x7ff6f81035e0 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81135a0 con 0x7ff6f8199aa0 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81138a0 con 0x7ff6f81086b0 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60104/0 (socket says 172.21.15.46:60104) 2024-03-08T23:17:01.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1679810256 learned_addr learned my addr 172.21.15.46:0/1679810256 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f81086b0 0x7ff6f819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8199aa0 0x7ff6f8199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1679810256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f81086b0 msgr2=0x7ff6f819fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f81086b0 0x7ff6f819fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1679810256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8199aa0 msgr2=0x7ff6f8199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8199aa0 0x7ff6f8199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.060+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1679810256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f819a5c0 con 0x7ff6f81035e0 2024-03-08T23:17:01.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800e9c0 tx=0x7ff6e800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6ef400640 1 -- 172.21.15.46:0/1679810256 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e800cde0 con 0x7ff6f81035e0 2024-03-08T23:17:01.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f819a8b0 con 0x7ff6f81035e0 2024-03-08T23:17:01.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6ef400640 1 -- 172.21.15.46:0/1679810256 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e80035f0 con 0x7ff6f81035e0 2024-03-08T23:17:01.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f81a6b50 con 0x7ff6f81035e0 2024-03-08T23:17:01.067 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.061+0000 7ff6ef400640 1 -- 172.21.15.46:0/1679810256 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e8018630 con 0x7ff6f81035e0 2024-03-08T23:17:01.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.064+0000 7ff70080c740 1 Processor -- start 2024-03-08T23:17:01.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.064+0000 7ff6ef400640 1 -- 172.21.15.46:0/1679810256 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff6e80037a0 con 0x7ff6f81035e0 2024-03-08T23:17:01.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 -- start start 2024-03-08T23:17:01.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4d9530 0x55829e4d9910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.068 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 0x55829e4e24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e43ddd0 con 0x55829e4d9e50 2024-03-08T23:17:01.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e3e6c70 con 0x55829e4e2a30 2024-03-08T23:17:01.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff70080c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e439e70 con 0x55829e4d9530 2024-03-08T23:17:01.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.071 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff6f6000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45810/0 (socket says 172.21.15.46:45810) 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.065+0000 7ff6f6000640 1 -- 172.21.15.46:0/3968750398 learned_addr learned my addr 172.21.15.46:0/3968750398 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f7400640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4d9530 0x55829e4d9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 0x55829e4e24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 -- 172.21.15.46:0/3968750398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4d9530 msgr2=0x55829e4d9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4d9530 0x55829e4d9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 -- 172.21.15.46:0/3968750398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 msgr2=0x55829e4e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 0x55829e4e24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.072 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 -- 172.21.15.46:0/3968750398 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55829e4a02b0 con 0x55829e4e2a30 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 0x55829e4e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6ef400640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 0x7ff6dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.066+0000 7ff6f6000640 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff6e402c180 tx=0x7ff6e4009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f886598e96ad9fc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.067+0000 7ff6ef400640 1 -- 172.21.15.46:0/1679810256 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff6e80cf4e0 con 0x7ff6f81035e0 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.067+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 0x7ff6dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.067+0000 7ff6e3e00640 1 -- 172.21.15.46:0/3968750398 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4043020 con 0x55829e4e2a30 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.067+0000 7ff6e3e00640 1 -- 172.21.15.46:0/3968750398 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e4034cb0 con 0x55829e4e2a30 2024-03-08T23:17:01.073 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.067+0000 7ff6e3e00640 1 -- 172.21.15.46:0/3968750398 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4039040 con 0x55829e4e2a30 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 -- 172.21.15.46:0/3968750398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 msgr2=0x55829e4e4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff6e402c180 tx=0x7ff6e4009510 comp rx=0 tx=0).stop 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 0x7ff6dc0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0005fd0 tx=0x7ff6f0009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 -- 172.21.15.46:0/3968750398 shutdown_connections 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4e2a30 0x55829e4e4e20 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9e50 0x55829e4e24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 --2- 172.21.15.46:0/3968750398 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4d9530 0x55829e4d9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 -- 172.21.15.46:0/3968750398 >> 172.21.15.46:0/3968750398 conn(0x55829e4d3c60 msgr2=0x55829e4d4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.074 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 -- 172.21.15.46:0/3968750398 shutdown_connections 2024-03-08T23:17:01.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.068+0000 7ff70080c740 1 -- 172.21.15.46:0/3968750398 wait complete. 2024-03-08T23:17:01.075 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 -- start start 2024-03-08T23:17:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 Processor -- start 2024-03-08T23:17:01.078 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4df6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e4e57c0 con 0x55829e4d9530 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e3fa0c0 con 0x55829e4d9e50 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55829e4e5940 con 0x55829e4e2a30 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff70080c740 10 client.0 _ll_register_callbacks cb 0x55829e434ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.069+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.079 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60120/0 (socket says 172.21.15.46:60120) 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1843349544 learned_addr learned my addr 172.21.15.46:0/1843349544 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4df6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1843349544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 msgr2=0x55829e4dfc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dfc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1843349544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 msgr2=0x55829e4df6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4df6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1843349544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55829e1523f0 con 0x55829e4d9530 2024-03-08T23:17:01.080 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4df6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800c6a0 tx=0x7ff6e809b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a6edeeee04db9b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e809b720 con 0x55829e4d9530 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff6e80cf170 con 0x55829e4d9530 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.070+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff6e4034cb0 con 0x55829e4d9530 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e809f870 con 0x55829e4d9530 2024-03-08T23:17:01.081 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 10 client.17337 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55829e21c7d0 con 0x55829e4d9530 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e4034cb0 con 0x55829e4d9530 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff6e80a3a30 con 0x55829e4d9530 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 10 client.17337 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 10 client.17337 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 20 client.17337 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e4f29f0 0x55829e4f4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 4 client.17337 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:01.082 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.071+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55829e4124f0 con 0x55829e4f29f0 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.072+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e4f29f0 0x55829e4f4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.072+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e4f29f0 0x55829e4f4de0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2a6c14af9210bb7 server_cookie=de531f8b4d2ef55d in_seq=0 out_seq=0 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.072+0000 7ff6e0c00640 10 client.17337 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1843349544 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55829e4124f0 con 0x55829e4f29f0 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff6e0c00640 10 client.17337 handle_command_reply: tid=0 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 client.17337 shutdown 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 msgr2=0x55829e4e22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.083 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800c6a0 tx=0x7ff6e809b5b0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 shutdown_connections 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e4f29f0 0x55829e4f4de0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dfc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4df6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 --2- 172.21.15.46:0/1843349544 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e22c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff6e0c00640 0 client.17337 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.073+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 >> 172.21.15.46:0/1843349544 conn(0x55829e4d3c60 msgr2=0x55829e4d59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.084 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.074+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 shutdown_connections 2024-03-08T23:17:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.074+0000 7ff70080c740 1 -- 172.21.15.46:0/1843349544 wait complete. 2024-03-08T23:17:01.085 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.074+0000 7ff70080c740 20 client.17337 trim_cache size 0 max 16384 2024-03-08T23:17:01.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 Processor -- start 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 -- start start 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4dc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81b2b70 con 0x55829e4d9530 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81b0710 con 0x55829e4d9e50 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff70080c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81ae7d0 con 0x55829e4e2a30 2024-03-08T23:17:01.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.087+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60124/0 (socket says 172.21.15.46:60124) 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 -- 172.21.15.46:0/3438071797 learned_addr learned my addr 172.21.15.46:0/3438071797 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f7400640 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6000640 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4dc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 -- 172.21.15.46:0/3438071797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 msgr2=0x55829e4dce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 -- 172.21.15.46:0/3438071797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 msgr2=0x55829e4dc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.093 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4dc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 -- 172.21.15.46:0/3438071797 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55829e4d8e50 con 0x55829e4d9530 2024-03-08T23:17:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80980a0 tx=0x7ff6e8099c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20831233f8344208 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6e3400640 1 -- 172.21.15.46:0/3438071797 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80b3070 con 0x55829e4d9530 2024-03-08T23:17:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6e3400640 1 -- 172.21.15.46:0/3438071797 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff6e809b720 con 0x55829e4d9530 2024-03-08T23:17:01.094 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.088+0000 7ff6e3400640 1 -- 172.21.15.46:0/3438071797 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e809ab70 con 0x55829e4d9530 2024-03-08T23:17:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 -- 172.21.15.46:0/3438071797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 msgr2=0x55829e4e1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7ff6e80980a0 tx=0x7ff6e8099c30 comp rx=0 tx=0).stop 2024-03-08T23:17:01.098 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 -- 172.21.15.46:0/3438071797 shutdown_connections 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4dce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4dc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 --2- 172.21.15.46:0/3438071797 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e1df0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 -- 172.21.15.46:0/3438071797 >> 172.21.15.46:0/3438071797 conn(0x55829e4d3c60 msgr2=0x55829e4ddf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 -- 172.21.15.46:0/3438071797 shutdown_connections 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.089+0000 7ff70080c740 1 -- 172.21.15.46:0/3438071797 wait complete. 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 -- start start 2024-03-08T23:17:01.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 Processor -- start 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4e8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4e9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81aedd0 con 0x55829e4d9530 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81b00b0 con 0x55829e4d9e50 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f81af3f0 con 0x55829e4e2a30 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.090+0000 7ff70080c740 10 client.0 _ll_register_callbacks cb 0x7ff6e8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60136/0 (socket says 172.21.15.46:60136) 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1792242633 learned_addr learned my addr 172.21.15.46:0/1792242633 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4e8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4e9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1792242633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 msgr2=0x55829e4e9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4e9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1792242633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 msgr2=0x55829e4e8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4e8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 -- 172.21.15.46:0/1792242633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55829e548f60 con 0x55829e4d9530 2024-03-08T23:17:01.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f7400640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6f6a00640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801b040 tx=0x7ff6e8009630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=433acfbc074e8be7 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.091+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e80b3070 con 0x55829e4d9530 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff6e809ad80 con 0x55829e4d9530 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55829e44b310 con 0x55829e4d9530 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6e8082050 con 0x55829e4d9530 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 10 client.17343 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:01.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55829e54a260 con 0x55829e4d9530 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55829e44b310 con 0x55829e4d9530 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff6e8097500 con 0x55829e4d9530 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 10 client.17343 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 10 client.17343 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 20 client.17343 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e54a810 0x55829e54cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 4 client.17343 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:01.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.092+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55829e445660 con 0x55829e54a810 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e54a810 0x55829e54cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff6f6000640 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e54a810 0x55829e54cc00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d0d5cbc7c530c81 server_cookie=f5f3b6324eb25085 in_seq=0 out_seq=0 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff6e0c00640 10 client.17343 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff6e0c00640 1 -- 172.21.15.46:0/1792242633 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55829e445660 con 0x55829e54a810 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff6e0c00640 10 client.17343 handle_command_reply: tid=0 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.093+0000 7ff70080c740 1 client.17343 shutdown 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 msgr2=0x55829e4e86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff6e801b040 tx=0x7ff6e8009630 comp rx=0 tx=0).stop 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 shutdown_connections 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55829e54a810 0x55829e54cc00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55829e4e2a30 0x55829e4e9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55829e4d9e50 0x55829e4e8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff6e0c00640 0 client.17343 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 --2- 172.21.15.46:0/1792242633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55829e4d9530 0x55829e4e86d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 >> 172.21.15.46:0/1792242633 conn(0x55829e4d3c60 msgr2=0x55829e4da370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 shutdown_connections 2024-03-08T23:17:01.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 1 -- 172.21.15.46:0/1792242633 wait complete. 2024-03-08T23:17:01.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.094+0000 7ff70080c740 20 client.17343 trim_cache size 0 max 16384 2024-03-08T23:17:01.106 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 msgr2=0x7ff6dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 0x7ff6dc0a4e30 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff6f0005fd0 tx=0x7ff6f0009500 comp rx=0 tx=0).stop 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 msgr2=0x7ff6f819f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff6e800e9c0 tx=0x7ff6e800ee80 comp rx=0 tx=0).stop 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 shutdown_connections 2024-03-08T23:17:01.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff6dc0a2970 0x7ff6dc0a4e30 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6f8199aa0 0x7ff6f8199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff6f81086b0 0x7ff6f819fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 --2- 172.21.15.46:0/1679810256 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff6f81035e0 0x7ff6f819f8f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 >> 172.21.15.46:0/1679810256 conn(0x7ff6f80fee20 msgr2=0x7ff6f8100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 shutdown_connections 2024-03-08T23:17:01.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.095+0000 7ff6f7e00640 1 -- 172.21.15.46:0/1679810256 wait complete. 2024-03-08T23:17:01.113 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:00 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1434991140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:01.115 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:01.116 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:17:01.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:00 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1434991140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:01.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:00 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1434991140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.321+0000 7fdb2f200640 1 -- 172.21.15.46:0/846764614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 msgr2=0x7fdb28075c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.321+0000 7fdb2f200640 1 --2- 172.21.15.46:0/846764614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb28075c00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c009900 tx=0x7fdb1c02bc50 comp rx=0 tx=0).stop 2024-03-08T23:17:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 -- 172.21.15.46:0/846764614 shutdown_connections 2024-03-08T23:17:01.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 --2- 172.21.15.46:0/846764614 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28112010 0x7fdb28114400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 --2- 172.21.15.46:0/846764614 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28076140 0x7fdb28111ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 --2- 172.21.15.46:0/846764614 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb28075c00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 -- 172.21.15.46:0/846764614 >> 172.21.15.46:0/846764614 conn(0x7fdb280fee00 msgr2=0x7fdb28101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 -- 172.21.15.46:0/846764614 shutdown_connections 2024-03-08T23:17:01.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.322+0000 7fdb2f200640 1 -- 172.21.15.46:0/846764614 wait complete. 2024-03-08T23:17:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 Processor -- start 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 -- start start 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28117b70 con 0x7fdb28075820 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281179f0 con 0x7fdb28112010 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.323+0000 7fdb2f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb28117cf0 con 0x7fdb28076140 2024-03-08T23:17:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60150/0 (socket says 172.21.15.46:60150) 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:39866/0 (socket says 172.21.15.46:39866) 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45852/0 (socket says 172.21.15.46:45852) 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2de00640 1 -- 172.21.15.46:0/2591575396 learned_addr learned my addr 172.21.15.46:0/2591575396 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 -- 172.21.15.46:0/2591575396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 msgr2=0x7fdb2819e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 -- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 msgr2=0x7fdb2819e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2e800640 1 -- 172.21.15.46:0/2591575396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb281a3380 con 0x7fdb28112010 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.324+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb2e800640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005fd0 tx=0x7fdb2400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb22a00640 1 -- 172.21.15.46:0/2591575396 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24003b70 con 0x7fdb28112010 2024-03-08T23:17:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb22a00640 1 -- 172.21.15.46:0/2591575396 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb24003d10 con 0x7fdb28112010 2024-03-08T23:17:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb22a00640 1 -- 172.21.15.46:0/2591575396 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb24010830 con 0x7fdb28112010 2024-03-08T23:17:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb281a3670 con 0x7fdb28112010 2024-03-08T23:17:01.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.325+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb281aaf00 con 0x7fdb28112010 2024-03-08T23:17:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.328+0000 7fdb33419740 1 Processor -- start 2024-03-08T23:17:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb22a00640 1 -- 172.21.15.46:0/2591575396 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fdb24004020 con 0x7fdb28112010 2024-03-08T23:17:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 -- start start 2024-03-08T23:17:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674530 0x563adb674910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb67fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb5d8dd0 con 0x563adb674e50 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb581c70 con 0x563adb674530 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb33419740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb5d4e70 con 0x563adb67da30 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb67fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60162/0 (socket says 172.21.15.46:60162) 2024-03-08T23:17:01.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb2de00640 1 -- 172.21.15.46:0/2826336661 learned_addr learned my addr 172.21.15.46:0/2826336661 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.329+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674530 0x563adb674910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 -- 172.21.15.46:0/2826336661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 msgr2=0x563adb67fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb67fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 -- 172.21.15.46:0/2826336661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674530 msgr2=0x563adb674910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674530 0x563adb674910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 -- 172.21.15.46:0/2826336661 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563adb63b2b0 con 0x563adb674e50 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb67fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:01.336 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb22a00640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c036040 tx=0x7fdb1c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d349c1ebd1913b70 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.330+0000 7fdb22a00640 1 -- 172.21.15.46:0/2591575396 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fdb240cf800 con 0x7fdb28112010 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb17000640 1 -- 172.21.15.46:0/2826336661 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c042070 con 0x563adb674e50 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb17000640 1 -- 172.21.15.46:0/2826336661 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb1c004a70 con 0x563adb674e50 2024-03-08T23:17:01.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb17000640 1 -- 172.21.15.46:0/2826336661 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c005160 con 0x563adb674e50 2024-03-08T23:17:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb33419740 1 -- 172.21.15.46:0/2826336661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 msgr2=0x563adb67d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb33419740 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c036040 tx=0x7fdb1c009560 comp rx=0 tx=0).stop 2024-03-08T23:17:01.341 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.331+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 0x7fdb100a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x563adb66fdf0 tx=0x7fdb18004730 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 -- 172.21.15.46:0/2826336661 shutdown_connections 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb67fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d4f0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 --2- 172.21.15.46:0/2826336661 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674530 0x563adb674910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 -- 172.21.15.46:0/2826336661 >> 172.21.15.46:0/2826336661 conn(0x563adb66ec60 msgr2=0x563adb66f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 -- 172.21.15.46:0/2826336661 shutdown_connections 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 -- 172.21.15.46:0/2826336661 wait complete. 2024-03-08T23:17:01.342 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 -- start start 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.332+0000 7fdb33419740 1 Processor -- start 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb67da30 0x563adb67ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb6807c0 con 0x563adb67da30 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb5950c0 con 0x563adb674e50 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x563adb680940 con 0x563adb674530 2024-03-08T23:17:01.343 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb33419740 10 client.0 _ll_register_callbacks cb 0x563adb5cfec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2e800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb67da30 0x563adb67ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45870/0 (socket says 172.21.15.46:45870) 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2d400640 1 -- 172.21.15.46:0/1836804530 learned_addr learned my addr 172.21.15.46:0/1836804530 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.333+0000 7fdb2d400640 1 -- 172.21.15.46:0/1836804530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 msgr2=0x563adb67d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb2d400640 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.344 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb2d400640 1 -- 172.21.15.46:0/1836804530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb67da30 msgr2=0x563adb67ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb2d400640 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb67da30 0x563adb67ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb2d400640 1 -- 172.21.15.46:0/1836804530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563adb2ed3f0 con 0x563adb674e50 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb2d400640 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb18003c50 tx=0x7fdb180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb957fb439931548 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb0b400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18030060 con 0x563adb674e50 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb1c004a70 con 0x563adb674e50 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.334+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fdb1801a050 con 0x563adb674e50 2024-03-08T23:17:01.345 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb18037050 con 0x563adb674e50 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 10 client.27518 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563adb3b77d0 con 0x563adb674e50 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb1c004a70 con 0x563adb674e50 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdb1801f3a0 con 0x563adb674e50 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 10 client.27518 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 10 client.27518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:01.346 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 20 client.27518 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb68da80 0x563adb68fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 4 client.27518 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.335+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563adb5ad4f0 con 0x563adb68da80 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.336+0000 7fdb2de00640 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb68da80 0x563adb68fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.336+0000 7fdb2de00640 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb68da80 0x563adb68fec0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76b0314a78292106 server_cookie=6fe8cac4f90f26fb in_seq=0 out_seq=0 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.336+0000 7fdb0b400640 10 client.27518 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb0b400640 1 -- 172.21.15.46:0/1836804530 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563adb5ad4f0 con 0x563adb68da80 2024-03-08T23:17:01.347 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb0b400640 10 client.27518 handle_command_reply: tid=0 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 client.27518 shutdown 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 msgr2=0x563adb67a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fdb18003c50 tx=0x7fdb180046c0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 shutdown_connections 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb68da80 0x563adb68fec0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.348 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb67da30 0x563adb67ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb67a790 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 --2- 172.21.15.46:0/1836804530 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 >> 172.21.15.46:0/1836804530 conn(0x563adb66ec60 msgr2=0x563adb6709b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb0b400640 0 client.27518 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 shutdown_connections 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 1 -- 172.21.15.46:0/1836804530 wait complete. 2024-03-08T23:17:01.349 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.337+0000 7fdb33419740 20 client.27518 trim_cache size 0 max 16384 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 Processor -- start 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 -- start start 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.360 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b6e60 con 0x563adb674e50 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b4a00 con 0x563adb67da30 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb33419740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b2ac0 con 0x563adb674530 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb2e800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45874/0 (socket says 172.21.15.46:45874) 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.351+0000 7fdb2e800640 1 -- 172.21.15.46:0/309008633 learned_addr learned my addr 172.21.15.46:0/309008633 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2de00640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.361 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 -- 172.21.15.46:0/309008633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 msgr2=0x563adb67cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 -- 172.21.15.46:0/309008633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 msgr2=0x563adb67d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 -- 172.21.15.46:0/309008633 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563adb6950b0 con 0x563adb67da30 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2de00640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:01.362 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2d400640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb2e800640 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400efd0 tx=0x7fdb2409b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dadbd7adc2f063d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb16600640 1 -- 172.21.15.46:0/309008633 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb240b9070 con 0x563adb67da30 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb16600640 1 -- 172.21.15.46:0/309008633 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb2409c140 con 0x563adb67da30 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.352+0000 7fdb16600640 1 -- 172.21.15.46:0/309008633 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb240b3030 con 0x563adb67da30 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- 172.21.15.46:0/309008633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 msgr2=0x563adb684b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdb2400efd0 tx=0x7fdb2409b920 comp rx=0 tx=0).stop 2024-03-08T23:17:01.363 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- 172.21.15.46:0/309008633 shutdown_connections 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb67da30 0x563adb684b80 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674e50 0x563adb67d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 --2- 172.21.15.46:0/309008633 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb674530 0x563adb67cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- 172.21.15.46:0/309008633 >> 172.21.15.46:0/309008633 conn(0x563adb66ec60 msgr2=0x563adb675370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- 172.21.15.46:0/309008633 shutdown_connections 2024-03-08T23:17:01.364 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- 172.21.15.46:0/309008633 wait complete. 2024-03-08T23:17:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.353+0000 7fdb33419740 1 -- start start 2024-03-08T23:17:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 Processor -- start 2024-03-08T23:17:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674530 0x563adb683c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.365 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb6e3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b30c0 con 0x563adb674530 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b43a0 con 0x563adb674e50 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb281b36e0 con 0x563adb67da30 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb33419740 10 client.0 _ll_register_callbacks cb 0x7fdb240093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb2de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674530 0x563adb683c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.366 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb2d400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45882/0 (socket says 172.21.15.46:45882) 2024-03-08T23:17:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb2d400640 1 -- 172.21.15.46:0/2454484809 learned_addr learned my addr 172.21.15.46:0/2454484809 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.354+0000 7fdb2e800640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb6e3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 -- 172.21.15.46:0/2454484809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 msgr2=0x563adb6e3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.367 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb6e3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 -- 172.21.15.46:0/2454484809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674530 msgr2=0x563adb683c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674530 0x563adb683c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 -- 172.21.15.46:0/2454484809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563adb6e44f0 con 0x563adb674e50 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb2d400640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x563adb67bd60 tx=0x7fdb18023ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36e4b9c1e92cb3af server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb0b400640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:17:01.368 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18040060 con 0x563adb674e50 2024-03-08T23:17:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdb2409c140 con 0x563adb674e50 2024-03-08T23:17:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.355+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fdb1801f930 con 0x563adb674e50 2024-03-08T23:17:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb18047050 con 0x563adb674e50 2024-03-08T23:17:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 10 client.27524 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:01.369 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563adb6e5720 con 0x563adb674e50 2024-03-08T23:17:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb2409c140 con 0x563adb674e50 2024-03-08T23:17:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdb1803ba10 con 0x563adb674e50 2024-03-08T23:17:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 10 client.27524 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:01.370 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 10 client.27524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 20 client.27524 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb6e5d90 0x563adb6e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 4 client.27524 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563adb5e6310 con 0x563adb6e5d90 2024-03-08T23:17:01.371 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.356+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb6e5d90 0x563adb6e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:01.372 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb2de00640 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb6e5d90 0x563adb6e81b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8307ebb76deb19d8 server_cookie=9dbd9b1d1f315c60 in_seq=0 out_seq=0 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb0b400640 10 client.27524 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb0b400640 1 -- 172.21.15.46:0/2454484809 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563adb5e6310 con 0x563adb6e5d90 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb0b400640 10 client.27524 handle_command_reply: tid=0 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb33419740 1 client.27524 shutdown 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 msgr2=0x563adb684180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.357+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x563adb67bd60 tx=0x7fdb18023ca0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.373 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 shutdown_connections 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x563adb6e5d90 0x563adb6e81b0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563adb67da30 0x563adb6e3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x563adb674e50 0x563adb684180 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 --2- 172.21.15.46:0/2454484809 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x563adb674530 0x563adb683c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb0b400640 0 client.27524 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 >> 172.21.15.46:0/2454484809 conn(0x563adb66ec60 msgr2=0x563adb675c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 shutdown_connections 2024-03-08T23:17:01.374 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 1 -- 172.21.15.46:0/2454484809 wait complete. 2024-03-08T23:17:01.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb33419740 20 client.27524 trim_cache size 0 max 16384 2024-03-08T23:17:01.375 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 msgr2=0x7fdb100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 0x7fdb100a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x563adb66fdf0 tx=0x7fdb18004730 comp rx=0 tx=0).stop 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 msgr2=0x7fdb281a2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.358+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fdb24005fd0 tx=0x7fdb2400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 shutdown_connections 2024-03-08T23:17:01.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fdb100a2970 0x7fdb100a4e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fdb28112010 0x7fdb281a2c50 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdb28076140 0x7fdb2819e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 --2- 172.21.15.46:0/2591575396 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fdb28075820 0x7fdb2819e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 >> 172.21.15.46:0/2591575396 conn(0x7fdb280fee00 msgr2=0x7fdb281011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 shutdown_connections 2024-03-08T23:17:01.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:01.359+0000 7fdb2f200640 1 -- 172.21.15.46:0/2591575396 wait complete. 2024-03-08T23:17:02.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:01 smithi062 ceph-mon[35761]: pgmap v286: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:02.214 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:01 smithi192 ceph-mon[36700]: pgmap v286: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:02.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:01 smithi046 ceph-mon[31859]: pgmap v286: 129 pgs: 129 active+clean; 78 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:02.380 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:17:02.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:17:02.381 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 -- 172.21.15.46:0/3751852378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01035e0 msgr2=0x7f0fd01039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3751852378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01035e0 0x7f0fd01039c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009860 tx=0x7f0fc402bb70 comp rx=0 tx=0).stop 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 -- 172.21.15.46:0/3751852378 shutdown_connections 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3751852378 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01086b0 0x7f0fd0110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3751852378 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd0103f90 0x7f0fd0107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.618 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3751852378 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01035e0 0x7f0fd01039c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 -- 172.21.15.46:0/3751852378 >> 172.21.15.46:0/3751852378 conn(0x7f0fd00fee00 msgr2=0x7f0fd0101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 -- 172.21.15.46:0/3751852378 shutdown_connections 2024-03-08T23:17:02.619 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.614+0000 7f0fd7000640 1 -- 172.21.15.46:0/3751852378 wait complete. 2024-03-08T23:17:02.620 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 Processor -- start 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 -- start start 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01086b0 0x7f0fd019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd019e860 0x7f0fd019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0113740 con 0x7f0fd01035e0 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01135c0 con 0x7f0fd019e860 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.616+0000 7f0fd7000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01138c0 con 0x7f0fd01086b0 2024-03-08T23:17:02.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60208/0 (socket says 172.21.15.46:60208) 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 -- 172.21.15.46:0/3522035603 learned_addr learned my addr 172.21.15.46:0/3522035603 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd6600640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd019e860 0x7f0fd019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5200640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01086b0 0x7f0fd019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 -- 172.21.15.46:0/3522035603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01086b0 msgr2=0x7f0fd019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01086b0 0x7f0fd019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 -- 172.21.15.46:0/3522035603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd019e860 msgr2=0x7f0fd019ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd019e860 0x7f0fd019ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.617+0000 7f0fd5c00640 1 -- 172.21.15.46:0/3522035603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd019f350 con 0x7f0fd01035e0 2024-03-08T23:17:02.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fd5c00640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4037040 tx=0x7f0fc40045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fce000640 1 -- 172.21.15.46:0/3522035603 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4041020 con 0x7f0fd01035e0 2024-03-08T23:17:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd019f5e0 con 0x7f0fd01035e0 2024-03-08T23:17:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fce000640 1 -- 172.21.15.46:0/3522035603 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc402c9f0 con 0x7f0fd01035e0 2024-03-08T23:17:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fce000640 1 -- 172.21.15.46:0/3522035603 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4039040 con 0x7f0fd01035e0 2024-03-08T23:17:02.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.618+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd01a6da0 con 0x7f0fd01035e0 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.621+0000 7f0fce000640 1 -- 172.21.15.46:0/3522035603 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0fc4044020 con 0x7f0fd01035e0 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.621+0000 7f0fdb37f740 1 Processor -- start 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 -- start start 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473ebb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f4760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 0x5592473f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55924734fa50 con 0x5592473f4ca0 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592472f8b00 con 0x5592473eb7a0 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55924734baf0 con 0x5592473ec0c0 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.622+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 0x5592473f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473ebb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 0x5592473f7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60210/0 (socket says 172.21.15.46:60210) 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd5200640 1 -- 172.21.15.46:0/4223526647 learned_addr learned my addr 172.21.15.46:0/4223526647 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 -- 172.21.15.46:0/4223526647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 msgr2=0x5592473f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 -- 172.21.15.46:0/4223526647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 msgr2=0x5592473f7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 0x5592473f7090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 -- 172.21.15.46:0/4223526647 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5592473161e0 con 0x5592473eb7a0 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fd6600640 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473ebb80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009f10 tx=0x7f0fc8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5dd38df5e0cd51f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.623+0000 7f0fce000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 0x7f0fb80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fc2a00640 1 -- 172.21.15.46:0/4223526647 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8004d20 con 0x5592473eb7a0 2024-03-08T23:17:02.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fc2a00640 1 -- 172.21.15.46:0/4223526647 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc8004050 con 0x5592473eb7a0 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fc2a00640 1 -- 172.21.15.46:0/4223526647 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8005260 con 0x5592473eb7a0 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fce000640 1 -- 172.21.15.46:0/3522035603 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0fc40f09c0 con 0x7f0fd01035e0 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fd5c00640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 0x7f0fb80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 -- 172.21.15.46:0/4223526647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 msgr2=0x5592473ebb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473ebb80 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8009f10 tx=0x7f0fc8011810 comp rx=0 tx=0).stop 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 -- 172.21.15.46:0/4223526647 shutdown_connections 2024-03-08T23:17:02.631 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473f4ca0 0x5592473f7090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/4223526647 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473ebb80 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.624+0000 7f0fdb37f740 1 -- 172.21.15.46:0/4223526647 >> 172.21.15.46:0/4223526647 conn(0x5592473e5ed0 msgr2=0x5592473e62b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.625+0000 7f0fd5c00640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 0x7f0fb80a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5592473eb270 tx=0x7f0fc4005740 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.625+0000 7f0fdb37f740 1 -- 172.21.15.46:0/4223526647 shutdown_connections 2024-03-08T23:17:02.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.625+0000 7f0fdb37f740 1 -- 172.21.15.46:0/4223526647 wait complete. 2024-03-08T23:17:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.625+0000 7f0fdb37f740 1 -- start start 2024-03-08T23:17:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.625+0000 7f0fdb37f740 1 Processor -- start 2024-03-08T23:17:02.636 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473f4ca0 0x5592473f2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592473f7a90 con 0x5592473eb7a0 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55924730bf60 con 0x5592473f4ca0 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592473f7c10 con 0x5592473ec0c0 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fdb37f740 10 client.0 _ll_register_callbacks cb 0x5592472f7b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fd6600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.637 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fd6600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60212/0 (socket says 172.21.15.46:60212) 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fd6600640 1 -- 172.21.15.46:0/3519569793 learned_addr learned my addr 172.21.15.46:0/3519569793 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fd5200640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.626+0000 7f0fd6600640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473f4ca0 0x5592473f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 -- 172.21.15.46:0/3519569793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 msgr2=0x5592473f2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 -- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473f4ca0 msgr2=0x5592473f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473f4ca0 0x5592473f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 -- 172.21.15.46:0/3519569793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55924712c7e0 con 0x5592473eb7a0 2024-03-08T23:17:02.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd5200640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fd6600640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8004970 tx=0x7f0fc80048e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=966e9b3d956bbe20 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fb7400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8004400 con 0x5592473eb7a0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0fc80045a0 con 0x5592473eb7a0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.627+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc8004050 con 0x5592473eb7a0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc8036050 con 0x5592473eb7a0 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 10 client.17373 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:02.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55924718cf70 con 0x5592473eb7a0 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc8004050 con 0x5592473eb7a0 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0fc8028640 con 0x5592473eb7a0 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 10 client.17373 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 10 client.17373 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 20 client.17373 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559247405070 0x559247407460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.640 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 4 client.17373 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.628+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55924735cf90 con 0x559247405070 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.629+0000 7f0fd5200640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559247405070 0x559247407460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.629+0000 7f0fd5200640 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559247405070 0x559247407460 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d000c1cb03664b87 server_cookie=15ba396813352800 in_seq=0 out_seq=0 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.629+0000 7f0fb7400640 10 client.17373 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fb7400640 1 -- 172.21.15.46:0/3519569793 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55924735cf90 con 0x559247405070 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fb7400640 10 client.17373 handle_command_reply: tid=0 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 client.17373 shutdown 2024-03-08T23:17:02.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 msgr2=0x5592473f1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f0fc8004970 tx=0x7f0fc80048e0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 shutdown_connections 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x559247405070 0x559247407460 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473f4ca0 0x5592473f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473f2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/3519569793 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x5592473f1d00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fb7400640 0 client.17373 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.630+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 >> 172.21.15.46:0/3519569793 conn(0x5592473e5ed0 msgr2=0x5592473e7c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.631+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 shutdown_connections 2024-03-08T23:17:02.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.631+0000 7f0fdb37f740 1 -- 172.21.15.46:0/3519569793 wait complete. 2024-03-08T23:17:02.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.631+0000 7f0fdb37f740 20 client.17373 trim_cache size 0 max 16384 2024-03-08T23:17:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.647+0000 7f0fdb37f740 1 Processor -- start 2024-03-08T23:17:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 -- start start 2024-03-08T23:17:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 0x5592473f4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473ec0c0 0x5592473fbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01b2ab0 con 0x5592473ec0c0 2024-03-08T23:17:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01b0650 con 0x5592473eb7a0 2024-03-08T23:17:02.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01ae710 con 0x5592473e5ed0 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:45928/0 (socket says 172.21.15.46:45928) 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fd5200640 1 -- 172.21.15.46:0/870267998 learned_addr learned my addr 172.21.15.46:0/870267998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fd6600640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473ec0c0 0x5592473fbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.648+0000 7f0fd6600640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 0x5592473f4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 -- 172.21.15.46:0/870267998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 msgr2=0x5592473f4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 0x5592473f4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 -- 172.21.15.46:0/870267998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473ec0c0 msgr2=0x5592473fbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473ec0c0 0x5592473fbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 -- 172.21.15.46:0/870267998 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559247024170 con 0x5592473eb7a0 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd6600640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 0x5592473f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fd5200640 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc000c00 tx=0x7f0fbc0039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de0dd86e45c48fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fc2000640 1 -- 172.21.15.46:0/870267998 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc01d5b0 con 0x5592473eb7a0 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fc2000640 1 -- 172.21.15.46:0/870267998 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fbc01d750 con 0x5592473eb7a0 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.649+0000 7f0fc2000640 1 -- 172.21.15.46:0/870267998 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0116a0 con 0x5592473eb7a0 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- 172.21.15.46:0/870267998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 msgr2=0x5592473e4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc000c00 tx=0x7f0fbc0039c0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- 172.21.15.46:0/870267998 shutdown_connections 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473ec0c0 0x5592473fbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473eb7a0 0x5592473e4df0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/870267998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473e5ed0 0x5592473f4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- 172.21.15.46:0/870267998 >> 172.21.15.46:0/870267998 conn(0x5592474077d0 msgr2=0x5592473e46e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- 172.21.15.46:0/870267998 shutdown_connections 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- 172.21.15.46:0/870267998 wait complete. 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.650+0000 7f0fdb37f740 1 -- start start 2024-03-08T23:17:02.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 Processor -- start 2024-03-08T23:17:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473e5ed0 0x559247455ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473fb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01aed10 con 0x5592473eb7a0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01afff0 con 0x5592473e5ed0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01af330 con 0x5592473ec0c0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fdb37f740 10 client.0 _ll_register_callbacks cb 0x7f0fc8009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.651+0000 7f0fd5200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:60238/0 (socket says 172.21.15.46:60238) 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 -- 172.21.15.46:0/2629311626 learned_addr learned my addr 172.21.15.46:0/2629311626 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd6600640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473e5ed0 0x559247455ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd6600640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473fb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 -- 172.21.15.46:0/2629311626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 msgr2=0x5592473fb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473fb610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 -- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473e5ed0 msgr2=0x559247455ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473e5ed0 0x559247455ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 -- 172.21.15.46:0/2629311626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55924731b050 con 0x5592473eb7a0 2024-03-08T23:17:02.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd6600640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fd5200640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc012ee0 tx=0x7f0fbc01d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1168f5a1be093a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.652+0000 7f0fb7400640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc005240 con 0x5592473eb7a0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0fbc0053e0 con 0x5592473eb7a0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fbc01d750 con 0x5592473eb7a0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fbc0055e0 con 0x5592473eb7a0 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fdb37f740 10 client.17376 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:02.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55924745d290 con 0x5592473eb7a0 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fbc01d750 con 0x5592473eb7a0 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.653+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0fbc02d020 con 0x5592473eb7a0 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 10 client.17376 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 10 client.17376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 20 client.17376 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5592473f4ca0 0x55924745f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 4 client.17376 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:17:02.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5592473572e0 con 0x5592473f4ca0 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fd6600640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5592473f4ca0 0x55924745f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fd6600640 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5592473f4ca0 0x55924745f9d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1714e50c5d3ff471 server_cookie=d38b38316402fc4a in_seq=0 out_seq=0 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.654+0000 7f0fb7400640 10 client.17376 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fb7400640 1 -- 172.21.15.46:0/2629311626 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5592473572e0 con 0x5592473f4ca0 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fb7400640 10 client.17376 handle_command_reply: tid=0 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 client.17376 shutdown 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 msgr2=0x559247456220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc012ee0 tx=0x7f0fbc01d430 comp rx=0 tx=0).stop 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 shutdown_connections 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5592473f4ca0 0x55924745f9d0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5592473ec0c0 0x5592473fb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5592473eb7a0 0x559247456220 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 --2- 172.21.15.46:0/2629311626 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5592473e5ed0 0x559247455ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 >> 172.21.15.46:0/2629311626 conn(0x5592474077d0 msgr2=0x5592473e4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fb7400640 0 client.17376 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 shutdown_connections 2024-03-08T23:17:02.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 1 -- 172.21.15.46:0/2629311626 wait complete. 2024-03-08T23:17:02.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.656+0000 7f0fdb37f740 20 client.17376 trim_cache size 0 max 16384 2024-03-08T23:17:02.668 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:02.668 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:17:02.668 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "9b0f402e-473a-4ab1-8660-02304972880b", 2024-03-08T23:17:02.669 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:17:02.669 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 msgr2=0x7f0fb80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 0x7f0fb80a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x5592473eb270 tx=0x7f0fc4005740 comp rx=0 tx=0).stop 2024-03-08T23:17:02.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 msgr2=0x7f0fd0199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4037040 tx=0x7f0fc40045b0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 shutdown_connections 2024-03-08T23:17:02.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0fb80a2970 0x7f0fb80a4e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.671 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0fd019e860 0x7f0fd019ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0fd01086b0 0x7f0fd019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 --2- 172.21.15.46:0/3522035603 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0fd01035e0 0x7f0fd0199fa0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 >> 172.21.15.46:0/3522035603 conn(0x7f0fd00fee00 msgr2=0x7f0fd0100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 shutdown_connections 2024-03-08T23:17:02.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.657+0000 7f0fd7000640 1 -- 172.21.15.46:0/3522035603 wait complete. 2024-03-08T23:17:02.682 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:17:02.682 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:17:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.962+0000 7f7551200640 1 -- 172.21.15.46:0/2598445977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 msgr2=0x7f754c10e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.962+0000 7f7551200640 1 --2- 172.21.15.46:0/2598445977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c10e190 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f7544009be0 tx=0x7f754402bef0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 -- 172.21.15.46:0/2598445977 shutdown_connections 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 --2- 172.21.15.46:0/2598445977 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c10e190 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 --2- 172.21.15.46:0/2598445977 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c1003b0 0x7f754c105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 --2- 172.21.15.46:0/2598445977 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c0ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 -- 172.21.15.46:0/2598445977 >> 172.21.15.46:0/2598445977 conn(0x7f754c076710 msgr2=0x7f754c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 -- 172.21.15.46:0/2598445977 shutdown_connections 2024-03-08T23:17:02.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.963+0000 7f7551200640 1 -- 172.21.15.46:0/2598445977 wait complete. 2024-03-08T23:17:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.964+0000 7f7551200640 1 Processor -- start 2024-03-08T23:17:02.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 -- start start 2024-03-08T23:17:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c19bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c195750 0x7f754c195bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c10f4d0 con 0x7f754c1065c0 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c10f350 con 0x7f754c0ffa00 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7551200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c10f650 con 0x7f754c195750 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7549e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.965+0000 7f7549e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53370/0 (socket says 172.21.15.46:53370) 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 -- 172.21.15.46:0/169647753 learned_addr learned my addr 172.21.15.46:0/169647753 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549400640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c19bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f754a800640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c195750 0x7f754c195bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 -- 172.21.15.46:0/169647753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c195750 msgr2=0x7f754c195bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c195750 0x7f754c195bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 -- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 msgr2=0x7f754c19bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c19bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 -- 172.21.15.46:0/169647753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c196350 con 0x7f754c0ffa00 2024-03-08T23:17:02.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549400640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.966+0000 7f7549e00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9c0 tx=0x7f753c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.967+0000 7f7542a00640 1 -- 172.21.15.46:0/169647753 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00cde0 con 0x7f754c0ffa00 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.967+0000 7f7542a00640 1 -- 172.21.15.46:0/169647753 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f753c0035f0 con 0x7f754c0ffa00 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.967+0000 7f7542a00640 1 -- 172.21.15.46:0/169647753 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c010690 con 0x7f754c0ffa00 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.967+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c1a2430 con 0x7f754c0ffa00 2024-03-08T23:17:02.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.967+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c1a27e0 con 0x7f754c0ffa00 2024-03-08T23:17:02.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.972+0000 7f7553ff7740 1 Processor -- start 2024-03-08T23:17:02.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.972+0000 7f7553ff7740 1 -- start start 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.972+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c020151d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020154680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7553ff7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c020102410 con 0x55c0201496b0 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7553ff7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0200a4d50 con 0x55c020152290 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7553ff7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c02004db80 con 0x55c020148d90 2024-03-08T23:17:02.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7542a00640 1 -- 172.21.15.46:0/169647753 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f753c0037a0 con 0x7f754c0ffa00 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f754a800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f754a800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:47666/0 (socket says 172.21.15.46:47666) 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f754a800640 1 -- 172.21.15.46:0/2152268054 learned_addr learned my addr 172.21.15.46:0/2152268054 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7549400640 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.973+0000 7f7549e00640 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c020151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 -- 172.21.15.46:0/2152268054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 msgr2=0x55c020154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.978 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 -- 172.21.15.46:0/2152268054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 msgr2=0x55c020151d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c020151d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 -- 172.21.15.46:0/2152268054 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c01fe837d0 con 0x55c020148d90 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f754a800640 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7544004af0 tx=0x7f7544009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf13b3ead0c6141e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f7537400640 1 -- 172.21.15.46:0/2152268054 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544043070 con 0x55c020148d90 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f7537400640 1 -- 172.21.15.46:0/2152268054 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7544033c80 con 0x55c020148d90 2024-03-08T23:17:02.979 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.974+0000 7f7537400640 1 -- 172.21.15.46:0/2152268054 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754402caf0 con 0x55c020148d90 2024-03-08T23:17:02.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.975+0000 7f7542a00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 0x7f75300a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.975+0000 7f7542a00640 1 -- 172.21.15.46:0/169647753 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f753c0cf6c0 con 0x7f754c0ffa00 2024-03-08T23:17:02.982 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.975+0000 7f7553ff7740 1 -- 172.21.15.46:0/2152268054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 msgr2=0x55c020149170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.975+0000 7f7553ff7740 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7544004af0 tx=0x7f7544009840 comp rx=0 tx=0).stop 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7549e00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 0x7f75300a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 -- 172.21.15.46:0/2152268054 shutdown_connections 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c020151d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 --2- 172.21.15.46:0/2152268054 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c020149170 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 -- 172.21.15.46:0/2152268054 >> 172.21.15.46:0/2152268054 conn(0x55c0201434c0 msgr2=0x55c0201438a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 -- 172.21.15.46:0/2152268054 shutdown_connections 2024-03-08T23:17:02.983 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.976+0000 7f7553ff7740 1 -- 172.21.15.46:0/2152268054 wait complete. 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.977+0000 7f7553ff7740 1 -- start start 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.977+0000 7f7549e00640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 0x7f75300a4ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f753c0ccb00 tx=0x7f753c000bc0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.977+0000 7f7553ff7740 1 Processor -- start 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.977+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c02014f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020152290 0x55c02014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c020060ee0 con 0x55c0201496b0 2024-03-08T23:17:02.984 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c0200a0df0 con 0x55c020148d90 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c020155020 con 0x55c020152290 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7553ff7740 10 client.0 _ll_register_callbacks cb 0x55c0200678d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53392/0 (socket says 172.21.15.46:53392) 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 -- 172.21.15.46:0/3944270231 learned_addr learned my addr 172.21.15.46:0/3944270231 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020152290 0x55c02014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f7549400640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c02014f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.985 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 -- 172.21.15.46:0/3944270231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020152290 msgr2=0x55c02014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020152290 0x55c02014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 -- 172.21.15.46:0/3944270231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 msgr2=0x55c02014f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.978+0000 7f754a800640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c02014f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f754a800640 1 -- 172.21.15.46:0/3944270231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c01fdb93f0 con 0x55c020148d90 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f754a800640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7544006010 tx=0x7f7544003cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d40ea39249f2a9db server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f752be00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544054060 con 0x55c020148d90 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7544033c80 con 0x55c020148d90 2024-03-08T23:17:02.986 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.979+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f7544033480 con 0x55c020148d90 2024-03-08T23:17:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f754405b050 con 0x55c020148d90 2024-03-08T23:17:02.988 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f7553ff7740 10 client.27554 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c01fd790f0 con 0x55c020148d90 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7544033c80 con 0x55c020148d90 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7544048450 con 0x55c020148d90 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.980+0000 7f7553ff7740 10 client.27554 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7553ff7740 10 client.27554 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7553ff7740 20 client.27554 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c020162350 0x55c020164740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:02.989 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7553ff7740 4 client.27554 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c0200b2290 con 0x55c020162350 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.981+0000 7f7549400640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c020162350 0x55c020164740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.982+0000 7f7549400640 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c020162350 0x55c020164740 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28e6c15a67a5e0ea server_cookie=d994bd0a01be3290 in_seq=0 out_seq=0 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.982+0000 7f752be00640 10 client.27554 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f752be00640 1 -- 172.21.15.46:0/3944270231 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c0200b2290 con 0x55c020162350 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f752be00640 10 client.27554 handle_command_reply: tid=0 2024-03-08T23:17:02.990 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 client.27554 shutdown 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 msgr2=0x55c02014f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7544006010 tx=0x7f7544003cb0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 shutdown_connections 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c020162350 0x55c020164740 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020152290 0x55c02014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c0201496b0 0x55c02014f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f752be00640 0 client.27554 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:02.991 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3944270231 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020148d90 0x55c02014f330 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 >> 172.21.15.46:0/3944270231 conn(0x55c0201434c0 msgr2=0x55c020144980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 shutdown_connections 2024-03-08T23:17:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.983+0000 7f7553ff7740 1 -- 172.21.15.46:0/3944270231 wait complete. 2024-03-08T23:17:02.992 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:02.984+0000 7f7553ff7740 20 client.27554 trim_cache size 0 max 16384 2024-03-08T23:17:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.008+0000 7f7553ff7740 1 Processor -- start 2024-03-08T23:17:03.011 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.008+0000 7f7553ff7740 1 -- start start 2024-03-08T23:17:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.008+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020148d90 0x55c020139d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020159290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7553ff7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1ae980 con 0x55c020148d90 2024-03-08T23:17:03.012 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7553ff7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1ac520 con 0x55c020152290 2024-03-08T23:17:03.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7553ff7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1aa5e0 con 0x55c0201496b0 2024-03-08T23:17:03.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f754a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020148d90 0x55c020139d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.013 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7549400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7549400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:47680/0 (socket says 172.21.15.46:47680) 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f7549400640 1 -- 172.21.15.46:0/3595296857 learned_addr learned my addr 172.21.15.46:0/3595296857 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.009+0000 7f754a800640 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 -- 172.21.15.46:0/3595296857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 msgr2=0x55c020159290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020159290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 -- 172.21.15.46:0/3595296857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020148d90 msgr2=0x55c020139d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020148d90 0x55c020139d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.014 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 -- 172.21.15.46:0/3595296857 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c0201494e0 con 0x55c0201496b0 2024-03-08T23:17:03.015 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.010+0000 7f7549400640 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55c020144650 tx=0x7f7538012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8808a57dbf3165db server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.011+0000 7f7536a00640 1 -- 172.21.15.46:0/3595296857 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753801d5b0 con 0x55c0201496b0 2024-03-08T23:17:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.011+0000 7f7536a00640 1 -- 172.21.15.46:0/3595296857 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7538021030 con 0x55c0201496b0 2024-03-08T23:17:03.018 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.011+0000 7f7536a00640 1 -- 172.21.15.46:0/3595296857 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753800e3b0 con 0x55c0201496b0 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 -- 172.21.15.46:0/3595296857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 msgr2=0x55c02014fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x55c020144650 tx=0x7f7538012820 comp rx=0 tx=0).stop 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 -- 172.21.15.46:0/3595296857 shutdown_connections 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c020152290 0x55c020159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c0201496b0 0x55c02014fbd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 --2- 172.21.15.46:0/3595296857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020148d90 0x55c020139d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 -- 172.21.15.46:0/3595296857 >> 172.21.15.46:0/3595296857 conn(0x55c0201434c0 msgr2=0x55c020139680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 -- 172.21.15.46:0/3595296857 shutdown_connections 2024-03-08T23:17:03.019 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.012+0000 7f7553ff7740 1 -- 172.21.15.46:0/3595296857 wait complete. 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 -- start start 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 Processor -- start 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020152290 0x55c0201accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.013+0000 7f7553ff7740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1aabe0 con 0x55c020152290 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f7553ff7740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1abec0 con 0x55c0201496b0 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f7553ff7740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c1ab200 con 0x55c020148d90 2024-03-08T23:17:03.020 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f7553ff7740 10 client.0 _ll_register_callbacks cb 0x7f753c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f7549400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f754a800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f754a800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:47692/0 (socket says 172.21.15.46:47692) 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f754a800640 1 -- 172.21.15.46:0/1473593552 learned_addr learned my addr 172.21.15.46:0/1473593552 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f7549400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:53420/0 (socket says 172.21.15.46:53420) 2024-03-08T23:17:03.021 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.014+0000 7f754a800640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020152290 0x55c0201accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 -- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 msgr2=0x55c0201b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 -- 172.21.15.46:0/1473593552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020152290 msgr2=0x55c0201accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020152290 0x55c0201accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 -- 172.21.15.46:0/1473593552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c0201ad3a0 con 0x55c020148d90 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f7549400640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f754a800640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7544009900 tx=0x7f75440333e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39ab1fe43cc9e401 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:03.022 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f752be00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:03.023 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544054040 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f7544033b40 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.015+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7538021030 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7544043070 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f7553ff7740 10 client.27505 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c0201ba0b0 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7538021030 con 0x55c020148d90 2024-03-08T23:17:03.025 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f754403f430 con 0x55c020148d90 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.016+0000 7f7553ff7740 10 client.27505 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7553ff7740 10 client.27505 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7553ff7740 20 client.27505 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c0201ba6d0 0x55c0201bcac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7553ff7740 4 client.27505 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c0200ac5e0 con 0x55c0201ba6d0 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.017+0000 7f7549400640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c0201ba6d0 0x55c0201bcac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:03.026 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7549400640 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c0201ba6d0 0x55c0201bcac0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8523f843e472a21f server_cookie=883b1bf620674e64 in_seq=0 out_seq=0 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f752be00640 10 client.27505 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f752be00640 1 -- 172.21.15.46:0/1473593552 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55c0200ac5e0 con 0x55c0201ba6d0 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f752be00640 10 client.27505 handle_command_reply: tid=0 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 client.27505 shutdown 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 msgr2=0x55c0201b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7544009900 tx=0x7f75440333e0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 shutdown_connections 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55c0201ba6d0 0x55c0201bcac0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.027 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c020152290 0x55c0201accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c0201496b0 0x55c0201b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 --2- 172.21.15.46:0/1473593552 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c020148d90 0x55c0201b28a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f752be00640 0 client.27505 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 >> 172.21.15.46:0/1473593552 conn(0x55c0201434c0 msgr2=0x55c020144680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 shutdown_connections 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.018+0000 7f7553ff7740 1 -- 172.21.15.46:0/1473593552 wait complete. 2024-03-08T23:17:03.028 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.019+0000 7f7553ff7740 20 client.27505 trim_cache size 0 max 16384 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "9b0f402e-473a-4ab1-8660-02304972880b": { 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "9b0f402e-473a-4ab1-8660-02304972880b", 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:17:03.029 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:17:03.030 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:17:03.030 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:03.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 msgr2=0x7f75300a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:03.030 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 0x7f75300a4ea0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f753c0ccb00 tx=0x7f753c000bc0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 msgr2=0x7f754c19b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f753c00e9c0 tx=0x7f753c00ee80 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 shutdown_connections 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75300a29e0 0x7f75300a4ea0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f754c195750 0x7f754c195bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f754c1065c0 0x7f754c19bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 --2- 172.21.15.46:0/169647753 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f754c0ffa00 0x7f754c19b690 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:03.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.020+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 >> 172.21.15.46:0/169647753 conn(0x7f754c076710 msgr2=0x7f754c1045c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.021+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 shutdown_connections 2024-03-08T23:17:03.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:03.021+0000 7f7551200640 1 -- 172.21.15.46:0/169647753 wait complete. 2024-03-08T23:17:03.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:02 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:03.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:02 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:03.044 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:02 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:03.046 INFO:tasks.cephfs.filesystem:scrub status for tag:9b0f402e-473a-4ab1-8660-02304972880b - {'path': '/', 'tag': '9b0f402e-473a-4ab1-8660-02304972880b', 'options': 'recursive,force'} 2024-03-08T23:17:03.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:02 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:03.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:02 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:03.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:02 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:03.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:02 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:03.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:02 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:03.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:02 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:04.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: pgmap v287: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:03 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: pgmap v287: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:17:04.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:03 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:17:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:04.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:04.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: pgmap v287: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 44 KiB/s wr, 24 op/s 2024-03-08T23:17:04.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:04.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:03 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:05.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:04 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:05.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:04 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:05.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:04 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:06.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:05 smithi062 ceph-mon[35761]: pgmap v288: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:06.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:05 smithi046 ceph-mon[31859]: pgmap v288: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:06.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:05 smithi192 ceph-mon[36700]: pgmap v288: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:07 smithi062 ceph-mon[35761]: pgmap v289: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:07 smithi062 ceph-mon[35761]: osdmap e101: 12 total, 12 up, 12 in 2024-03-08T23:17:08.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:07 smithi046 ceph-mon[31859]: pgmap v289: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:08.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:07 smithi046 ceph-mon[31859]: osdmap e101: 12 total, 12 up, 12 in 2024-03-08T23:17:08.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:07 smithi192 ceph-mon[36700]: pgmap v289: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:17:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:07 smithi192 ceph-mon[36700]: osdmap e101: 12 total, 12 up, 12 in 2024-03-08T23:17:09.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:08 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:17:09.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:08 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:17:09.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:08 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:17:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:10 smithi062 ceph-mon[35761]: pgmap v291: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-08T23:17:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:10 smithi062 ceph-mon[35761]: osdmap e102: 12 total, 12 up, 12 in 2024-03-08T23:17:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:10 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:17:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:10 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:17:10.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:10 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:17:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:10 smithi046 ceph-mon[31859]: pgmap v291: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-08T23:17:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:10 smithi046 ceph-mon[31859]: osdmap e102: 12 total, 12 up, 12 in 2024-03-08T23:17:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:10 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:17:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:10 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:17:10.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:10 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:17:10.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:10 smithi192 ceph-mon[36700]: pgmap v291: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 27 op/s 2024-03-08T23:17:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:10 smithi192 ceph-mon[36700]: osdmap e102: 12 total, 12 up, 12 in 2024-03-08T23:17:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:10 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:17:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:10 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:17:10.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:10 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:17:12.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:12 smithi062 ceph-mon[35761]: pgmap v293: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-08T23:17:12.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:12 smithi046 ceph-mon[31859]: pgmap v293: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-08T23:17:12.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:12 smithi192 ceph-mon[36700]: pgmap v293: 129 pgs: 129 active+clean; 78 MiB data, 522 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 127 B/s wr, 4 op/s 2024-03-08T23:17:14.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:14 smithi062 ceph-mon[35761]: pgmap v294: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:14.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:14 smithi046 ceph-mon[31859]: pgmap v294: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:14.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:14 smithi192 ceph-mon[36700]: pgmap v294: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:16.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:16 smithi062 ceph-mon[35761]: pgmap v295: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:16.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:16 smithi046 ceph-mon[31859]: pgmap v295: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:16.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:16 smithi192 ceph-mon[36700]: pgmap v295: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-08T23:17:17.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:17 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:17 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:17.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:17 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:17 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:17.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:17 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:17.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:17 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:18.626 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:18 smithi046 ceph-mon[31859]: pgmap v296: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 207 B/s wr, 6 op/s 2024-03-08T23:17:18.627 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:18 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:18.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:18 smithi062 ceph-mon[35761]: pgmap v296: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 207 B/s wr, 6 op/s 2024-03-08T23:17:18.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:18 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:18.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:18 smithi192 ceph-mon[36700]: pgmap v296: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 207 B/s wr, 6 op/s 2024-03-08T23:17:18.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:18 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:20.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:20 smithi062 ceph-mon[35761]: pgmap v297: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:17:20.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:20 smithi046 ceph-mon[31859]: pgmap v297: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:17:20.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:20 smithi192 ceph-mon[36700]: pgmap v297: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:17:22.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:22 smithi062 ceph-mon[35761]: pgmap v298: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2024-03-08T23:17:22.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:22 smithi046 ceph-mon[31859]: pgmap v298: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2024-03-08T23:17:22.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:22 smithi192 ceph-mon[36700]: pgmap v298: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 B/s wr, 5 op/s 2024-03-08T23:17:24.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:24 smithi062 ceph-mon[35761]: pgmap v299: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:17:24.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:24 smithi046 ceph-mon[31859]: pgmap v299: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:17:24.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:24 smithi192 ceph-mon[36700]: pgmap v299: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 7 op/s 2024-03-08T23:17:26.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:26 smithi062 ceph-mon[35761]: pgmap v300: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:26.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:26 smithi046 ceph-mon[31859]: pgmap v300: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:26.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:26 smithi192 ceph-mon[36700]: pgmap v300: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:27.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:27 smithi046 ceph-mon[31859]: pgmap v301: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:27.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:27 smithi192 ceph-mon[36700]: pgmap v301: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:27.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:27 smithi062 ceph-mon[35761]: pgmap v301: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:30.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:29 smithi192 ceph-mon[36700]: pgmap v302: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:30.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:29 smithi062 ceph-mon[35761]: pgmap v302: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:30.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:29 smithi046 ceph-mon[31859]: pgmap v302: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:32.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:31 smithi192 ceph-mon[36700]: pgmap v303: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:32.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:31 smithi062 ceph-mon[35761]: pgmap v303: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:32.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:31 smithi046 ceph-mon[31859]: pgmap v303: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:33.047 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:17:33.047 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:17:33.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 -- 172.21.15.46:0/4127175926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34112130 msgr2=0x7f0d34114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 --2- 172.21.15.46:0/4127175926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34112130 0x7f0d34114520 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c009930 tx=0x7f0d2c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:17:33.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 -- 172.21.15.46:0/4127175926 shutdown_connections 2024-03-08T23:17:33.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 --2- 172.21.15.46:0/4127175926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34112130 0x7f0d34114520 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 --2- 172.21.15.46:0/4127175926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34076140 0x7f0d34111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 --2- 172.21.15.46:0/4127175926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34075820 0x7f0d34075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 -- 172.21.15.46:0/4127175926 >> 172.21.15.46:0/4127175926 conn(0x7f0d340fefa0 msgr2=0x7f0d341013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 -- 172.21.15.46:0/4127175926 shutdown_connections 2024-03-08T23:17:33.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.280+0000 7f0d3a600640 1 -- 172.21.15.46:0/4127175926 wait complete. 2024-03-08T23:17:33.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 Processor -- start 2024-03-08T23:17:33.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 -- start start 2024-03-08T23:17:33.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 0x7f0d3419e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.286 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34076140 0x7f0d3419e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34117ce0 con 0x7f0d34112130 2024-03-08T23:17:33.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34117b60 con 0x7f0d34075820 2024-03-08T23:17:33.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d3a600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34117e60 con 0x7f0d34076140 2024-03-08T23:17:33.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.282+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44108/0 (socket says 172.21.15.46:44108) 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 -- 172.21.15.46:0/341501608 learned_addr learned my addr 172.21.15.46:0/341501608 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33400640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 0x7f0d3419e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d32a00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34076140 0x7f0d3419e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 -- 172.21.15.46:0/341501608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34076140 msgr2=0x7f0d3419e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34076140 0x7f0d3419e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 -- 172.21.15.46:0/341501608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 msgr2=0x7f0d3419e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 0x7f0d3419e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 -- 172.21.15.46:0/341501608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d341a3190 con 0x7f0d34112130 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33400640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 0x7f0d3419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.283+0000 7f0d33e00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c036040 tx=0x7f0d2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.284+0000 7f0d2be00640 1 -- 172.21.15.46:0/341501608 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c043020 con 0x7f0d34112130 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.284+0000 7f0d2be00640 1 -- 172.21.15.46:0/341501608 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c005000 con 0x7f0d34112130 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.284+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d341a3420 con 0x7f0d34112130 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.284+0000 7f0d2be00640 1 -- 172.21.15.46:0/341501608 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c003ab0 con 0x7f0d34112130 2024-03-08T23:17:33.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.284+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341aad10 con 0x7f0d34112130 2024-03-08T23:17:33.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.288+0000 7f0d3d434740 1 Processor -- start 2024-03-08T23:17:33.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.288+0000 7f0d2be00640 1 -- 172.21.15.46:0/341501608 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0d2c046020 con 0x7f0d34112130 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 -- start start 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f9a80 0x56311e6fbe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e6b2410 con 0x56311e6f0ea0 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e654d50 con 0x56311e6f9a80 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d3d434740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e5fdb80 con 0x56311e6f0580 2024-03-08T23:17:33.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44112/0 (socket says 172.21.15.46:44112) 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d33400640 1 -- 172.21.15.46:0/4257646926 learned_addr learned my addr 172.21.15.46:0/4257646926 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.289+0000 7f0d32a00640 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d32a00640 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f9a80 0x56311e6fbe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 -- 172.21.15.46:0/4257646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 msgr2=0x56311e6f0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 -- 172.21.15.46:0/4257646926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f9a80 msgr2=0x56311e6fbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f9a80 0x56311e6fbe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.296 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 -- 172.21.15.46:0/4257646926 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56311e4337d0 con 0x56311e6f0ea0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d33400640 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0d24011860 tx=0x7f0d24011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee75ebb745f1be4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d2be00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 0x7f0d180a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d28c00640 1 -- 172.21.15.46:0/4257646926 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d240112f0 con 0x56311e6f0ea0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d28c00640 1 -- 172.21.15.46:0/4257646926 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2400f9a0 con 0x56311e6f0ea0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.290+0000 7f0d28c00640 1 -- 172.21.15.46:0/4257646926 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24005700 con 0x56311e6f0ea0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d2be00640 1 -- 172.21.15.46:0/341501608 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0d2c0f1260 con 0x7f0d34112130 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d32a00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 0x7f0d180a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.297 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 -- 172.21.15.46:0/4257646926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 msgr2=0x56311e6f9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f0d24011860 tx=0x7f0d24011d20 comp rx=0 tx=0).stop 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 -- 172.21.15.46:0/4257646926 shutdown_connections 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f9a80 0x56311e6fbe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0ea0 0x56311e6f9540 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 --2- 172.21.15.46:0/4257646926 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 -- 172.21.15.46:0/4257646926 >> 172.21.15.46:0/4257646926 conn(0x56311e6eacb0 msgr2=0x56311e6eb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 -- 172.21.15.46:0/4257646926 shutdown_connections 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d32a00640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 0x7f0d180a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x56311e6ebe40 tx=0x7f0d1c0046c0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.298 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.291+0000 7f0d3d434740 1 -- 172.21.15.46:0/4257646926 wait complete. 2024-03-08T23:17:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.292+0000 7f0d3d434740 1 -- start start 2024-03-08T23:17:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.292+0000 7f0d3d434740 1 Processor -- start 2024-03-08T23:17:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e6f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e610ee0 con 0x56311e6f9a80 2024-03-08T23:17:33.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e650df0 con 0x56311e6f0ea0 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x56311e6fc870 con 0x56311e6f0580 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d3d434740 10 client.0 _ll_register_callbacks cb 0x56311e6178d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e6f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e6f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44128/0 (socket says 172.21.15.46:44128) 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33e00640 1 -- 172.21.15.46:0/1236735851 learned_addr learned my addr 172.21.15.46:0/1236735851 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 -- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 msgr2=0x56311e6f6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 -- 172.21.15.46:0/1236735851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 msgr2=0x56311e6f7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e6f7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.293+0000 7f0d33400640 1 -- 172.21.15.46:0/1236735851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56311e3693f0 con 0x56311e6f0580 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d33400640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f69b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400f790 tx=0x7f0d24009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66b8bcbbb9fc324f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d21600640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24028030 con 0x56311e6f0580 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d2400f9a0 con 0x56311e6f0580 2024-03-08T23:17:33.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0d24004c50 con 0x56311e6f0580 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d24032050 con 0x56311e6f0580 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d3d434740 10 client.27517 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.294+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56311e3290f0 con 0x56311e6f0580 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2400f9a0 con 0x56311e6f0580 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0d24018070 con 0x56311e6f0580 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 10 client.27517 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 10 client.27517 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 20 client.27517 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:33.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e709b20 0x56311e70bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 4 client.27517 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56311e662290 con 0x56311e709b20 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.295+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e709b20 0x56311e70bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.296+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e709b20 0x56311e70bf10 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58b4f8abbf91474f server_cookie=e730a9eabb07dc90 in_seq=0 out_seq=0 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.296+0000 7f0d21600640 10 client.27517 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d21600640 1 -- 172.21.15.46:0/1236735851 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56311e662290 con 0x56311e709b20 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d21600640 10 client.27517 handle_command_reply: tid=0 2024-03-08T23:17:33.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 client.27517 shutdown 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 msgr2=0x56311e6f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f69b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0d2400f790 tx=0x7f0d24009b20 comp rx=0 tx=0).stop 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 shutdown_connections 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e709b20 0x56311e70bf10 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e6f7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1236735851 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f69b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d21600640 0 client.27517 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:33.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 >> 172.21.15.46:0/1236735851 conn(0x56311e6eacb0 msgr2=0x56311e6eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 shutdown_connections 2024-03-08T23:17:33.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 1 -- 172.21.15.46:0/1236735851 wait complete. 2024-03-08T23:17:33.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.297+0000 7f0d3d434740 20 client.27517 trim_cache size 0 max 16384 2024-03-08T23:17:33.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 Processor -- start 2024-03-08T23:17:33.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 -- start start 2024-03-08T23:17:33.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b71b0 con 0x56311e6f9a80 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b4d50 con 0x56311e6f0ea0 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d3d434740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b2e10 con 0x56311e6f0580 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d33e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44138/0 (socket says 172.21.15.46:44138) 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.314+0000 7f0d33e00640 1 -- 172.21.15.46:0/3407045304 learned_addr learned my addr 172.21.15.46:0/3407045304 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33400640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d32a00640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 -- 172.21.15.46:0/3407045304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 msgr2=0x56311e6f90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 -- 172.21.15.46:0/3407045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 msgr2=0x56311e6f58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 -- 172.21.15.46:0/3407045304 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56311e6fbe70 con 0x56311e6f9a80 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33400640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.315+0000 7f0d33e00640 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0bd2b0 tx=0x7f0d2c0bd440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=496343778238f8eb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.320 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d23e00640 1 -- 172.21.15.46:0/3407045304 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0d6070 con 0x56311e6f9a80 2024-03-08T23:17:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d23e00640 1 -- 172.21.15.46:0/3407045304 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c0bdcb0 con 0x56311e6f9a80 2024-03-08T23:17:33.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d23e00640 1 -- 172.21.15.46:0/3407045304 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c0bb8b0 con 0x56311e6f9a80 2024-03-08T23:17:33.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 -- 172.21.15.46:0/3407045304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 msgr2=0x56311e700b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c0bd2b0 tx=0x7f0d2c0bd440 comp rx=0 tx=0).stop 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 -- 172.21.15.46:0/3407045304 shutdown_connections 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f9a80 0x56311e700b10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e6f58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 --2- 172.21.15.46:0/3407045304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f0580 0x56311e6f90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 -- 172.21.15.46:0/3407045304 >> 172.21.15.46:0/3407045304 conn(0x56311e6eacb0 msgr2=0x56311e6f5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 -- 172.21.15.46:0/3407045304 shutdown_connections 2024-03-08T23:17:33.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.316+0000 7f0d3d434740 1 -- 172.21.15.46:0/3407045304 wait complete. 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.317+0000 7f0d3d434740 1 -- start start 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.317+0000 7f0d3d434740 1 Processor -- start 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.317+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.317+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e7547c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d3d434740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f9a80 0x56311e7591c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d3d434740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b3410 con 0x56311e6f0580 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d3d434740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b46f0 con 0x56311e6f0ea0 2024-03-08T23:17:33.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d3d434740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341b3a30 con 0x56311e6f9a80 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d3d434740 10 client.0 _ll_register_callbacks cb 0x7f0d24009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44144/0 (socket says 172.21.15.46:44144) 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 -- 172.21.15.46:0/1229025890 learned_addr learned my addr 172.21.15.46:0/1229025890 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e7547c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33e00640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f9a80 0x56311e7591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 -- 172.21.15.46:0/1229025890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f9a80 msgr2=0x56311e7591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f9a80 0x56311e7591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 -- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 msgr2=0x56311e7547c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e7547c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 -- 172.21.15.46:0/1229025890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56311e7599c0 con 0x56311e6f0580 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d33400640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0d24005d70 tx=0x7f0d2400fe30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f218f298bf7fcdb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.318+0000 7f0d21600640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d24028040 con 0x56311e6f0580 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d2c0bdcb0 con 0x56311e6f0580 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0d2400f7c0 con 0x56311e6f0580 2024-03-08T23:17:33.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d24024be0 con 0x56311e6f0580 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d3d434740 10 client.17412 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56311e760d70 con 0x56311e6f0580 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d2c0bdcb0 con 0x56311e6f0580 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.319+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0d24018040 con 0x56311e6f0580 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 10 client.17412 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 10 client.17412 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 20 client.17412 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e761350 0x56311e763770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 4 client.17412 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56311e65c5e0 con 0x56311e761350 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.320+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e761350 0x56311e763770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d32a00640 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e761350 0x56311e763770 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e7126eed3a24106 server_cookie=53ad8ba6fea22fb4 in_seq=0 out_seq=0 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d21600640 10 client.17412 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d21600640 1 -- 172.21.15.46:0/1229025890 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56311e65c5e0 con 0x56311e761350 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d21600640 10 client.17412 handle_command_reply: tid=0 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 client.17412 shutdown 2024-03-08T23:17:33.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 msgr2=0x56311e754280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f0d24005d70 tx=0x7f0d2400fe30 comp rx=0 tx=0).stop 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 shutdown_connections 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x56311e761350 0x56311e763770 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x56311e6f9a80 0x56311e7591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x56311e6f0ea0 0x56311e7547c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 --2- 172.21.15.46:0/1229025890 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x56311e6f0580 0x56311e754280 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d21600640 0 client.17412 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:33.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 >> 172.21.15.46:0/1229025890 conn(0x56311e6eacb0 msgr2=0x56311e6e9f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 shutdown_connections 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 1 -- 172.21.15.46:0/1229025890 wait complete. 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.321+0000 7f0d3d434740 20 client.17412 trim_cache size 0 max 16384 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:17:33.332 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:17:33.333 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:33.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 msgr2=0x7f0d180a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 0x7f0d180a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x56311e6ebe40 tx=0x7f0d1c0046c0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 msgr2=0x7f0d341a2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c036040 tx=0x7f0d2c004270 comp rx=0 tx=0).stop 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 shutdown_connections 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f0d180a2970 0x7f0d180a4e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0d34112130 0x7f0d341a2a60 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0d34076140 0x7f0d3419e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 --2- 172.21.15.46:0/341501608 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0d34075820 0x7f0d3419e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 >> 172.21.15.46:0/341501608 conn(0x7f0d340fefa0 msgr2=0x7f0d34101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 shutdown_connections 2024-03-08T23:17:33.335 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.322+0000 7f0d3a600640 1 -- 172.21.15.46:0/341501608 wait complete. 2024-03-08T23:17:33.342 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:17:33.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 -- 172.21.15.46:0/1433784413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 msgr2=0x7fae68111d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 --2- 172.21.15.46:0/1433784413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae68111d50 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fae58009900 tx=0x7fae5802bcd0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 -- 172.21.15.46:0/1433784413 shutdown_connections 2024-03-08T23:17:33.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 --2- 172.21.15.46:0/1433784413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68112290 0x7fae68114680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 --2- 172.21.15.46:0/1433784413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae68111d50 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 --2- 172.21.15.46:0/1433784413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68073690 0x7fae68073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 -- 172.21.15.46:0/1433784413 >> 172.21.15.46:0/1433784413 conn(0x7fae680fcd90 msgr2=0x7fae680ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 -- 172.21.15.46:0/1433784413 shutdown_connections 2024-03-08T23:17:33.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.563+0000 7fae70800640 1 -- 172.21.15.46:0/1433784413 wait complete. 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.564+0000 7fae70800640 1 Processor -- start 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.564+0000 7fae70800640 1 -- start start 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 0x7fae6819e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68112290 0x7fae681a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae68117c90 con 0x7fae68073fb0 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae68117b10 con 0x7fae68073690 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae70800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae68117e10 con 0x7fae68112290 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6de00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 0x7fae6819e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6d400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44162/0 (socket says 172.21.15.46:44162) 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6d400640 1 -- 172.21.15.46:0/2210336262 learned_addr learned my addr 172.21.15.46:0/2210336262 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6e800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68112290 0x7fae681a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.565+0000 7fae6de00640 1 -- 172.21.15.46:0/2210336262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68112290 msgr2=0x7fae681a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6de00640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68112290 0x7fae681a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6de00640 1 -- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 msgr2=0x7fae6819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6de00640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6de00640 1 -- 172.21.15.46:0/2210336262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae681a31b0 con 0x7fae68073690 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6d400640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae6de00640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 0x7fae6819e210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9f0 tx=0x7fae5c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c00cd80 con 0x7fae68073690 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae681a34a0 con 0x7fae68073690 2024-03-08T23:17:33.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae5c0035f0 con 0x7fae68073690 2024-03-08T23:17:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c010670 con 0x7fae68073690 2024-03-08T23:17:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.566+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae681aad30 con 0x7fae68073690 2024-03-08T23:17:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.567+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae68068530 con 0x7fae68073690 2024-03-08T23:17:33.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.569+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fae5c0037a0 con 0x7fae68073690 2024-03-08T23:17:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.571+0000 7fae62a00640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 0x7fae500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:33.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.571+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fae5c0cf4c0 con 0x7fae68073690 2024-03-08T23:17:33.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.571+0000 7fae6d400640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 0x7fae500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:33.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.572+0000 7fae6d400640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 0x7fae500a4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fae58000c00 tx=0x7fae5802cf90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:33.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.575+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fae5c09be00 con 0x7fae68073690 2024-03-08T23:17:33.821 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.819+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae68104760 con 0x7fae68073690 2024-03-08T23:17:33.822 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.820+0000 7fae62a00640 1 -- 172.21.15.46:0/2210336262 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fae5c09b700 con 0x7fae68073690 2024-03-08T23:17:33.823 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:17:33.823 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:17:33.823 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:17:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.823+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 msgr2=0x7fae500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.823+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 0x7fae500a4eb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fae58000c00 tx=0x7fae5802cf90 comp rx=0 tx=0).stop 2024-03-08T23:17:33.826 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 msgr2=0x7fae6819e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 0x7fae6819e210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fae5c00e9f0 tx=0x7fae5c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 shutdown_connections 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fae500a29f0 0x7fae500a4eb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fae68112290 0x7fae681a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fae68073fb0 0x7fae6819e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 --2- 172.21.15.46:0/2210336262 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fae68073690 0x7fae6819e210 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:33.827 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 >> 172.21.15.46:0/2210336262 conn(0x7fae680fcd90 msgr2=0x7fae680fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 shutdown_connections 2024-03-08T23:17:33.828 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:33.824+0000 7fae70800640 1 -- 172.21.15.46:0/2210336262 wait complete. 2024-03-08T23:17:33.847 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:33.847 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:17:34.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:33 smithi192 ceph-mon[36700]: pgmap v304: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:34.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.083+0000 7fc46be00640 1 -- 172.21.15.46:0/1774706858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 msgr2=0x7fc46c10c6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.083+0000 7fc46be00640 1 --2- 172.21.15.46:0/1774706858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c10c6e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fc46400adb0 tx=0x7fc46402cff0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 -- 172.21.15.46:0/1774706858 shutdown_connections 2024-03-08T23:17:34.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 --2- 172.21.15.46:0/1774706858 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c10c6e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.086 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 --2- 172.21.15.46:0/1774706858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c105df0 0x7fc46c0ffc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 --2- 172.21.15.46:0/1774706858 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c067770 0x7fc46c0ff6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 -- 172.21.15.46:0/1774706858 >> 172.21.15.46:0/1774706858 conn(0x7fc46c074e90 msgr2=0x7fc46c0752a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 -- 172.21.15.46:0/1774706858 shutdown_connections 2024-03-08T23:17:34.087 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.084+0000 7fc46be00640 1 -- 172.21.15.46:0/1774706858 wait complete. 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 Processor -- start 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 -- start start 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c105df0 0x7fc46c19fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.089 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c113800 con 0x7fc46c10a2f0 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c113680 con 0x7fc46c105df0 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.085+0000 7fc46be00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c113980 con 0x7fc46c067770 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59566/0 (socket says 172.21.15.46:59566) 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 -- 172.21.15.46:0/3748559052 learned_addr learned my addr 172.21.15.46:0/3748559052 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc463e00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c105df0 0x7fc46c19fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.090 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 -- 172.21.15.46:0/3748559052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c105df0 msgr2=0x7fc46c19fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c105df0 0x7fc46c19fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 -- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 msgr2=0x7fc46c199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 -- 172.21.15.46:0/3748559052 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c19a280 con 0x7fc46c067770 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e9f0 tx=0x7fc45c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.086+0000 7fc463400640 1 -- 172.21.15.46:0/3748559052 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00cd80 con 0x7fc46c067770 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.087+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c19a570 con 0x7fc46c067770 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.087+0000 7fc463400640 1 -- 172.21.15.46:0/3748559052 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc45c010910 con 0x7fc46c067770 2024-03-08T23:17:34.091 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.087+0000 7fc463400640 1 -- 172.21.15.46:0/3748559052 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c0188f0 con 0x7fc46c067770 2024-03-08T23:17:34.092 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.087+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c1a6810 con 0x7fc46c067770 2024-03-08T23:17:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc463400640 1 -- 172.21.15.46:0/3748559052 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc45c010ac0 con 0x7fc46c067770 2024-03-08T23:17:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 Processor -- start 2024-03-08T23:17:34.099 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 -- start start 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b1c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b254f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456a80dd0 con 0x55a456b1c530 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456a29c70 con 0x55a456b25a30 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.090+0000 7fc4747e3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456a7ce70 con 0x55a456b1ce50 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc46b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b1c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.100 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc46b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b1c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44176/0 (socket says 172.21.15.46:44176) 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36850/0 (socket says 172.21.15.46:36850) 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc46b400640 1 -- 172.21.15.46:0/3997898068 learned_addr learned my addr 172.21.15.46:0/3997898068 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b254f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 -- 172.21.15.46:0/3997898068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 msgr2=0x55a456b254f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b254f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 -- 172.21.15.46:0/3997898068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 msgr2=0x55a456b1c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b1c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.101 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.091+0000 7fc463e00640 1 -- 172.21.15.46:0/3997898068 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a456ae32b0 con 0x55a456b25a30 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc463400640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 0x7fc4500a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc463e00640 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc46c19ac40 tx=0x7fc454009370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4ad8506853e947e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc463400640 1 -- 172.21.15.46:0/3748559052 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc45c0cf490 con 0x7fc46c067770 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc46b400640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 0x7fc4500a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc45be00640 1 -- 172.21.15.46:0/3997898068 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454004050 con 0x55a456b25a30 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc45be00640 1 -- 172.21.15.46:0/3997898068 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4540041f0 con 0x55a456b25a30 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.092+0000 7fc45be00640 1 -- 172.21.15.46:0/3997898068 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc454019640 con 0x55a456b25a30 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- 172.21.15.46:0/3997898068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 msgr2=0x55a456b27e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.102 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc46c19ac40 tx=0x7fc454009370 comp rx=0 tx=0).stop 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc46b400640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 0x7fc4500a4ea0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc464006010 tx=0x7fc46402ff70 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- 172.21.15.46:0/3997898068 shutdown_connections 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b27e20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b254f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3997898068 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b1c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- 172.21.15.46:0/3997898068 >> 172.21.15.46:0/3997898068 conn(0x55a456b16c60 msgr2=0x55a456b17040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- 172.21.15.46:0/3997898068 shutdown_connections 2024-03-08T23:17:34.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- 172.21.15.46:0/3997898068 wait complete. 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.093+0000 7fc4747e3740 1 -- start start 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 Processor -- start 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 0x55a456b253c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 0x55a456b22750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b22cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456b28820 con 0x55a456b1ce50 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456a3d0c0 con 0x55a456b25a30 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a456b289a0 con 0x55a456b1c530 2024-03-08T23:17:34.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc4747e3740 10 client.0 _ll_register_callbacks cb 0x55a456a77ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc463e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 0x55a456b22750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc46aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 0x55a456b253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.094+0000 7fc463e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 0x55a456b22750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44178/0 (socket says 172.21.15.46:44178) 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46b400640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b22cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc463e00640 1 -- 172.21.15.46:0/3236740813 learned_addr learned my addr 172.21.15.46:0/3236740813 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 -- 172.21.15.46:0/3236740813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 msgr2=0x55a456b22cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b22cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 -- 172.21.15.46:0/3236740813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 msgr2=0x55a456b22750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 0x55a456b22750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 -- 172.21.15.46:0/3236740813 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4567953f0 con 0x55a456b1c530 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 0x55a456b253c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0cc8c0 tx=0x7fc45c009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3404ebdd8e5140b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc458c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c09b1e0 con 0x55a456b1c530 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4540041f0 con 0x55a456b1c530 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.095+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc45c0860e0 con 0x55a456b1c530 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc45c09b880 con 0x55a456b1c530 2024-03-08T23:17:34.106 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 10 client.27541 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a45685f7d0 con 0x55a456b1c530 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4540041f0 con 0x55a456b1c530 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc45c018af0 con 0x55a456b1c530 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 10 client.27541 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 10 client.27541 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 20 client.27541 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b35a40 0x55a456b37e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 4 client.27541 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.096+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a456a554f0 con 0x55a456b35a40 2024-03-08T23:17:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.097+0000 7fc463e00640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b35a40 0x55a456b37e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.097+0000 7fc463e00640 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b35a40 0x55a456b37e30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8fadd4d27c299b45 server_cookie=b07da6fc50c83cb7 in_seq=0 out_seq=0 2024-03-08T23:17:34.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.097+0000 7fc458c00640 10 client.27541 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc458c00640 1 -- 172.21.15.46:0/3236740813 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a456a554f0 con 0x55a456b35a40 2024-03-08T23:17:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc458c00640 10 client.27541 handle_command_reply: tid=0 2024-03-08T23:17:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 client.27541 shutdown 2024-03-08T23:17:34.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 msgr2=0x55a456b253c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 0x55a456b253c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0cc8c0 tx=0x7fc45c009310 comp rx=0 tx=0).stop 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 shutdown_connections 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b35a40 0x55a456b37e30 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b22cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1ce50 0x55a456b22750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3236740813 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1c530 0x55a456b253c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc458c00640 0 client.27541 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:34.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.098+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 >> 172.21.15.46:0/3236740813 conn(0x55a456b16c60 msgr2=0x55a456b189b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.099+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 shutdown_connections 2024-03-08T23:17:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.099+0000 7fc4747e3740 1 -- 172.21.15.46:0/3236740813 wait complete. 2024-03-08T23:17:34.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.099+0000 7fc4747e3740 20 client.27541 trim_cache size 0 max 16384 2024-03-08T23:17:34.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 Processor -- start 2024-03-08T23:17:34.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 -- start start 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b253c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b21020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1b2a50 con 0x55a456b1c530 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1b05f0 con 0x55a456b25a30 2024-03-08T23:17:34.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.112+0000 7fc4747e3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1ae6b0 con 0x55a456b1ce50 2024-03-08T23:17:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44180/0 (socket says 172.21.15.46:44180) 2024-03-08T23:17:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 -- 172.21.15.46:0/719507664 learned_addr learned my addr 172.21.15.46:0/719507664 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46b400640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b21020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc463e00640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b253c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 -- 172.21.15.46:0/719507664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 msgr2=0x55a456b253c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b253c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 -- 172.21.15.46:0/719507664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 msgr2=0x55a456b21020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b21020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc46aa00640 1 -- 172.21.15.46:0/719507664 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a456b22520 con 0x55a456b1c530 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.113+0000 7fc463e00640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc46aa00640 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc45c09b580 tx=0x7fc45c00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c10b0dca7b43d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc45b400640 1 -- 172.21.15.46:0/719507664 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c09f920 con 0x55a456b1c530 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc45b400640 1 -- 172.21.15.46:0/719507664 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc45c0a3f70 con 0x55a456b1c530 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc45b400640 1 -- 172.21.15.46:0/719507664 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45c097620 con 0x55a456b1c530 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 -- 172.21.15.46:0/719507664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 msgr2=0x55a456b24e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fc45c09b580 tx=0x7fc45c00c7d0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 -- 172.21.15.46:0/719507664 shutdown_connections 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b25a30 0x55a456b21020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b253c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.114+0000 7fc4747e3740 1 --2- 172.21.15.46:0/719507664 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b1c530 0x55a456b24e80 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.115+0000 7fc4747e3740 1 -- 172.21.15.46:0/719507664 >> 172.21.15.46:0/719507664 conn(0x55a456b16c60 msgr2=0x55a456b3c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.115+0000 7fc4747e3740 1 -- 172.21.15.46:0/719507664 shutdown_connections 2024-03-08T23:17:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.115+0000 7fc4747e3740 1 -- 172.21.15.46:0/719507664 wait complete. 2024-03-08T23:17:34.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.115+0000 7fc4747e3740 1 -- start start 2024-03-08T23:17:34.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.115+0000 7fc4747e3740 1 Processor -- start 2024-03-08T23:17:34.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 0x55a456b2cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b25a30 0x55a456b2b090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1aecb0 con 0x55a456b25a30 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1aff90 con 0x55a456b1c530 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c1af2d0 con 0x55a456b1ce50 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc4747e3740 10 client.0 _ll_register_callbacks cb 0x7fc454009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc46b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b25a30 0x55a456b2b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59588/0 (socket says 172.21.15.46:59588) 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 -- 172.21.15.46:0/3125977102 learned_addr learned my addr 172.21.15.46:0/3125977102 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 0x55a456b2cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 -- 172.21.15.46:0/3125977102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 msgr2=0x55a456b2cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 0x55a456b2cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 -- 172.21.15.46:0/3125977102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b25a30 msgr2=0x55a456b2b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b25a30 0x55a456b2b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc463e00640 1 -- 172.21.15.46:0/3125977102 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a456b2b890 con 0x55a456b1ce50 2024-03-08T23:17:34.125 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.116+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 0x55a456b2cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc463e00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc454005f30 tx=0x7fc454005c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b863b75053c697b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc458c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc45402d070 con 0x55a456b1ce50 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fc454011c90 con 0x55a456b1ce50 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc45c0a3f70 con 0x55a456b1ce50 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc454039050 con 0x55a456b1ce50 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc4747e3740 10 client.27547 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.117+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a456b8caa0 con 0x55a456b1ce50 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc45c0a3f70 con 0x55a456b1ce50 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc454018070 con 0x55a456b1ce50 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 10 client.27547 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 10 client.27547 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 20 client.27547 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b8d040 0x55a456b8f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 4 client.27547 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a456a8e310 con 0x55a456b8d040 2024-03-08T23:17:34.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.118+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b8d040 0x55a456b8f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc46aa00640 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b8d040 0x55a456b8f460 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6cf3fb52d823d6ff server_cookie=5d6de4a3bdc569d7 in_seq=0 out_seq=0 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc458c00640 10 client.27547 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc458c00640 1 -- 172.21.15.46:0/3125977102 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a456a8e310 con 0x55a456b8d040 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc458c00640 10 client.27547 handle_command_reply: tid=0 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 client.27547 shutdown 2024-03-08T23:17:34.128 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 msgr2=0x55a456b2d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc454005f30 tx=0x7fc454005c50 comp rx=0 tx=0).stop 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 shutdown_connections 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x55a456b8d040 0x55a456b8f460 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55a456b25a30 0x55a456b2b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55a456b1ce50 0x55a456b2d040 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc458c00640 0 client.27547 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 --2- 172.21.15.46:0/3125977102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55a456b1c530 0x55a456b2cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 >> 172.21.15.46:0/3125977102 conn(0x55a456b16c60 msgr2=0x55a456b20f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.129 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 shutdown_connections 2024-03-08T23:17:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 1 -- 172.21.15.46:0/3125977102 wait complete. 2024-03-08T23:17:34.130 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.119+0000 7fc4747e3740 20 client.27547 trim_cache size 0 max 16384 2024-03-08T23:17:34.130 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 msgr2=0x7fc4500a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 0x7fc4500a4ea0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc464006010 tx=0x7fc46402ff70 comp rx=0 tx=0).stop 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 msgr2=0x7fc46c19f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc45c00e9f0 tx=0x7fc45c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 shutdown_connections 2024-03-08T23:17:34.131 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fc4500a29e0 0x7fc4500a4ea0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fc46c10a2f0 0x7fc46c199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fc46c105df0 0x7fc46c19fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 --2- 172.21.15.46:0/3748559052 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc46c067770 0x7fc46c19f950 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.132 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 >> 172.21.15.46:0/3748559052 conn(0x7fc46c074e90 msgr2=0x7fc46c10b140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 shutdown_connections 2024-03-08T23:17:34.133 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.120+0000 7fc46be00640 1 -- 172.21.15.46:0/3748559052 wait complete. 2024-03-08T23:17:34.137 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:33 smithi046 ceph-mon[31859]: pgmap v304: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:34.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:33 smithi062 ceph-mon[35761]: pgmap v304: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:17:34.142 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:34.142 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:17:34.375 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 -- 172.21.15.46:0/4152506303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 msgr2=0x7fec14110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 --2- 172.21.15.46:0/4152506303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec14110330 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009c10 tx=0x7fec0c02bee0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 -- 172.21.15.46:0/4152506303 shutdown_connections 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 --2- 172.21.15.46:0/4152506303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec14110330 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 --2- 172.21.15.46:0/4152506303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 0x7fec14103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 --2- 172.21.15.46:0/4152506303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 0x7fec14109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.376 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 -- 172.21.15.46:0/4152506303 >> 172.21.15.46:0/4152506303 conn(0x7fec140fef40 msgr2=0x7fec14101360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 -- 172.21.15.46:0/4152506303 shutdown_connections 2024-03-08T23:17:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.373+0000 7fec1b800640 1 -- 172.21.15.46:0/4152506303 wait complete. 2024-03-08T23:17:34.377 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.374+0000 7fec1b800640 1 Processor -- start 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.374+0000 7fec1b800640 1 -- start start 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 0x7fec1419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 0x7fec14199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14113880 con 0x7fec141034b0 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14113700 con 0x7fec14103e70 2024-03-08T23:17:34.378 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec1b800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14113a00 con 0x7fec14109570 2024-03-08T23:17:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec13e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36908/0 (socket says 172.21.15.46:36908) 2024-03-08T23:17:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec13e00640 1 -- 172.21.15.46:0/4236555704 learned_addr learned my addr 172.21.15.46:0/4236555704 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec19800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 0x7fec14199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.375+0000 7fec18e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 0x7fec1419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 -- 172.21.15.46:0/4236555704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 msgr2=0x7fec14199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 0x7fec14199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.380 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 -- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 msgr2=0x7fec1419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 0x7fec1419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 -- 172.21.15.46:0/4236555704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1419a3e0 con 0x7fec14103e70 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.376+0000 7fec13e00640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.377+0000 7fec11600640 1 -- 172.21.15.46:0/4236555704 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0400cde0 con 0x7fec14103e70 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.377+0000 7fec11600640 1 -- 172.21.15.46:0/4236555704 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec040035f0 con 0x7fec14103e70 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.377+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1419a6d0 con 0x7fec14103e70 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.377+0000 7fec11600640 1 -- 172.21.15.46:0/4236555704 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec04010690 con 0x7fec14103e70 2024-03-08T23:17:34.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.377+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec141a69e0 con 0x7fec14103e70 2024-03-08T23:17:34.382 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.380+0000 7fec11600640 1 -- 172.21.15.46:0/4236555704 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fec040037a0 con 0x7fec14103e70 2024-03-08T23:17:34.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.380+0000 7fec1e5cd740 1 Processor -- start 2024-03-08T23:17:34.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 -- start start 2024-03-08T23:17:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa55c70 0x55eecfa5e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa5e850 0x55eecfa60c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecf9b9dd0 con 0x55eecfa55c70 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecf962c70 con 0x55eecfa5e850 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.381+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecf9b5e70 con 0x55eecfa55350 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59624/0 (socket says 172.21.15.46:59624) 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 -- 172.21.15.46:0/3228160566 learned_addr learned my addr 172.21.15.46:0/3228160566 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec18e00640 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa55c70 0x55eecfa5e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec13e00640 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa5e850 0x55eecfa60c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 -- 172.21.15.46:0/3228160566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa5e850 msgr2=0x55eecfa60c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa5e850 0x55eecfa60c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 -- 172.21.15.46:0/3228160566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa55c70 msgr2=0x55eecfa5e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa55c70 0x55eecfa5e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 -- 172.21.15.46:0/3228160566 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eecfa1c2b0 con 0x55eecfa55350 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec11600640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec19800640 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009be0 tx=0x7fec0c009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23994a4fbb56d14b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.382+0000 7fec11600640 1 -- 172.21.15.46:0/4236555704 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fec040cf540 con 0x7fec14103e70 2024-03-08T23:17:34.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.383+0000 7fec18e00640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.383+0000 7fec02000640 1 -- 172.21.15.46:0/3228160566 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c043030 con 0x55eecfa55350 2024-03-08T23:17:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.383+0000 7fec02000640 1 -- 172.21.15.46:0/3228160566 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0c033c40 con 0x55eecfa55350 2024-03-08T23:17:34.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.383+0000 7fec02000640 1 -- 172.21.15.46:0/3228160566 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c02c880 con 0x55eecfa55350 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3228160566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 msgr2=0x55eecfa55730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec18e00640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 0x7febfc0a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55eecfa54e20 tx=0x7fec08005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009be0 tx=0x7fec0c009840 comp rx=0 tx=0).stop 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3228160566 shutdown_connections 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa5e850 0x55eecfa60c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa55c70 0x55eecfa5e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3228160566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55350 0x55eecfa55730 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3228160566 >> 172.21.15.46:0/3228160566 conn(0x55eecfa4fa80 msgr2=0x55eecfa4fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3228160566 shutdown_connections 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3228160566 wait complete. 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.384+0000 7fec1e5cd740 1 -- start start 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 Processor -- start 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa5e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecfa615e0 con 0x55eecfa5e850 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecf9760c0 con 0x55eecfa55350 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eecfa61760 con 0x55eecfa55c70 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.385+0000 7fec1e5cd740 10 client.0 _ll_register_callbacks cb 0x55eecf9b0ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec13e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec13e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59640/0 (socket says 172.21.15.46:59640) 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec13e00640 1 -- 172.21.15.46:0/2701689665 learned_addr learned my addr 172.21.15.46:0/2701689665 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec18e00640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa5e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 -- 172.21.15.46:0/2701689665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 msgr2=0x55eecfa5b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 -- 172.21.15.46:0/2701689665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 msgr2=0x55eecfa5e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa5e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 -- 172.21.15.46:0/2701689665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eecf6ce3f0 con 0x55eecfa5e850 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec13e00640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7fec19800640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5b9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009b30 tx=0x7fec0c033500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb90c2a9eb8fb8d server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.396 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.386+0000 7febf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c04c030 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fec0c0336f0 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec0c033c40 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec0c059050 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7fec1e5cd740 10 client.17448 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eecf7987d0 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.387+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0c033c40 con 0x55eecfa5e850 2024-03-08T23:17:34.397 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec0c004530 con 0x55eecfa5e850 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 10 client.17448 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 10 client.17448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 20 client.17448 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfa6e650 0x55eecfa70a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 4 client.17448 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eecf98e4f0 con 0x55eecfa6e650 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.388+0000 7fec18e00640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfa6e650 0x55eecfa70a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.398 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.389+0000 7fec18e00640 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfa6e650 0x55eecfa70a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1f86c15c91e6dde0 server_cookie=1f5ee9c823118f4a in_seq=0 out_seq=0 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.389+0000 7febf6a00640 10 client.17448 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.389+0000 7febf6a00640 1 -- 172.21.15.46:0/2701689665 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eecf98e4f0 con 0x55eecfa6e650 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.389+0000 7febf6a00640 10 client.17448 handle_command_reply: tid=0 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.389+0000 7fec1e5cd740 1 client.17448 shutdown 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 msgr2=0x55eecfa5b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5b9c0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009b30 tx=0x7fec0c033500 comp rx=0 tx=0).stop 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 shutdown_connections 2024-03-08T23:17:34.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5b9c0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfa6e650 0x55eecfa70a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/2701689665 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa5e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7febf6a00640 0 client.17448 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 >> 172.21.15.46:0/2701689665 conn(0x55eecfa4fa80 msgr2=0x55eecfa517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 shutdown_connections 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 1 -- 172.21.15.46:0/2701689665 wait complete. 2024-03-08T23:17:34.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.390+0000 7fec1e5cd740 20 client.17448 trim_cache size 0 max 16384 2024-03-08T23:17:34.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.403+0000 7fec1e5cd740 1 Processor -- start 2024-03-08T23:17:34.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.403+0000 7fec1e5cd740 1 -- start start 2024-03-08T23:17:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa4e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141b2c10 con 0x55eecfa5e850 2024-03-08T23:17:34.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141b07b0 con 0x55eecfa55350 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141ae870 con 0x55eecfa55c70 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44234/0 (socket says 172.21.15.46:44234) 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 -- 172.21.15.46:0/3133686638 learned_addr learned my addr 172.21.15.46:0/3133686638 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec13e00640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec18e00640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa4e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 -- 172.21.15.46:0/3133686638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 msgr2=0x55eecfa5dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 -- 172.21.15.46:0/3133686638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 msgr2=0x55eecfa4e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa4e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec19800640 1 -- 172.21.15.46:0/3133686638 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eecfa61630 con 0x55eecfa5e850 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.404+0000 7fec18e00640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa4e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec19800640 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fec0c004950 tx=0x7fec0c000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11cc22531a92b8e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec01600640 1 -- 172.21.15.46:0/3133686638 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c04c070 con 0x55eecfa5e850 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec01600640 1 -- 172.21.15.46:0/3133686638 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0c009800 con 0x55eecfa5e850 2024-03-08T23:17:34.409 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec01600640 1 -- 172.21.15.46:0/3133686638 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c043040 con 0x55eecfa5e850 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3133686638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 msgr2=0x55eecfa5e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.405+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fec0c004950 tx=0x7fec0c000f40 comp rx=0 tx=0).stop 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3133686638 shutdown_connections 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa5e0f0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa5dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3133686638 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa4e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3133686638 >> 172.21.15.46:0/3133686638 conn(0x55eecfa4fa80 msgr2=0x55eecfa4e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.414 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3133686638 shutdown_connections 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3133686638 wait complete. 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 -- start start 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.406+0000 7fec1e5cd740 1 Processor -- start 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa71160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141aee70 con 0x55eecfa5e850 2024-03-08T23:17:34.415 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141b0150 con 0x55eecfa55350 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec141af490 con 0x55eecfa55c70 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec1e5cd740 10 client.0 _ll_register_callbacks cb 0x7fec08005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec19800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44242/0 (socket says 172.21.15.46:44242) 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec19800640 1 -- 172.21.15.46:0/3167588208 learned_addr learned my addr 172.21.15.46:0/3167588208 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36950/0 (socket says 172.21.15.46:36950) 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec13e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa71160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.416 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 -- 172.21.15.46:0/3167588208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 msgr2=0x55eecfa71160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa71160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 -- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 msgr2=0x55eecfa716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec18e00640 1 -- 172.21.15.46:0/3167588208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eecfa71ed0 con 0x55eecfa55350 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.407+0000 7fec19800640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7fec18e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55eecfa5ca50 tx=0x7fec080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b8a97b821b50bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7febf6a00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:17:34.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec080043a0 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec0c009800 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fec08003b40 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec08003d40 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7fec1e5cd740 10 client.27620 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.408+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eecfac5770 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec0c009800 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec0801f9f0 con 0x55eecfa55350 2024-03-08T23:17:34.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 10 client.27620 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 10 client.27620 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 20 client.27620 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfac5e50 0x55eecfac8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 4 client.27620 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55eecf9c7310 con 0x55eecfac5e50 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.409+0000 7fec13e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfac5e50 0x55eecfac8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec13e00640 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfac5e50 0x55eecfac8270 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6144aae1c14c75b3 server_cookie=32ea6ac5ac66a51d in_seq=0 out_seq=0 2024-03-08T23:17:34.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7febf6a00640 10 client.27620 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7febf6a00640 1 -- 172.21.15.46:0/3167588208 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55eecf9c7310 con 0x55eecfac5e50 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7febf6a00640 10 client.27620 handle_command_reply: tid=0 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 client.27620 shutdown 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 msgr2=0x55eecfa72dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55eecfa5ca50 tx=0x7fec080046c0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 shutdown_connections 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55eecfa5e850 0x55eecfa716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55eecfac5e50 0x55eecfac8270 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55eecfa55c70 0x55eecfa71160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 --2- 172.21.15.46:0/3167588208 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55eecfa55350 0x55eecfa72dc0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 >> 172.21.15.46:0/3167588208 conn(0x55eecfa4fa80 msgr2=0x55eecfa4ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7febf6a00640 0 client.27620 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 shutdown_connections 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 1 -- 172.21.15.46:0/3167588208 wait complete. 2024-03-08T23:17:34.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.410+0000 7fec1e5cd740 20 client.27620 trim_cache size 0 max 16384 2024-03-08T23:17:34.422 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:34.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 msgr2=0x7febfc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 0x7febfc0a4eb0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x55eecfa54e20 tx=0x7fec08005f90 comp rx=0 tx=0).stop 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 msgr2=0x7fec1419ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fec0400e9c0 tx=0x7fec0400ee80 comp rx=0 tx=0).stop 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 shutdown_connections 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7febfc0a29f0 0x7febfc0a4eb0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fec14109570 0x7fec14199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fec14103e70 0x7fec1419ffa0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 --2- 172.21.15.46:0/4236555704 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fec141034b0 0x7fec1419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 >> 172.21.15.46:0/4236555704 conn(0x7fec140fef40 msgr2=0x7fec14100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 shutdown_connections 2024-03-08T23:17:34.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.411+0000 7fec1b800640 1 -- 172.21.15.46:0/4236555704 wait complete. 2024-03-08T23:17:34.430 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:17:34.431 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:17:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 -- 172.21.15.46:0/3635267817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b410d7d0 msgr2=0x7f45b410dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3635267817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b410d7d0 0x7f45b410dbb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009900 tx=0x7f45a402bcc0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 -- 172.21.15.46:0/3635267817 shutdown_connections 2024-03-08T23:17:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3635267817 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b41081d0 0x7f45b4114620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.638 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3635267817 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b4107830 0x7f45b4107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3635267817 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b410d7d0 0x7f45b410dbb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 -- 172.21.15.46:0/3635267817 >> 172.21.15.46:0/3635267817 conn(0x7f45b4077110 msgr2=0x7f45b4106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 -- 172.21.15.46:0/3635267817 shutdown_connections 2024-03-08T23:17:34.639 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.635+0000 7f45b3e00640 1 -- 172.21.15.46:0/3635267817 wait complete. 2024-03-08T23:17:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.636+0000 7f45b3e00640 1 Processor -- start 2024-03-08T23:17:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 -- start start 2024-03-08T23:17:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 0x7f45b419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.641 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 0x7f45b419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 0x7f45b41a2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4117b90 con 0x7f45b4107830 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4117a10 con 0x7f45b41081d0 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b4117d10 con 0x7f45b410d7d0 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 0x7f45b419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 0x7f45b419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b3400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 0x7f45b41a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 0x7f45b419e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44266/0 (socket says 172.21.15.46:44266) 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.637+0000 7f45b2a00640 1 -- 172.21.15.46:0/3982120236 learned_addr learned my addr 172.21.15.46:0/3982120236 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 -- 172.21.15.46:0/3982120236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 msgr2=0x7f45b419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 0x7f45b419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 -- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 msgr2=0x7f45b419e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 0x7f45b419e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 -- 172.21.15.46:0/3982120236 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b41a30c0 con 0x7f45b410d7d0 2024-03-08T23:17:34.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b2000640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 0x7f45b419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45b3400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 0x7f45b41a2990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005fd0 tx=0x7f45ac00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.638+0000 7f45ab400640 1 -- 172.21.15.46:0/3982120236 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac00cd80 con 0x7f45b410d7d0 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.639+0000 7f45ab400640 1 -- 172.21.15.46:0/3982120236 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45ac005020 con 0x7f45b410d7d0 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.639+0000 7f45ab400640 1 -- 172.21.15.46:0/3982120236 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ac010720 con 0x7f45b410d7d0 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.639+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45b41a33b0 con 0x7f45b410d7d0 2024-03-08T23:17:34.644 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.639+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45b41aac40 con 0x7f45b410d7d0 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.642+0000 7f45bc76b740 1 Processor -- start 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.642+0000 7f45ab400640 1 -- 172.21.15.46:0/3982120236 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f45ac0051d0 con 0x7f45b410d7d0 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.642+0000 7f45bc76b740 1 -- start start 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.642+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 0x5557471ef910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471f84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471fae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45bc76b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555747153dd0 con 0x5557471efe50 2024-03-08T23:17:34.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45bc76b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557470fcc70 con 0x5557471f8a30 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45bc76b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55574714fe70 con 0x5557471ef530 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471f84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 0x5557471ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 0x5557471ef910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59688/0 (socket says 172.21.15.46:59688) 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2000640 1 -- 172.21.15.46:0/833169532 learned_addr learned my addr 172.21.15.46:0/833169532 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2000640 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471fae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45ab400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 0x7f45980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 -- 172.21.15.46:0/833169532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 msgr2=0x5557471ef910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 0x5557471ef910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 -- 172.21.15.46:0/833169532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 msgr2=0x5557471fae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471fae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.643+0000 7f45b2a00640 1 -- 172.21.15.46:0/833169532 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557471b62b0 con 0x5557471efe50 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f45ab400640 1 -- 172.21.15.46:0/3982120236 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f45ac0cf7a0 con 0x7f45b410d7d0 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f45b3400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 0x7f45980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f45b2a00640 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471f84f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009230 tx=0x7f45a40095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fcdef38b2f1c42a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f459fe00640 1 -- 172.21.15.46:0/833169532 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4045070 con 0x5557471efe50 2024-03-08T23:17:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f459fe00640 1 -- 172.21.15.46:0/833169532 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45a4004040 con 0x5557471efe50 2024-03-08T23:17:34.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.644+0000 7f459fe00640 1 -- 172.21.15.46:0/833169532 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4004b50 con 0x5557471efe50 2024-03-08T23:17:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 -- 172.21.15.46:0/833169532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 msgr2=0x5557471f84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471f84f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009230 tx=0x7f45a40095e0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45b3400640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 0x7f45980a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0058f0 tx=0x7f45ac005840 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 -- 172.21.15.46:0/833169532 shutdown_connections 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471fae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471f84f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 --2- 172.21.15.46:0/833169532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471ef530 0x5557471ef910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 -- 172.21.15.46:0/833169532 >> 172.21.15.46:0/833169532 conn(0x5557471e9c60 msgr2=0x5557471ea040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 -- 172.21.15.46:0/833169532 shutdown_connections 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.645+0000 7f45bc76b740 1 -- 172.21.15.46:0/833169532 wait complete. 2024-03-08T23:17:34.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 -- start start 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 Processor -- start 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557471fb7c0 con 0x5557471f8a30 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557471100c0 con 0x5557471ef530 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.646+0000 7f45bc76b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5557471fb940 con 0x5557471efe50 2024-03-08T23:17:34.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45bc76b740 10 client.0 _ll_register_callbacks cb 0x55574714aec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44288/0 (socket says 172.21.15.46:44288) 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 -- 172.21.15.46:0/2392064490 learned_addr learned my addr 172.21.15.46:0/2392064490 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2a00640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 -- 172.21.15.46:0/2392064490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 msgr2=0x5557471f5780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f5780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 -- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 msgr2=0x5557471f83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471f83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.647+0000 7f45b2000640 1 -- 172.21.15.46:0/2392064490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555746e683f0 con 0x5557471f8a30 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f45b2000640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f45a0012f90 tx=0x7f45a000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7f3e37ed9ec3788 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f459cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a0004320 con 0x5557471f8a30 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f45a0005020 con 0x5557471f8a30 2024-03-08T23:17:34.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45a4004040 con 0x5557471f8a30 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45a00044c0 con 0x5557471f8a30 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f45bc76b740 10 client.17466 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555746f327d0 con 0x5557471f8a30 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.648+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45a4004040 con 0x5557471f8a30 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45a00035f0 con 0x5557471f8a30 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 10 client.17466 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 20 client.17466 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555747208ae0 0x55574720af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 4 client.17466 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5557471284f0 con 0x555747208ae0 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.649+0000 7f45b2a00640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555747208ae0 0x55574720af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.650+0000 7f45b2a00640 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555747208ae0 0x55574720af10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a66a779815fc11d4 server_cookie=fcfa2a1523dd0058 in_seq=0 out_seq=0 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.650+0000 7f459cc00640 10 client.17466 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f459cc00640 1 -- 172.21.15.46:0/2392064490 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5557471284f0 con 0x555747208ae0 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f459cc00640 10 client.17466 handle_command_reply: tid=0 2024-03-08T23:17:34.660 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 client.17466 shutdown 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 msgr2=0x5557471f5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f45a0012f90 tx=0x7f45a000b390 comp rx=0 tx=0).stop 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 shutdown_connections 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555747208ae0 0x55574720af10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471f8a30 0x5557471f5d90 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f5780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 --2- 172.21.15.46:0/2392064490 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471f83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.661 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f459cc00640 0 client.17466 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 >> 172.21.15.46:0/2392064490 conn(0x5557471e9c60 msgr2=0x5557471eb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 shutdown_connections 2024-03-08T23:17:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 1 -- 172.21.15.46:0/2392064490 wait complete. 2024-03-08T23:17:34.662 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.651+0000 7f45bc76b740 20 client.17466 trim_cache size 0 max 16384 2024-03-08T23:17:34.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.664+0000 7f45bc76b740 1 Processor -- start 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.664+0000 7f45bc76b740 1 -- start start 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471ef530 0x5557471f7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471ffb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b6e80 con 0x5557471ef530 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b4a20 con 0x5557471f8a30 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45bc76b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b2ae0 con 0x5557471efe50 2024-03-08T23:17:34.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45b2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471ef530 0x5557471f7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59702/0 (socket says 172.21.15.46:59702) 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45b2a00640 1 -- 172.21.15.46:0/3458487998 learned_addr learned my addr 172.21.15.46:0/3458487998 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.665+0000 7f45b2000640 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471ffb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 -- 172.21.15.46:0/3458487998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 msgr2=0x5557471ffb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471ffb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 -- 172.21.15.46:0/3458487998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471ef530 msgr2=0x5557471f7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471ef530 0x5557471f7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 -- 172.21.15.46:0/3458487998 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5557471fb150 con 0x5557471efe50 2024-03-08T23:17:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f45b2a00640 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45a4036040 tx=0x7f45a4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f207d783f996053 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f459f400640 1 -- 172.21.15.46:0/3458487998 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4055070 con 0x5557471efe50 2024-03-08T23:17:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f459f400640 1 -- 172.21.15.46:0/3458487998 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45a40046c0 con 0x5557471efe50 2024-03-08T23:17:34.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.666+0000 7f459f400640 1 -- 172.21.15.46:0/3458487998 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4045040 con 0x5557471efe50 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- 172.21.15.46:0/3458487998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 msgr2=0x5557471f83f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f45a4036040 tx=0x7f45a4009260 comp rx=0 tx=0).stop 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- 172.21.15.46:0/3458487998 shutdown_connections 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471f8a30 0x5557471ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471efe50 0x5557471f83f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 --2- 172.21.15.46:0/3458487998 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471ef530 0x5557471f7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- 172.21.15.46:0/3458487998 >> 172.21.15.46:0/3458487998 conn(0x5557471e9c60 msgr2=0x5557471e8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- 172.21.15.46:0/3458487998 shutdown_connections 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- 172.21.15.46:0/3458487998 wait complete. 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.667+0000 7f45bc76b740 1 -- start start 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 Processor -- start 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471f8a30 0x5557471fe910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b30e0 con 0x5557471efe50 2024-03-08T23:17:34.675 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b43c0 con 0x5557471ef530 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45b41b3700 con 0x5557471f8a30 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45bc76b740 10 client.0 _ll_register_callbacks cb 0x7f45a4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.668+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44306/0 (socket says 172.21.15.46:44306) 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 -- 172.21.15.46:0/1604617352 learned_addr learned my addr 172.21.15.46:0/1604617352 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2000640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471f8a30 0x5557471fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 -- 172.21.15.46:0/1604617352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471f8a30 msgr2=0x5557471fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.676 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471f8a30 0x5557471fe910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 -- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 msgr2=0x5557471fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471fff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 -- 172.21.15.46:0/1604617352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5557471ff190 con 0x5557471efe50 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2000640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f45b2a00640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009560 tx=0x7f45a4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1448838ef8488acb server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f459cc00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.669+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45a4055070 con 0x5557471efe50 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45a40046c0 con 0x5557471efe50 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f45a4004870 con 0x5557471efe50 2024-03-08T23:17:34.677 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45a4062050 con 0x5557471efe50 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 10 client.17472 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5557472600d0 con 0x5557471efe50 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45a40046c0 con 0x5557471efe50 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45a404a450 con 0x5557471efe50 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 10 client.17472 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 20 client.17472 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:34.678 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5557472603b0 0x5557472627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 4 client.17472 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:17:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.670+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555747161310 con 0x5557472603b0 2024-03-08T23:17:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.671+0000 7f45b2000640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5557472603b0 0x5557472627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.671+0000 7f45b2000640 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5557472603b0 0x5557472627a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=398ff487cdbf05a8 server_cookie=7ccb77ba3116d3aa in_seq=0 out_seq=0 2024-03-08T23:17:34.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.671+0000 7f459cc00640 10 client.17472 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.671+0000 7f459cc00640 1 -- 172.21.15.46:0/1604617352 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555747161310 con 0x5557472603b0 2024-03-08T23:17:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.671+0000 7f459cc00640 10 client.17472 handle_command_reply: tid=0 2024-03-08T23:17:34.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 client.17472 shutdown 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 msgr2=0x5557471fe350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f45a4009560 tx=0x7f45a4000bc0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 shutdown_connections 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x5557472603b0 0x5557472627a0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5557471f8a30 0x5557471fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5557471efe50 0x5557471fe350 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 --2- 172.21.15.46:0/1604617352 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5557471ef530 0x5557471fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f459cc00640 0 client.17472 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 >> 172.21.15.46:0/1604617352 conn(0x5557471e9c60 msgr2=0x5557471e8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 shutdown_connections 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 1 -- 172.21.15.46:0/1604617352 wait complete. 2024-03-08T23:17:34.682 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.672+0000 7f45bc76b740 20 client.17472 trim_cache size 0 max 16384 2024-03-08T23:17:34.683 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:17:34.683 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 msgr2=0x7f45980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 0x7f45980a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f45ac0058f0 tx=0x7f45ac005840 comp rx=0 tx=0).stop 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 msgr2=0x7f45b41a2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 0x7f45b41a2990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f45ac005fd0 tx=0x7f45ac00ee60 comp rx=0 tx=0).stop 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 shutdown_connections 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f45980a2970 0x7f45980a4e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f45b410d7d0 0x7f45b41a2990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f45b41081d0 0x7f45b419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.684 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 --2- 172.21.15.46:0/3982120236 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f45b4107830 0x7f45b419e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 >> 172.21.15.46:0/3982120236 conn(0x7f45b4077110 msgr2=0x7f45b40781a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 shutdown_connections 2024-03-08T23:17:34.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:34.673+0000 7f45b3e00640 1 -- 172.21.15.46:0/3982120236 wait complete. 2024-03-08T23:17:34.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:34 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2210336262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:35.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:34 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2210336262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:35.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:34 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2210336262' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:17:35.692 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-08T23:17:35.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:17:35.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-08T23:17:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.922+0000 7f120b200640 1 -- 172.21.15.46:0/320900102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041065c0 msgr2=0x7f120410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.922+0000 7f120b200640 1 --2- 172.21.15.46:0/320900102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041065c0 0x7f120410e190 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009ca0 tx=0x7f11fc02c100 comp rx=0 tx=0).stop 2024-03-08T23:17:35.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 -- 172.21.15.46:0/320900102 shutdown_connections 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 --2- 172.21.15.46:0/320900102 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041065c0 0x7f120410e190 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 --2- 172.21.15.46:0/320900102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1204101ea0 0x7f1204105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 --2- 172.21.15.46:0/320900102 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041014f0 0x7f12041018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 -- 172.21.15.46:0/320900102 >> 172.21.15.46:0/320900102 conn(0x7f12040786f0 msgr2=0x7f1204078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 -- 172.21.15.46:0/320900102 shutdown_connections 2024-03-08T23:17:35.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.923+0000 7f120b200640 1 -- 172.21.15.46:0/320900102 wait complete. 2024-03-08T23:17:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 Processor -- start 2024-03-08T23:17:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 -- start start 2024-03-08T23:17:35.928 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12041065c0 0x7f120419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 0x7f1204197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1204111680 con 0x7f12041979e0 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1204111500 con 0x7f12041014f0 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.924+0000 7f120b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1204111800 con 0x7f12041065c0 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:36998/0 (socket says 172.21.15.46:36998) 2024-03-08T23:17:35.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 -- 172.21.15.46:0/2975670415 learned_addr learned my addr 172.21.15.46:0/2975670415 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1209200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 0x7f1204197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203400640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12041065c0 0x7f120419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 -- 172.21.15.46:0/2975670415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12041065c0 msgr2=0x7f120419dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12041065c0 0x7f120419dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 -- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 msgr2=0x7f1204197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 0x7f1204197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1203e00640 1 -- 172.21.15.46:0/2975670415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1204198500 con 0x7f12041014f0 2024-03-08T23:17:35.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.925+0000 7f1209200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 0x7f1204197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f1203e00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9f0 tx=0x7f11f400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f1200c00640 1 -- 172.21.15.46:0/2975670415 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f400cd80 con 0x7f12041014f0 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12041987f0 con 0x7f12041014f0 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f1200c00640 1 -- 172.21.15.46:0/2975670415 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11f4010910 con 0x7f12041014f0 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f1200c00640 1 -- 172.21.15.46:0/2975670415 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f40188f0 con 0x7f12041014f0 2024-03-08T23:17:35.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.926+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12041a4a40 con 0x7f12041014f0 2024-03-08T23:17:35.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.929+0000 7f120dfce740 1 Processor -- start 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.929+0000 7f1200c00640 1 -- 172.21.15.46:0/2975670415 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f11f4010ac0 con 0x7f12041014f0 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.929+0000 7f120dfce740 1 -- start start 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d849b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d872e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578cdba50 con 0x555578d84ef0 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578c84b00 con 0x555578d7b9f0 2024-03-08T23:17:35.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f120dfce740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578cd7af0 con 0x555578d7c310 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1209200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1203400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d872e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1209200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37006/0 (socket says 172.21.15.46:37006) 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1209200640 1 -- 172.21.15.46:0/832732303 learned_addr learned my addr 172.21.15.46:0/832732303 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1203e00640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d849b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1209200640 1 -- 172.21.15.46:0/832732303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 msgr2=0x555578d849b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.930+0000 7f1209200640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d849b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1209200640 1 -- 172.21.15.46:0/832732303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 msgr2=0x555578d872e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1209200640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d872e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1209200640 1 -- 172.21.15.46:0/832732303 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555578ca21e0 con 0x555578d7b9f0 2024-03-08T23:17:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1203400640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d872e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:35.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1203e00640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d849b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:35.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1209200640 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f11fc02c710 tx=0x7f11fc005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdc3b3db7f9f2662 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1200c00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 0x7f11e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.939 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1200c00640 1 -- 172.21.15.46:0/2975670415 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f11f40cf5e0 con 0x7f12041014f0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f11f9600640 1 -- 172.21.15.46:0/832732303 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc043070 con 0x555578d7b9f0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f1203e00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 0x7f11e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f11f9600640 1 -- 172.21.15.46:0/832732303 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11fc032c90 con 0x555578d7b9f0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.931+0000 7f11f9600640 1 -- 172.21.15.46:0/832732303 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc039040 con 0x555578d7b9f0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 -- 172.21.15.46:0/832732303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 msgr2=0x555578d7bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f11fc02c710 tx=0x7f11fc005f80 comp rx=0 tx=0).stop 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f1203e00640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 0x7f11e80a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11f40cca20 tx=0x7f11f401f000 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.940 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 -- 172.21.15.46:0/832732303 shutdown_connections 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d872e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d849b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 --2- 172.21.15.46:0/832732303 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d7bdd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.932+0000 7f120dfce740 1 -- 172.21.15.46:0/832732303 >> 172.21.15.46:0/832732303 conn(0x555578d76120 msgr2=0x555578d76500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.933+0000 7f120dfce740 1 -- 172.21.15.46:0/832732303 shutdown_connections 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.933+0000 7f120dfce740 1 -- 172.21.15.46:0/832732303 wait complete. 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.933+0000 7f120dfce740 1 -- start start 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.933+0000 7f120dfce740 1 Processor -- start 2024-03-08T23:17:35.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7b9f0 0x555578d81ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 0x555578d82920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578d87c80 con 0x555578d7c310 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578c97f60 con 0x555578d84ef0 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x555578d87e00 con 0x555578d7b9f0 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f120dfce740 10 client.0 _ll_register_callbacks cb 0x555578c83b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44338/0 (socket says 172.21.15.46:44338) 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 -- 172.21.15.46:0/1142574096 learned_addr learned my addr 172.21.15.46:0/1142574096 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1209200640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 0x555578d82920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1209200640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7b9f0 0x555578d81ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 -- 172.21.15.46:0/1142574096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7b9f0 msgr2=0x555578d81ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7b9f0 0x555578d81ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 -- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 msgr2=0x555578d82920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 0x555578d82920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1203400640 1 -- 172.21.15.46:0/1142574096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555578ab87e0 con 0x555578d7c310 2024-03-08T23:17:35.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.934+0000 7f1209200640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 0x555578d82920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f1203400640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f11f0012730 tx=0x7f11f0012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9f4063bffbe4375 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f11ee000640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f000f920 con 0x555578d7c310 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f11f000fac0 con 0x555578d7c310 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11fc032c90 con 0x555578d7c310 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f0025050 con 0x555578d7c310 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f120dfce740 10 client.17484 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.935+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555578b18f70 con 0x555578d7c310 2024-03-08T23:17:35.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11fc032c90 con 0x555578d7c310 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11f0010430 con 0x555578d7c310 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 10 client.17484 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 10 client.17484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 20 client.17484 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578d95270 0x555578d97660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 4 client.17484 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555578ce8f90 con 0x555578d95270 2024-03-08T23:17:35.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.936+0000 7f1209200640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578d95270 0x555578d97660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.937+0000 7f1209200640 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578d95270 0x555578d97660 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d54911b20b31221 server_cookie=5352286b206fcbfe in_seq=0 out_seq=0 2024-03-08T23:17:35.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.937+0000 7f11ee000640 10 client.17484 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f11ee000640 1 -- 172.21.15.46:0/1142574096 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555578ce8f90 con 0x555578d95270 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f11ee000640 10 client.17484 handle_command_reply: tid=0 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 client.17484 shutdown 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 msgr2=0x555578d823e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f11f0012730 tx=0x7f11f0012bf0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 shutdown_connections 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578d95270 0x555578d97660 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d84ef0 0x555578d82920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d7c310 0x555578d823e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 --2- 172.21.15.46:0/1142574096 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7b9f0 0x555578d81ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 >> 172.21.15.46:0/1142574096 conn(0x555578d76120 msgr2=0x555578d77e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f11ee000640 0 client.17484 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 shutdown_connections 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 1 -- 172.21.15.46:0/1142574096 wait complete. 2024-03-08T23:17:35.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.938+0000 7f120dfce740 20 client.17484 trim_cache size 0 max 16384 2024-03-08T23:17:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.955+0000 7f120dfce740 1 Processor -- start 2024-03-08T23:17:35.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.955+0000 7f120dfce740 1 -- start start 2024-03-08T23:17:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d70c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d8c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041ae900 con 0x555578d84ef0 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041ac4a0 con 0x555578d7b9f0 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f120dfce740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041aa560 con 0x555578d7c310 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37024/0 (socket says 172.21.15.46:37024) 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 -- 172.21.15.46:0/3067826810 learned_addr learned my addr 172.21.15.46:0/3067826810 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:35.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d8c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1203400640 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d70c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 -- 172.21.15.46:0/3067826810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 msgr2=0x555578d70c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d70c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 -- 172.21.15.46:0/3067826810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 msgr2=0x555578d8c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d8c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.956+0000 7f1209200640 1 -- 172.21.15.46:0/3067826810 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555578d979d0 con 0x555578d7b9f0 2024-03-08T23:17:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.957+0000 7f1209200640 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009870 tx=0x7f11fc005f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86a81c98f33ce4e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.957+0000 7f11f8c00640 1 -- 172.21.15.46:0/3067826810 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc039040 con 0x555578d7b9f0 2024-03-08T23:17:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.957+0000 7f11f8c00640 1 -- 172.21.15.46:0/3067826810 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11fc032c90 con 0x555578d7b9f0 2024-03-08T23:17:35.963 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.957+0000 7f11f8c00640 1 -- 172.21.15.46:0/3067826810 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11fc043040 con 0x555578d7b9f0 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- 172.21.15.46:0/3067826810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 msgr2=0x555578d845d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f11fc009870 tx=0x7f11fc005f80 comp rx=0 tx=0).stop 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- 172.21.15.46:0/3067826810 shutdown_connections 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578d8c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578d70c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 --2- 172.21.15.46:0/3067826810 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578d845d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- 172.21.15.46:0/3067826810 >> 172.21.15.46:0/3067826810 conn(0x555578d76120 msgr2=0x555578d70580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- 172.21.15.46:0/3067826810 shutdown_connections 2024-03-08T23:17:35.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- 172.21.15.46:0/3067826810 wait complete. 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 -- start start 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.958+0000 7f120dfce740 1 Processor -- start 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578de16e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578de2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041aab60 con 0x555578d84ef0 2024-03-08T23:17:35.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041abe40 con 0x555578d7b9f0 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12041ab180 con 0x555578d7c310 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f120dfce740 10 client.0 _ll_register_callbacks cb 0x7f11f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1209200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578de16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1209200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578de2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1203400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1203400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59744/0 (socket says 172.21.15.46:59744) 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1203400640 1 -- 172.21.15.46:0/2022489400 learned_addr learned my addr 172.21.15.46:0/2022489400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.959+0000 7f1209200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578de2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:44352/0 (socket says 172.21.15.46:44352) 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 -- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 msgr2=0x555578de16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578de16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 -- 172.21.15.46:0/2022489400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 msgr2=0x555578de2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578de2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 -- 172.21.15.46:0/2022489400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5555789b0170 con 0x555578d7c310 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f1203400640 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005e00 tx=0x7f11f0010660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117b93474a167cd server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:35.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f11ee000640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f002d020 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f11f0018040 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.960+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11fc032c90 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11f0038050 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 10 client.27601 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555578ca7050 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11fc032c90 con 0x555578d7c310 2024-03-08T23:17:35.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11f0005340 con 0x555578d7c310 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 10 client.27601 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 10 client.27601 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 20 client.27601 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578dedf50 0x555578df0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 4 client.27601 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-08T23:17:35.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.961+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555578ce32e0 con 0x555578dedf50 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.962+0000 7f1209200640 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578dedf50 0x555578df0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.962+0000 7f1209200640 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578dedf50 0x555578df0370 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3246808c9c29c4fb server_cookie=e7df74b4df53d9a7 in_seq=0 out_seq=0 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.962+0000 7f11ee000640 10 client.27601 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.963+0000 7f11ee000640 1 -- 172.21.15.46:0/2022489400 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555578ce32e0 con 0x555578dedf50 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.963+0000 7f11ee000640 10 client.27601 handle_command_reply: tid=0 2024-03-08T23:17:35.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.963+0000 7f120dfce740 1 client.27601 shutdown 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.963+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 msgr2=0x555578de1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.963+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f11f0005e00 tx=0x7f11f0010660 comp rx=0 tx=0).stop 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 shutdown_connections 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x555578dedf50 0x555578df0370 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x555578d84ef0 0x555578de2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x555578d7c310 0x555578de1c20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 --2- 172.21.15.46:0/2022489400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x555578d7b9f0 0x555578de16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f11ee000640 0 client.27601 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:35.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 >> 172.21.15.46:0/2022489400 conn(0x555578d76120 msgr2=0x555578d70e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 shutdown_connections 2024-03-08T23:17:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 1 -- 172.21.15.46:0/2022489400 wait complete. 2024-03-08T23:17:35.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.964+0000 7f120dfce740 20 client.27601 trim_cache size 0 max 16384 2024-03-08T23:17:35.973 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:35.973 INFO:teuthology.orchestra.run.smithi046.stdout: "return_code": 0, 2024-03-08T23:17:35.974 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_tag": "4b1fb251-6074-4163-bb4b-696bd66c7513", 2024-03-08T23:17:35.974 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": "asynchronous" 2024-03-08T23:17:35.974 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:35.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 msgr2=0x7f11e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 0x7f11e80a4e30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f11f40cca20 tx=0x7f11f401f000 comp rx=0 tx=0).stop 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 msgr2=0x7f120419d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f11f400e9f0 tx=0x7f11f400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 shutdown_connections 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f11e80a2970 0x7f11e80a4e30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f12041979e0 0x7f1204197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f12041065c0 0x7f120419dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.975 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 --2- 172.21.15.46:0/2975670415 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f12041014f0 0x7f120419d7e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:35.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 >> 172.21.15.46:0/2975670415 conn(0x7f12040786f0 msgr2=0x7f12040ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:35.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 shutdown_connections 2024-03-08T23:17:35.976 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:35.965+0000 7f120b200640 1 -- 172.21.15.46:0/2975670415 wait complete. 2024-03-08T23:17:35.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:17:35.991 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:17:36.267 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.265+0000 7f0131600640 1 -- 172.21.15.46:0/269363857 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c032a70 con 0x7f013410db00 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.265+0000 7f0139e00640 1 -- 172.21.15.46:0/269363857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013410db00 msgr2=0x7f013410ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.265+0000 7f0139e00640 1 --2- 172.21.15.46:0/269363857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013410db00 0x7f013410ff10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f012c009ca0 tx=0x7f012c02c190 comp rx=0 tx=0).stop 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 -- 172.21.15.46:0/269363857 shutdown_connections 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 --2- 172.21.15.46:0/269363857 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f013410db00 0x7f013410ff10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 --2- 172.21.15.46:0/269363857 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0134073fe0 0x7f013410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 --2- 172.21.15.46:0/269363857 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f01340736c0 0x7f0134073aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.268 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 -- 172.21.15.46:0/269363857 >> 172.21.15.46:0/269363857 conn(0x7f01340fcd50 msgr2=0x7f01340ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 -- 172.21.15.46:0/269363857 shutdown_connections 2024-03-08T23:17:36.269 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.266+0000 7f0139e00640 1 -- 172.21.15.46:0/269363857 wait complete. 2024-03-08T23:17:36.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.267+0000 7f0139e00640 1 Processor -- start 2024-03-08T23:17:36.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.268+0000 7f0139e00640 1 -- start start 2024-03-08T23:17:36.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.268+0000 7f0139e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01340736c0 0x7f0134199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.268+0000 7f0139e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 0x7f013419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0139e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 0x7f013419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0139e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0134113870 con 0x7f01340736c0 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0139e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341136f0 con 0x7f013410db00 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0139e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341139f0 con 0x7f0134073fe0 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0132000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 0x7f013419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0132a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01340736c0 0x7f0134199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0132000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 0x7f013419a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59764/0 (socket says 172.21.15.46:59764) 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0132000640 1 -- 172.21.15.46:0/1335953924 learned_addr learned my addr 172.21.15.46:0/1335953924 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:36.272 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.269+0000 7f0133400640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 0x7f013419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.273 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 -- 172.21.15.46:0/1335953924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 msgr2=0x7f013419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 0x7f013419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 -- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01340736c0 msgr2=0x7f0134199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01340736c0 0x7f0134199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 -- 172.21.15.46:0/1335953924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013419ed70 con 0x7f013410db00 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.270+0000 7f0133400640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 0x7f013419e640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f012c02c7a0 tx=0x7f012c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.271+0000 7f0138a00640 1 -- 172.21.15.46:0/1335953924 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c039040 con 0x7f013410db00 2024-03-08T23:17:36.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.271+0000 7f0138a00640 1 -- 172.21.15.46:0/1335953924 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f012c044070 con 0x7f013410db00 2024-03-08T23:17:36.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.271+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013419f000 con 0x7f013410db00 2024-03-08T23:17:36.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.271+0000 7f0138a00640 1 -- 172.21.15.46:0/1335953924 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012c0328d0 con 0x7f013410db00 2024-03-08T23:17:36.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.271+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01341a68f0 con 0x7f013410db00 2024-03-08T23:17:36.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.276+0000 7f0138a00640 1 -- 172.21.15.46:0/1335953924 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f012c032a70 con 0x7f013410db00 2024-03-08T23:17:36.278 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.276+0000 7f013cb66740 1 Processor -- start 2024-03-08T23:17:36.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 -- start start 2024-03-08T23:17:36.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff451d90 0x55adff452170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.279 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff45d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff40b410 con 0x55adff451d90 2024-03-08T23:17:36.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff3add50 con 0x55adff45b290 2024-03-08T23:17:36.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f013cb66740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff356b80 con 0x55adff4526b0 2024-03-08T23:17:36.280 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f0132a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f0132a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59774/0 (socket says 172.21.15.46:59774) 2024-03-08T23:17:36.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f0132a00640 1 -- 172.21.15.46:0/2935571443 learned_addr learned my addr 172.21.15.46:0/2935571443 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:36.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.277+0000 7f0132000640 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff451d90 0x55adff452170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.281 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.278+0000 7f0132a00640 1 -- 172.21.15.46:0/2935571443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 msgr2=0x55adff45d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.278+0000 7f0132a00640 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff45d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.278+0000 7f0132a00640 1 -- 172.21.15.46:0/2935571443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff451d90 msgr2=0x55adff452170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.278+0000 7f0132a00640 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff451d90 0x55adff452170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.278+0000 7f0132a00640 1 -- 172.21.15.46:0/2935571443 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adff18c7d0 con 0x55adff4526b0 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.279+0000 7f0132a00640 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0124009cf0 tx=0x7f0124011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85dcf1cac19ad8e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.279+0000 7f0138a00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 0x7f01180a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.279+0000 7f011fe00640 1 -- 172.21.15.46:0/2935571443 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124004230 con 0x55adff4526b0 2024-03-08T23:17:36.282 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.279+0000 7f011fe00640 1 -- 172.21.15.46:0/2935571443 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01240043d0 con 0x55adff4526b0 2024-03-08T23:17:36.283 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.279+0000 7f011fe00640 1 -- 172.21.15.46:0/2935571443 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124019960 con 0x55adff4526b0 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.280+0000 7f0138a00640 1 -- 172.21.15.46:0/1335953924 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f012c0f14c0 con 0x7f013410db00 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.280+0000 7f0132000640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 0x7f01180a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.280+0000 7f013cb66740 1 -- 172.21.15.46:0/2935571443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 msgr2=0x55adff45ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.280+0000 7f013cb66740 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0124009cf0 tx=0x7f0124011810 comp rx=0 tx=0).stop 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 -- 172.21.15.46:0/2935571443 shutdown_connections 2024-03-08T23:17:36.284 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff45d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff4526b0 0x55adff45ad50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 --2- 172.21.15.46:0/2935571443 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff451d90 0x55adff452170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 -- 172.21.15.46:0/2935571443 >> 172.21.15.46:0/2935571443 conn(0x55adff44c4c0 msgr2=0x55adff44c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f0132000640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 0x7f01180a4eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55adff44d650 tx=0x7f01200040f0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 -- 172.21.15.46:0/2935571443 shutdown_connections 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 -- 172.21.15.46:0/2935571443 wait complete. 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.281+0000 7f013cb66740 1 -- start start 2024-03-08T23:17:36.285 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 Processor -- start 2024-03-08T23:17:36.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff458d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff369ee0 con 0x55adff4526b0 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff3a9df0 con 0x55adff45b290 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55adff45e080 con 0x55adff451d90 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.282+0000 7f013cb66740 10 client.0 _ll_register_callbacks cb 0x55adff3708d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:59776/0 (socket says 172.21.15.46:59776) 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 -- 172.21.15.46:0/2642948491 learned_addr learned my addr 172.21.15.46:0/2642948491 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:36.288 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0133400640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff458d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132000640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 -- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 msgr2=0x55adff458d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff458d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 -- 172.21.15.46:0/2642948491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 msgr2=0x55adff4587c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4587c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.283+0000 7f0132a00640 1 -- 172.21.15.46:0/2642948491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55adff0c23f0 con 0x55adff451d90 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f0132000640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:17:36.289 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f0132a00640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f01240195c0 tx=0x7f0124003820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caa9d4f77d15966a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f011cc00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124027070 con 0x55adff451d90 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01240043d0 con 0x55adff451d90 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.284+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f0124005240 con 0x55adff451d90 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0124033050 con 0x55adff451d90 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 10 client.27613 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55adff0820f0 con 0x55adff451d90 2024-03-08T23:17:36.290 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01240043d0 con 0x55adff451d90 2024-03-08T23:17:36.291 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0124004ba0 con 0x55adff451d90 2024-03-08T23:17:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 10 client.27613 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 10 client.27613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 20 client.27613 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:36.292 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff46b3e0 0x55adff46d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 4 client.27613 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.285+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55adff3bb290 con 0x55adff46b3e0 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.286+0000 7f0132000640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff46b3e0 0x55adff46d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.287+0000 7f0132000640 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff46b3e0 0x55adff46d7d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c450bfb41173dfe4 server_cookie=b70d026a1201b998 in_seq=0 out_seq=0 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.287+0000 7f011cc00640 10 client.27613 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.287+0000 7f011cc00640 1 -- 172.21.15.46:0/2642948491 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55adff3bb290 con 0x55adff46b3e0 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.287+0000 7f011cc00640 10 client.27613 handle_command_reply: tid=0 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 client.27613 shutdown 2024-03-08T23:17:36.293 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 msgr2=0x55adff458280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f01240195c0 tx=0x7f0124003820 comp rx=0 tx=0).stop 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 shutdown_connections 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff46b3e0 0x55adff46d7d0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff45b290 0x55adff458d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4587c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 --2- 172.21.15.46:0/2642948491 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff458280 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 >> 172.21.15.46:0/2642948491 conn(0x55adff44c4c0 msgr2=0x55adff44d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.294 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.288+0000 7f011cc00640 0 client.27613 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:36.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.289+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 shutdown_connections 2024-03-08T23:17:36.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.289+0000 7f013cb66740 1 -- 172.21.15.46:0/2642948491 wait complete. 2024-03-08T23:17:36.295 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.289+0000 7f013cb66740 20 client.27613 trim_cache size 0 max 16384 2024-03-08T23:17:36.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.309+0000 7f013cb66740 1 Processor -- start 2024-03-08T23:17:36.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.309+0000 7f013cb66740 1 -- start start 2024-03-08T23:17:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4622c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 0x55adff462800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341ae920 con 0x55adff4526b0 2024-03-08T23:17:36.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341ac4c0 con 0x55adff451d90 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f013cb66740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341aa580 con 0x55adff45b290 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f0132a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f0132a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37090/0 (socket says 172.21.15.46:37090) 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f0132a00640 1 -- 172.21.15.46:0/3764221101 learned_addr learned my addr 172.21.15.46:0/3764221101 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f0132000640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.310+0000 7f0133400640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 0x55adff462800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 -- 172.21.15.46:0/3764221101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 msgr2=0x55adff462800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 0x55adff462800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.314 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 -- 172.21.15.46:0/3764221101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 msgr2=0x55adff4622c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4622c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 -- 172.21.15.46:0/3764221101 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55adff43f6d0 con 0x55adff451d90 2024-03-08T23:17:36.315 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0133400640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 0x55adff462800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:36.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.311+0000 7f0132a00640 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0124009ba0 tx=0x7f0124009bd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c586b33b06c60d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.312+0000 7f011f400640 1 -- 172.21.15.46:0/3764221101 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012403c070 con 0x55adff451d90 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.312+0000 7f011f400640 1 -- 172.21.15.46:0/3764221101 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0124003e10 con 0x55adff451d90 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.312+0000 7f011f400640 1 -- 172.21.15.46:0/3764221101 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0124027070 con 0x55adff451d90 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- 172.21.15.46:0/3764221101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 msgr2=0x55adff45a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0124009ba0 tx=0x7f0124009bd0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- 172.21.15.46:0/3764221101 shutdown_connections 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff45b290 0x55adff462800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff4526b0 0x55adff4622c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 --2- 172.21.15.46:0/3764221101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff451d90 0x55adff45a9b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- 172.21.15.46:0/3764221101 >> 172.21.15.46:0/3764221101 conn(0x55adff44c4c0 msgr2=0x55adff44c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- 172.21.15.46:0/3764221101 shutdown_connections 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- 172.21.15.46:0/3764221101 wait complete. 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.313+0000 7f013cb66740 1 -- start start 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 Processor -- start 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff4bb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 0x55adff4b5690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341aab80 con 0x55adff45b290 2024-03-08T23:17:36.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341abe60 con 0x55adff4526b0 2024-03-08T23:17:36.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.314+0000 7f013cb66740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01341ab1a0 con 0x55adff451d90 2024-03-08T23:17:36.319 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f013cb66740 10 client.0 _ll_register_callbacks cb 0x7f01240097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:17:36.321 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0132000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0132000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:37104/0 (socket says 172.21.15.46:37104) 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0132000640 1 -- 172.21.15.46:0/3891297125 learned_addr learned my addr 172.21.15.46:0/3891297125 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0133400640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 0x55adff4b5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0132a00640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff4bb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.315+0000 7f0132000640 1 -- 172.21.15.46:0/3891297125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 msgr2=0x55adff4bb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132000640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff4bb120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132000640 1 -- 172.21.15.46:0/3891297125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 msgr2=0x55adff4b5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:17:36.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132000640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 0x55adff4b5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132000640 1 -- 172.21.15.46:0/3891297125 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55adff4b5d80 con 0x55adff4526b0 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132a00640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff4bb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0133400640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 0x55adff4b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f0132000640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0120003e10 tx=0x7f0120014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4b5f03bf68e1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.316+0000 7f011cc00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120034060 con 0x55adff4526b0 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f012001cb20 con 0x55adff4526b0 2024-03-08T23:17:36.323 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0124003e10 con 0x55adff4526b0 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f012003b050 con 0x55adff4526b0 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f013cb66740 10 client.27671 fetch_fsmap learned FSMap version 31 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.317+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55adff4c2980 con 0x55adff4526b0 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0124003e10 con 0x55adff4526b0 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f012002e030 con 0x55adff4526b0 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 10 client.27671 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 10 client.27671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:17:36.324 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 20 client.27671 populate_metadata read hostname 'smithi046' 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff4c3090 0x55adff4c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 4 client.27671 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.318+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55adff3b55e0 con 0x55adff4c3090 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f0132a00640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff4c3090 0x55adff4c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f0132a00640 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff4c3090 0x55adff4c54b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0e5bde8d29a350f server_cookie=2e9b64d97d87267a in_seq=0 out_seq=0 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f011cc00640 10 client.27671 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f011cc00640 1 -- 172.21.15.46:0/3891297125 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55adff3b55e0 con 0x55adff4c3090 2024-03-08T23:17:36.325 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f011cc00640 10 client.27671 handle_command_reply: tid=0 2024-03-08T23:17:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.319+0000 7f013cb66740 1 client.27671 shutdown 2024-03-08T23:17:36.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 msgr2=0x55adff4bb660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0120003e10 tx=0x7f0120014040 comp rx=0 tx=0).stop 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 shutdown_connections 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x55adff4c3090 0x55adff4c54b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55adff45b290 0x55adff4b5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55adff4526b0 0x55adff4bb660 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 --2- 172.21.15.46:0/3891297125 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55adff451d90 0x55adff4bb120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 >> 172.21.15.46:0/3891297125 conn(0x55adff44c4c0 msgr2=0x55adff453570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f011cc00640 0 client.27671 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 shutdown_connections 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 1 -- 172.21.15.46:0/3891297125 wait complete. 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.320+0000 7f013cb66740 20 client.27671 trim_cache size 0 max 16384 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:17:36.328 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": { 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: "4b1fb251-6074-4163-bb4b-696bd66c7513": { 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: "path": "/", 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: "tag": "4b1fb251-6074-4163-bb4b-696bd66c7513", 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: "options": "recursive,force" 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:17:36.329 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:17:36.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 msgr2=0x7f01180a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 0x7f01180a4eb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x55adff44d650 tx=0x7f01200040f0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.330 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 msgr2=0x7f013419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 0x7f013419e640 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f012c02c7a0 tx=0x7f012c035d10 comp rx=0 tx=0).stop 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 shutdown_connections 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f01180a29f0 0x7f01180a4eb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f013410db00 0x7f013419e640 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0134073fe0 0x7f013419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 --2- 172.21.15.46:0/1335953924 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01340736c0 0x7f0134199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:36.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.321+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 >> 172.21.15.46:0/1335953924 conn(0x7f01340fcd50 msgr2=0x7f01340fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.322+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 shutdown_connections 2024-03-08T23:17:36.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:36.322+0000 7f0139e00640 1 -- 172.21.15.46:0/1335953924 wait complete. 2024-03-08T23:17:36.347 INFO:tasks.cephfs.filesystem:scrub status for tag:4b1fb251-6074-4163-bb4b-696bd66c7513 - {'path': '/', 'tag': '4b1fb251-6074-4163-bb4b-696bd66c7513', 'options': 'recursive,force'} 2024-03-08T23:17:36.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:36 smithi062 ceph-mon[35761]: pgmap v305: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:36.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:36 smithi046 ceph-mon[31859]: pgmap v305: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:36.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:36 smithi192 ceph-mon[36700]: pgmap v305: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub queued for path: / 2024-03-08T23:17:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:37.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: active paths [/] 2024-03-08T23:17:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:37 smithi062 ceph-mon[35761]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub queued for path: / 2024-03-08T23:17:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:37.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: active paths [/] 2024-03-08T23:17:37.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.726 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:37 smithi046 ceph-mon[31859]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub queued for path: / 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: active paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:37.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:37 smithi192 ceph-mon[36700]: scrub summary: idle+waiting paths [/] 2024-03-08T23:17:38.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:38 smithi062 ceph-mon[35761]: pgmap v306: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:38.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:38 smithi046 ceph-mon[31859]: pgmap v306: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:38.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:38 smithi192 ceph-mon[36700]: pgmap v306: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:40.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:40 smithi062 ceph-mon[35761]: pgmap v307: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:40.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:40 smithi046 ceph-mon[31859]: pgmap v307: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:40.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:40 smithi192 ceph-mon[36700]: pgmap v307: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:42.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:42 smithi062 ceph-mon[35761]: pgmap v308: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:42.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:42 smithi046 ceph-mon[31859]: pgmap v308: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:42.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:42 smithi192 ceph-mon[36700]: pgmap v308: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:44.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:44 smithi062 ceph-mon[35761]: pgmap v309: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:44.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:44 smithi046 ceph-mon[31859]: pgmap v309: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:44.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:44 smithi192 ceph-mon[36700]: pgmap v309: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:46.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:46 smithi062 ceph-mon[35761]: pgmap v310: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:46.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:46 smithi046 ceph-mon[31859]: pgmap v310: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:46.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:46 smithi192 ceph-mon[36700]: pgmap v310: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:47.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:47.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:47 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:47.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:47.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:47 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:47.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:47.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:47 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:48.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:48 smithi062 ceph-mon[35761]: pgmap v311: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:48.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:48 smithi062 ceph-mon[35761]: scrub summary: idle 2024-03-08T23:17:48.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:48 smithi046 ceph-mon[31859]: pgmap v311: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:48.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:48 smithi046 ceph-mon[31859]: scrub summary: idle 2024-03-08T23:17:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:48 smithi192 ceph-mon[36700]: pgmap v311: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-08T23:17:48.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:48 smithi192 ceph-mon[36700]: scrub summary: idle 2024-03-08T23:17:49.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:49 smithi046 ceph-mon[31859]: pgmap v312: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:49.815 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:49 smithi192 ceph-mon[36700]: pgmap v312: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:49.872 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-08T23:17:49.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:49 smithi062 ceph-mon[35761]: pgmap v312: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-08T23:17:50.103 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 -- 172.21.15.46:0/2196356566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 msgr2=0x7fcce8110370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 --2- 172.21.15.46:0/2196356566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce8110370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fcce0009be0 tx=0x7fcce002bef0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 -- 172.21.15.46:0/2196356566 shutdown_connections 2024-03-08T23:17:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 --2- 172.21.15.46:0/2196356566 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce8110370 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 --2- 172.21.15.46:0/2196356566 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81034f0 0x7fcce8103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.104 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 --2- 172.21.15.46:0/2196356566 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81095b0 0x7fcce8109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 -- 172.21.15.46:0/2196356566 >> 172.21.15.46:0/2196356566 conn(0x7fcce80fefa0 msgr2=0x7fcce81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 -- 172.21.15.46:0/2196356566 shutdown_connections 2024-03-08T23:17:50.105 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.101+0000 7fccef800640 1 -- 172.21.15.46:0/2196356566 wait complete. 2024-03-08T23:17:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.102+0000 7fccef800640 1 Processor -- start 2024-03-08T23:17:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.102+0000 7fccef800640 1 -- start start 2024-03-08T23:17:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81034f0 0x7fcce819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce81138c0 con 0x7fcce8103eb0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8113740 con 0x7fcce81095b0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccef800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce8113a40 con 0x7fcce81034f0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fcce7e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fcced800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fcced800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:43384/0 (socket says 172.21.15.46:43384) 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fcced800640 1 -- 172.21.15.46:0/2472942404 learned_addr learned my addr 172.21.15.46:0/2472942404 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.103+0000 7fccece00640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81034f0 0x7fcce819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 -- 172.21.15.46:0/2472942404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81034f0 msgr2=0x7fcce819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81034f0 0x7fcce819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 -- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 msgr2=0x7fcce819fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce819fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 -- 172.21.15.46:0/2472942404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce819a180 con 0x7fcce81095b0 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcced800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcce0033420 tx=0x7fcce00336b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce0043020 con 0x7fcce81095b0 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce819a410 con 0x7fcce81095b0 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcce0003e50 con 0x7fcce81095b0 2024-03-08T23:17:50.109 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce003e6b0 con 0x7fcce81095b0 2024-03-08T23:17:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.104+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce81a6710 con 0x7fcce81095b0 2024-03-08T23:17:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.105+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce804f050 con 0x7fcce81095b0 2024-03-08T23:17:50.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.108+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcce0046020 con 0x7fcce81095b0 2024-03-08T23:17:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.109+0000 7fcce5600640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 0x7fccd00a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.109+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fcce00f1a10 con 0x7fcce81095b0 2024-03-08T23:17:50.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.109+0000 7fccece00640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 0x7fccd00a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.110+0000 7fccece00640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 0x7fccd00a4ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcce819ab40 tx=0x7fccdc005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:50.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.111+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcce004d030 con 0x7fcce81095b0 2024-03-08T23:17:50.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.336+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7fcce8100c50 con 0x7fccd00a2a10 2024-03-08T23:17:50.353 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.351+0000 7fcce5600640 1 -- 172.21.15.46:0/2472942404 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7fcce8100c50 con 0x7fccd00a2a10 2024-03-08T23:17:50.354 INFO:teuthology.orchestra.run.smithi046.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-08T00:00:00", "created": "2024-03-08T23:07:47", "first": "2024-03-08T23:08:00", "last": "2024-03-08T23:17:00", "last_pruned": "2024-03-08T23:17:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-08T23:17:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.354+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 msgr2=0x7fccd00a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.354+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 0x7fccd00a4ed0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcce819ab40 tx=0x7fccdc005f90 comp rx=0 tx=0).stop 2024-03-08T23:17:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.354+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 msgr2=0x7fcce8199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.357 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.354+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcce0033420 tx=0x7fcce00336b0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.354+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 shutdown_connections 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fccd00a2a10 0x7fccd00a4ed0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fcce81095b0 0x7fcce8199a00 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fcce8103eb0 0x7fcce819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 --2- 172.21.15.46:0/2472942404 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcce81034f0 0x7fcce819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 >> 172.21.15.46:0/2472942404 conn(0x7fcce80fefa0 msgr2=0x7fcce8100540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 shutdown_connections 2024-03-08T23:17:50.358 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.355+0000 7fccef800640 1 -- 172.21.15.46:0/2472942404 wait complete. 2024-03-08T23:17:50.379 DEBUG:teuthology.orchestra.run.smithi046:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 -- 172.21.15.46:0/879456430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd0101500 msgr2=0x7f7cd01018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/879456430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd0101500 0x7f7cd01018e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0009860 tx=0x7f7cc002bb70 comp rx=0 tx=0).stop 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 -- 172.21.15.46:0/879456430 shutdown_connections 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/879456430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 0x7f7cd010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/879456430 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101eb0 0x7f7cd0105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.589 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/879456430 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd0101500 0x7f7cd01018e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 -- 172.21.15.46:0/879456430 >> 172.21.15.46:0/879456430 conn(0x7f7cd00fcd80 msgr2=0x7f7cd00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 -- 172.21.15.46:0/879456430 shutdown_connections 2024-03-08T23:17:50.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.586+0000 7f7cd6c00640 1 -- 172.21.15.46:0/879456430 wait complete. 2024-03-08T23:17:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.587+0000 7f7cd6c00640 1 Processor -- start 2024-03-08T23:17:50.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.587+0000 7f7cd6c00640 1 -- start start 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 0x7f7cd019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 0x7f7cd019ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd0111660 con 0x7f7cd0101500 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01114e0 con 0x7f7cd019e5e0 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd6c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd01117e0 con 0x7f7cd01065d0 2024-03-08T23:17:50.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7ccfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7ccfe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49712/0 (socket says 172.21.15.46:49712) 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7ccfe00640 1 -- 172.21.15.46:0/2987232149 learned_addr learned my addr 172.21.15.46:0/2987232149 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7cd4c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 0x7f7cd019ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.588+0000 7f7ccf400640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 0x7f7cd019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 -- 172.21.15.46:0/2987232149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 msgr2=0x7f7cd019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 0x7f7cd019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 -- 172.21.15.46:0/2987232149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 msgr2=0x7f7cd019ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 0x7f7cd019ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 -- 172.21.15.46:0/2987232149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd019f0d0 con 0x7f7cd0101500 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7cd4c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 0x7f7cd019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7ccfe00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0037040 tx=0x7f7cc00045b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0041020 con 0x7f7cd0101500 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7cc002c9f0 con 0x7f7cd0101500 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cd019f360 con 0x7f7cd0101500 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.589+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc0039040 con 0x7f7cd0101500 2024-03-08T23:17:50.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.590+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cd01a6b20 con 0x7f7cd0101500 2024-03-08T23:17:50.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.591+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cd004f050 con 0x7f7cd0101500 2024-03-08T23:17:50.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.593+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7cc0044020 con 0x7f7cd0101500 2024-03-08T23:17:50.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.594+0000 7f7cccc00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 0x7f7cb40a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:17:50.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.594+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f7cc00f0cd0 con 0x7f7cd0101500 2024-03-08T23:17:50.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.594+0000 7f7ccf400640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 0x7f7cb40a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:17:50.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.595+0000 7f7ccf400640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 0x7f7cb40a4ed0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0075580 tx=0x7f7cbc009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:17:50.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.597+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7cc004b020 con 0x7f7cd0101500 2024-03-08T23:17:50.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:50 smithi046 ceph-mon[31859]: from='client.27677 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:50.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:50 smithi192 ceph-mon[36700]: from='client.27677 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:50.825 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.822+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 --> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7cd0101f90 con 0x7f7cb40a2a10 2024-03-08T23:17:50.839 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.836+0000 7f7cccc00640 1 -- 172.21.15.46:0/2987232149 <== mgr.25559 v2:172.21.15.46:6800/2505290629 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f7cd0101f90 con 0x7f7cb40a2a10 2024-03-08T23:17:50.839 INFO:teuthology.orchestra.run.smithi046.stdout:/ 1m 6m3h 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 msgr2=0x7f7cb40a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 0x7f7cb40a4ed0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0075580 tx=0x7f7cbc009260 comp rx=0 tx=0).stop 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 msgr2=0x7f7cd0199d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0037040 tx=0x7f7cc00045b0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 shutdown_connections 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f7cb40a2a10 0x7f7cb40a4ed0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.842 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7cd019e5e0 0x7f7cd019ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7cd01065d0 0x7f7cd019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 --2- 172.21.15.46:0/2987232149 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7cd0101500 0x7f7cd0199d20 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:17:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 >> 172.21.15.46:0/2987232149 conn(0x7f7cd00fcd80 msgr2=0x7f7cd010c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:17:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.839+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 shutdown_connections 2024-03-08T23:17:50.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:17:50.840+0000 7f7cd6c00640 1 -- 172.21.15.46:0/2987232149 wait complete. 2024-03-08T23:17:50.860 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-08T23:17:50.874 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-08T23:17:50.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:50 smithi062 ceph-mon[35761]: from='client.27677 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:51.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:51 smithi046 ceph-mon[31859]: pgmap v313: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:51.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:51 smithi046 ceph-mon[31859]: from='client.17514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:51.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:51 smithi192 ceph-mon[36700]: pgmap v313: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:51.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:51 smithi192 ceph-mon[36700]: from='client.17514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:51.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:51 smithi062 ceph-mon[35761]: pgmap v313: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:17:51.891 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:51 smithi062 ceph-mon[35761]: from='client.17514 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:17:54.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:53 smithi192 ceph-mon[36700]: pgmap v314: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:17:54.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:53 smithi062 ceph-mon[35761]: pgmap v314: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:17:54.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:53 smithi046 ceph-mon[31859]: pgmap v314: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:17:56.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:55 smithi192 ceph-mon[36700]: pgmap v315: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:17:56.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:55 smithi062 ceph-mon[35761]: pgmap v315: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:17:56.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:55 smithi046 ceph-mon[31859]: pgmap v315: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:17:58.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:57 smithi062 ceph-mon[35761]: pgmap v316: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:17:58.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:57 smithi046 ceph-mon[31859]: pgmap v316: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:17:58.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:57 smithi192 ceph-mon[36700]: pgmap v316: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:18:00.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:17:59 smithi062 ceph-mon[35761]: pgmap v317: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:18:00.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:17:59 smithi046 ceph-mon[31859]: pgmap v317: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:18:00.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:17:59 smithi192 ceph-mon[36700]: pgmap v317: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-08T23:18:02.115 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:01 smithi192 ceph-mon[36700]: pgmap v318: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:18:02.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:01 smithi062 ceph-mon[35761]: pgmap v318: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:18:02.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:01 smithi046 ceph-mon[31859]: pgmap v318: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-08T23:18:04.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:03 smithi062 ceph-mon[35761]: pgmap v319: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-08T23:18:04.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:03 smithi046 ceph-mon[31859]: pgmap v319: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-08T23:18:04.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:03 smithi192 ceph-mon[36700]: pgmap v319: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 44 KiB/s wr, 27 op/s 2024-03-08T23:18:06.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:05 smithi062 ceph-mon[35761]: pgmap v320: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:06.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:05 smithi046 ceph-mon[31859]: pgmap v320: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:06.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:05 smithi192 ceph-mon[36700]: pgmap v320: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:06.348 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-08T23:18:06.348 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-08T23:18:06.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 -- 172.21.15.46:0/3522043621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 msgr2=0x7f64b4109950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 --2- 172.21.15.46:0/3522043621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 0x7f64b4109950 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009cd0 tx=0x7f64a402c0f0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 -- 172.21.15.46:0/3522043621 shutdown_connections 2024-03-08T23:18:06.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 --2- 172.21.15.46:0/3522043621 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b4110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 --2- 172.21.15.46:0/3522043621 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 0x7f64b4103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.581 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 --2- 172.21.15.46:0/3522043621 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 0x7f64b4109950 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 -- 172.21.15.46:0/3522043621 >> 172.21.15.46:0/3522043621 conn(0x7f64b40fefa0 msgr2=0x7f64b41013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 -- 172.21.15.46:0/3522043621 shutdown_connections 2024-03-08T23:18:06.582 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.578+0000 7f64ba200640 1 -- 172.21.15.46:0/3522043621 wait complete. 2024-03-08T23:18:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.579+0000 7f64ba200640 1 Processor -- start 2024-03-08T23:18:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.579+0000 7f64ba200640 1 -- start start 2024-03-08T23:18:06.584 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.579+0000 7f64ba200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 0x7f64b4199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.579+0000 7f64ba200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64ba200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 0x7f64b419e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64ba200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b4113850 con 0x7f64b4103e70 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64ba200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41136d0 con 0x7f64b41034b0 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64ba200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41139d0 con 0x7f64b4109570 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.585 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55538/0 (socket says 172.21.15.46:55538) 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 -- 172.21.15.46:0/2177150268 learned_addr learned my addr 172.21.15.46:0/2177150268 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b3400640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 0x7f64b4199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 -- 172.21.15.46:0/2177150268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 msgr2=0x7f64b419e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 0x7f64b419e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 -- 172.21.15.46:0/2177150268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 msgr2=0x7f64b4199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 0x7f64b4199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.586 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.580+0000 7f64b2a00640 1 -- 172.21.15.46:0/2177150268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b419ed80 con 0x7f64b4103e70 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9f0 tx=0x7f649c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64abe00640 1 -- 172.21.15.46:0/2177150268 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c00cd80 con 0x7f64b4103e70 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b419f070 con 0x7f64b4103e70 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64abe00640 1 -- 172.21.15.46:0/2177150268 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f649c005500 con 0x7f64b4103e70 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64abe00640 1 -- 172.21.15.46:0/2177150268 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c018630 con 0x7f64b4103e70 2024-03-08T23:18:06.587 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.581+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b41a6900 con 0x7f64b4103e70 2024-03-08T23:18:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.584+0000 7f64bce19740 1 Processor -- start 2024-03-08T23:18:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.584+0000 7f64abe00640 1 -- 172.21.15.46:0/2177150268 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f649c0046c0 con 0x7f64b4103e70 2024-03-08T23:18:06.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 -- start start 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f55c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f515410 con 0x556f1f55ca80 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f4b7d50 con 0x556f1f553ea0 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64bce19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f460b80 con 0x556f1f553580 2024-03-08T23:18:06.592 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64b3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64b3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48532/0 (socket says 172.21.15.46:48532) 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64b3e00640 1 -- 172.21.15.46:0/3829460450 learned_addr learned my addr 172.21.15.46:0/3829460450 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64b2a00640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.585+0000 7f64b3400640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f55c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 -- 172.21.15.46:0/3829460450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 msgr2=0x556f1f55c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f55c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 -- 172.21.15.46:0/3829460450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 msgr2=0x556f1f55ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.593 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 -- 172.21.15.46:0/3829460450 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f1f2967d0 con 0x556f1f553580 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b2a00640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64abe00640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 0x7f64980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3e00640 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009c40 tx=0x7f64ac00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec593cf2fbffbbd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64abe00640 1 -- 172.21.15.46:0/2177150268 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f649c0cf8b0 con 0x7f64b4103e70 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64b3400640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 0x7f64980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.586+0000 7f64a3e00640 1 -- 172.21.15.46:0/3829460450 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac005020 con 0x556f1f553580 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64a3e00640 1 -- 172.21.15.46:0/3829460450 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64ac0047a0 con 0x556f1f553580 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64a3e00640 1 -- 172.21.15.46:0/3829460450 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac010690 con 0x556f1f553580 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 -- 172.21.15.46:0/3829460450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 msgr2=0x556f1f553960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64b3400640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 0x7f64980a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009ca0 tx=0x7f64a40093a0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f64ac009c40 tx=0x7f64ac00e720 comp rx=0 tx=0).stop 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 -- 172.21.15.46:0/3829460450 shutdown_connections 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f55c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 --2- 172.21.15.46:0/3829460450 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f553960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 -- 172.21.15.46:0/3829460450 >> 172.21.15.46:0/3829460450 conn(0x556f1f54dcb0 msgr2=0x556f1f54e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 -- 172.21.15.46:0/3829460450 shutdown_connections 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.587+0000 7f64bce19740 1 -- 172.21.15.46:0/3829460450 wait complete. 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.588+0000 7f64bce19740 1 -- start start 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.588+0000 7f64bce19740 1 Processor -- start 2024-03-08T23:18:06.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f559a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f559fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f473ee0 con 0x556f1f55ca80 2024-03-08T23:18:06.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f4b3df0 con 0x556f1f553ea0 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f1f55f870 con 0x556f1f553580 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64bce19740 10 client.0 _ll_register_callbacks cb 0x556f1f47a8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f559fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55554/0 (socket says 172.21.15.46:55554) 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b3e00640 1 -- 172.21.15.46:0/2139520176 learned_addr learned my addr 172.21.15.46:0/2139520176 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b3400640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f559a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 -- 172.21.15.46:0/2139520176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 msgr2=0x556f1f559a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f559a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 -- 172.21.15.46:0/2139520176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 msgr2=0x556f1f55a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.599 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.589+0000 7f64b2a00640 1 -- 172.21.15.46:0/2139520176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f1f1cc3f0 con 0x556f1f553ea0 2024-03-08T23:18:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f559fd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f649c0ccbb0 tx=0x7f649c09b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c68968963440597f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:18:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649c0cd070 con 0x556f1f553ea0 2024-03-08T23:18:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64ac0047a0 con 0x556f1f553ea0 2024-03-08T23:18:06.602 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f649c0cf630 con 0x556f1f553ea0 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f649c09baf0 con 0x556f1f553ea0 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64bce19740 10 client.27695 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556f1f18c0f0 con 0x556f1f553ea0 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.590+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64ac0047a0 con 0x556f1f553ea0 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f649c0ba130 con 0x556f1f553ea0 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 10 client.27695 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:06.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 10 client.27695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 20 client.27695 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f56ccb0 0x556f1f56f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 4 client.27695 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556f1f4c5290 con 0x556f1f56ccb0 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.591+0000 7f64b3400640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f56ccb0 0x556f1f56f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.592+0000 7f64b3400640 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f56ccb0 0x556f1f56f0a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8992ba4cd9c199eb server_cookie=54e437f1d29a5fa1 in_seq=0 out_seq=0 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.592+0000 7f64a0c00640 10 client.27695 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:06.604 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.592+0000 7f64a0c00640 1 -- 172.21.15.46:0/2139520176 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556f1f4c5290 con 0x556f1f56ccb0 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.592+0000 7f64a0c00640 10 client.27695 handle_command_reply: tid=0 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 client.27695 shutdown 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 msgr2=0x556f1f559fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f559fd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f649c0ccbb0 tx=0x7f649c09b940 comp rx=0 tx=0).stop 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 shutdown_connections 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f56ccb0 0x556f1f56f0a0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f55ca80 0x556f1f55a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.605 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f559fd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64a0c00640 0 client.27695 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 --2- 172.21.15.46:0/2139520176 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f553580 0x556f1f559a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 >> 172.21.15.46:0/2139520176 conn(0x556f1f54dcb0 msgr2=0x556f1f54cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 shutdown_connections 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 1 -- 172.21.15.46:0/2139520176 wait complete. 2024-03-08T23:18:06.606 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.593+0000 7f64bce19740 20 client.27695 trim_cache size 0 max 16384 2024-03-08T23:18:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 Processor -- start 2024-03-08T23:18:06.612 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 -- start start 2024-03-08T23:18:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f55c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.613 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f563ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41b2dc0 con 0x556f1f553580 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41b0960 con 0x556f1f553ea0 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.610+0000 7f64bce19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41aea20 con 0x556f1f55ca80 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.614 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f55c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f563ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:32794/0 (socket says 172.21.15.46:32794) 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b2a00640 1 -- 172.21.15.46:0/2997075992 learned_addr learned my addr 172.21.15.46:0/2997075992 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 -- 172.21.15.46:0/2997075992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 msgr2=0x556f1f563ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 -- 172.21.15.46:0/2997075992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 msgr2=0x556f1f55c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.615 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f55c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 -- 172.21.15.46:0/2997075992 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f1f55ee70 con 0x556f1f55ca80 2024-03-08T23:18:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b2a00640 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3400640 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f55c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:06.616 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.611+0000 7f64b3e00640 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f563ff0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00ef10 tx=0x7f64ac009a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cebdc76ce0067c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.612+0000 7f64a3400640 1 -- 172.21.15.46:0/2997075992 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac027030 con 0x556f1f55ca80 2024-03-08T23:18:06.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.612+0000 7f64a3400640 1 -- 172.21.15.46:0/2997075992 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64ac004520 con 0x556f1f55ca80 2024-03-08T23:18:06.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.612+0000 7f64a3400640 1 -- 172.21.15.46:0/2997075992 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ac01d040 con 0x556f1f55ca80 2024-03-08T23:18:06.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- 172.21.15.46:0/2997075992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 msgr2=0x556f1f563ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.621 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f563ff0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00ef10 tx=0x7f64ac009a00 comp rx=0 tx=0).stop 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- 172.21.15.46:0/2997075992 shutdown_connections 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f563ff0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f563ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 --2- 172.21.15.46:0/2997075992 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f55c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- 172.21.15.46:0/2997075992 >> 172.21.15.46:0/2997075992 conn(0x556f1f54dcb0 msgr2=0x556f1f554dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.622 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- 172.21.15.46:0/2997075992 shutdown_connections 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- 172.21.15.46:0/2997075992 wait complete. 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.613+0000 7f64bce19740 1 -- start start 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 Processor -- start 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f5b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f5bb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.623 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41af020 con 0x556f1f553580 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41b0300 con 0x556f1f553ea0 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64b41af640 con 0x556f1f55ca80 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64bce19740 10 client.0 _ll_register_callbacks cb 0x7f64ac009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64b3400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55578/0 (socket says 172.21.15.46:55578) 2024-03-08T23:18:06.624 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.614+0000 7f64b3400640 1 -- 172.21.15.46:0/3094755085 learned_addr learned my addr 172.21.15.46:0/3094755085 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3e00640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f5bb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b2a00640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f5b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 -- 172.21.15.46:0/3094755085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 msgr2=0x556f1f5bb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f5bb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 -- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 msgr2=0x556f1f5b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f5b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 -- 172.21.15.46:0/3094755085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f1f5bbe00 con 0x556f1f553580 2024-03-08T23:18:06.625 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64b3400640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009430 tx=0x7f64a402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e797d9a3a77d9f server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a4039040 con 0x556f1f553580 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f64a4032dd0 con 0x556f1f553580 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.615+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64ac004520 con 0x556f1f553580 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64a4062050 con 0x556f1f553580 2024-03-08T23:18:06.626 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 10 client.17532 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556f1f5c43b0 con 0x556f1f553580 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64ac004520 con 0x556f1f553580 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f64a4004030 con 0x556f1f553580 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 10 client.17532 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 10 client.17532 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 20 client.17532 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:06.627 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f5c4ac0 0x556f1f5c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 4 client.17532 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.616+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556f1f4bf5e0 con 0x556f1f5c4ac0 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64b2a00640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f5c4ac0 0x556f1f5c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64b2a00640 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f5c4ac0 0x556f1f5c6ee0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec27ca59a0100ebe server_cookie=ca9c58af9296d18c in_seq=0 out_seq=0 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64a0c00640 10 client.17532 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64a0c00640 1 -- 172.21.15.46:0/3094755085 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556f1f4bf5e0 con 0x556f1f5c4ac0 2024-03-08T23:18:06.628 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64a0c00640 10 client.17532 handle_command_reply: tid=0 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.617+0000 7f64bce19740 1 client.17532 shutdown 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 msgr2=0x556f1f5b72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009430 tx=0x7f64a402c0f0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 shutdown_connections 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x556f1f5c4ac0 0x556f1f5c6ee0 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x556f1f55ca80 0x556f1f5bb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.629 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x556f1f553ea0 0x556f1f5b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 --2- 172.21.15.46:0/3094755085 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x556f1f553580 0x556f1f5b72a0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64a0c00640 0 client.17532 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 >> 172.21.15.46:0/3094755085 conn(0x556f1f54dcb0 msgr2=0x556f1f54c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 shutdown_connections 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 1 -- 172.21.15.46:0/3094755085 wait complete. 2024-03-08T23:18:06.630 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64bce19740 20 client.17532 trim_cache size 0 max 16384 2024-03-08T23:18:06.631 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:06.631 INFO:teuthology.orchestra.run.smithi046.stdout: "status": "no active scrubs running", 2024-03-08T23:18:06.631 INFO:teuthology.orchestra.run.smithi046.stdout: "scrubs": {} 2024-03-08T23:18:06.631 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 msgr2=0x7f64980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.618+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 0x7f64980a4e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f64a4009ca0 tx=0x7f64a40093a0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 msgr2=0x7f64b419a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.632 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f649c00e9f0 tx=0x7f649c00edf0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 shutdown_connections 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f64980a2970 0x7f64980a4e30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b4109570 0x7f64b419e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f64b4103e70 0x7f64b419a2d0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 --2- 172.21.15.46:0/2177150268 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f64b41034b0 0x7f64b4199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 >> 172.21.15.46:0/2177150268 conn(0x7f64b40fefa0 msgr2=0x7f64b4100740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.633 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 shutdown_connections 2024-03-08T23:18:06.634 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.619+0000 7f64ba200640 1 -- 172.21.15.46:0/2177150268 wait complete. 2024-03-08T23:18:06.639 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-08T23:18:06.640 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:18:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.860+0000 7fb7abe00640 1 -- 172.21.15.46:0/3261142852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a410b8f0 msgr2=0x7fb7a410dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.860+0000 7fb7abe00640 1 --2- 172.21.15.46:0/3261142852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a410b8f0 0x7fb7a410dd20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fb79c009ca0 tx=0x7fb79c02c180 comp rx=0 tx=0).stop 2024-03-08T23:18:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 -- 172.21.15.46:0/3261142852 shutdown_connections 2024-03-08T23:18:06.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 --2- 172.21.15.46:0/3261142852 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a410b8f0 0x7fb7a410dd20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 --2- 172.21.15.46:0/3261142852 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a410b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 --2- 172.21.15.46:0/3261142852 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a4102520 0x7fb7a4102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 -- 172.21.15.46:0/3261142852 >> 172.21.15.46:0/3261142852 conn(0x7fb7a4076570 msgr2=0x7fb7a4076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:06.864 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 -- 172.21.15.46:0/3261142852 shutdown_connections 2024-03-08T23:18:06.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.861+0000 7fb7abe00640 1 -- 172.21.15.46:0/3261142852 wait complete. 2024-03-08T23:18:06.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.863+0000 7fb7abe00640 1 Processor -- start 2024-03-08T23:18:06.865 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.863+0000 7fb7abe00640 1 -- start start 2024-03-08T23:18:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4102520 0x7fb7a4199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a419a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 0x7fb7a419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.866 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4100c30 con 0x7fb7a4102520 2024-03-08T23:18:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4100ab0 con 0x7fb7a410b8f0 2024-03-08T23:18:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.864+0000 7fb7abe00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a4100db0 con 0x7fb7a4102e40 2024-03-08T23:18:06.867 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a8a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a419a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a8a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a419a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48576/0 (socket says 172.21.15.46:48576) 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a8a00640 1 -- 172.21.15.46:0/2453872855 learned_addr learned my addr 172.21.15.46:0/2453872855 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 0x7fb7a419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9400640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4102520 0x7fb7a4199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 -- 172.21.15.46:0/2453872855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 msgr2=0x7fb7a419a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a419a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.868 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 -- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4102520 msgr2=0x7fb7a4199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4102520 0x7fb7a4199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.865+0000 7fb7a9e00640 1 -- 172.21.15.46:0/2453872855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a419ed40 con 0x7fb7a410b8f0 2024-03-08T23:18:06.869 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7a9e00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 0x7fb7a419e610 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb79c037040 tx=0x7fb79c035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c039040 con 0x7fb7a410b8f0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb79c044070 con 0x7fb7a410b8f0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a419efd0 con 0x7fb7a410b8f0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb79c0328d0 con 0x7fb7a410b8f0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.866+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a41a68c0 con 0x7fb7a410b8f0 2024-03-08T23:18:06.870 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.867+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4069170 con 0x7fb7a410b8f0 2024-03-08T23:18:06.871 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.869+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb79c003b70 con 0x7fb7a410b8f0 2024-03-08T23:18:06.872 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.870+0000 7fb7a2000640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 0x7fb78c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.871+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fb79c0f1540 con 0x7fb7a410b8f0 2024-03-08T23:18:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.871+0000 7fb7a9400640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 0x7fb78c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:06.873 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.872+0000 7fb7a9400640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 0x7fb78c0a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb798009b60 tx=0x7fb798005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:06.877 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:06.875+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb79c0bdf40 con 0x7fb7a410b8f0 2024-03-08T23:18:07.135 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.133+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7a41059e0 con 0x7fb7a410b8f0 2024-03-08T23:18:07.136 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.134+0000 7fb7a2000640 1 -- 172.21.15.46:0/2453872855 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7fb79c0bd840 con 0x7fb7a410b8f0 2024-03-08T23:18:07.137 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:07.137 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:18:07.137 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:18:07.139 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 msgr2=0x7fb78c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 0x7fb78c0a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb798009b60 tx=0x7fb798005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 msgr2=0x7fb7a419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 0x7fb7a419e610 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb79c037040 tx=0x7fb79c035d80 comp rx=0 tx=0).stop 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 shutdown_connections 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb78c0a2970 0x7fb78c0a4e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.140 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb7a410b8f0 0x7fb7a419e610 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb7a4102e40 0x7fb7a419a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 --2- 172.21.15.46:0/2453872855 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb7a4102520 0x7fb7a4199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 >> 172.21.15.46:0/2453872855 conn(0x7fb7a4076570 msgr2=0x7fb7a410d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 shutdown_connections 2024-03-08T23:18:07.141 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.137+0000 7fb7abe00640 1 -- 172.21.15.46:0/2453872855 wait complete. 2024-03-08T23:18:07.158 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:18:07.158 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-08T23:18:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 -- 172.21.15.46:0/457317134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073fb0 msgr2=0x7fa26c111d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 --2- 172.21.15.46:0/457317134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073fb0 0x7fa26c111d50 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa254009900 tx=0x7fa25402bcc0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.383 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 -- 172.21.15.46:0/457317134 shutdown_connections 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 --2- 172.21.15.46:0/457317134 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c112290 0x7fa26c114680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 --2- 172.21.15.46:0/457317134 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073fb0 0x7fa26c111d50 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 --2- 172.21.15.46:0/457317134 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c073690 0x7fa26c073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 -- 172.21.15.46:0/457317134 >> 172.21.15.46:0/457317134 conn(0x7fa26c0786f0 msgr2=0x7fa26c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 -- 172.21.15.46:0/457317134 shutdown_connections 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.378+0000 7fa271400640 1 -- 172.21.15.46:0/457317134 wait complete. 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.379+0000 7fa271400640 1 Processor -- start 2024-03-08T23:18:07.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 -- start start 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c073fb0 0x7fa26c19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c112290 0x7fa26c1a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c117ce0 con 0x7fa26c073690 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c117b60 con 0x7fa26c112290 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa271400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c117e60 con 0x7fa26c073fb0 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55616/0 (socket says 172.21.15.46:55616) 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa26a800640 1 -- 172.21.15.46:0/1247540440 learned_addr learned my addr 172.21.15.46:0/1247540440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa26b200640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c112290 0x7fa26c1a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.380+0000 7fa263e00640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c073fb0 0x7fa26c19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 -- 172.21.15.46:0/1247540440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c073fb0 msgr2=0x7fa26c19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c073fb0 0x7fa26c19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 -- 172.21.15.46:0/1247540440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c112290 msgr2=0x7fa26c1a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c112290 0x7fa26c1a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 -- 172.21.15.46:0/1247540440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c1a31d0 con 0x7fa26c073690 2024-03-08T23:18:07.387 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa26a800640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00e9f0 tx=0x7fa25c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa263400640 1 -- 172.21.15.46:0/1247540440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c00cd80 con 0x7fa26c073690 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa263400640 1 -- 172.21.15.46:0/1247540440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25c010910 con 0x7fa26c073690 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa26c1a34c0 con 0x7fa26c073690 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa263400640 1 -- 172.21.15.46:0/1247540440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25c018990 con 0x7fa26c073690 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.381+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa26c1aad50 con 0x7fa26c073690 2024-03-08T23:18:07.388 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.385+0000 7fa274192740 1 Processor -- start 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.385+0000 7fa263400640 1 -- 172.21.15.46:0/1247540440 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa25c010ac0 con 0x7fa26c073690 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.385+0000 7fa274192740 1 -- start start 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48ca730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.389 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c482edd0 con 0x5591c48ca350 2024-03-08T23:18:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c47d7c70 con 0x5591c48d3850 2024-03-08T23:18:07.390 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa274192740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c482ae70 con 0x5591c48cac70 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa26b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48ca730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa263e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa263e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:32850/0 (socket says 172.21.15.46:32850) 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa263e00640 1 -- 172.21.15.46:0/1607128675 learned_addr learned my addr 172.21.15.46:0/1607128675 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.386+0000 7fa26a800640 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 -- 172.21.15.46:0/1607128675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 msgr2=0x5591c48d3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.392 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 -- 172.21.15.46:0/1607128675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 msgr2=0x5591c48ca730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48ca730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 -- 172.21.15.46:0/1607128675 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591c48912b0 con 0x5591c48d3850 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa26b200640 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 0x7fa2500a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.387+0000 7fa263e00640 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c00 tx=0x7fa254009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac1a43f53060c18b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.388+0000 7fa263400640 1 -- 172.21.15.46:0/1247540440 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fa25c0cf510 con 0x7fa26c073690 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.388+0000 7fa26b200640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 0x7fa2500a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.393 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.388+0000 7fa25b400640 1 -- 172.21.15.46:0/1607128675 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254043070 con 0x5591c48d3850 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.388+0000 7fa25b400640 1 -- 172.21.15.46:0/1607128675 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25402c6e0 con 0x5591c48d3850 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.388+0000 7fa25b400640 1 -- 172.21.15.46:0/1607128675 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254004a40 con 0x5591c48d3850 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 -- 172.21.15.46:0/1607128675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 msgr2=0x5591c48d5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa26b200640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 0x7fa2500a4eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa26c06ab90 tx=0x7fa264009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fa254000c00 tx=0x7fa254009560 comp rx=0 tx=0).stop 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 -- 172.21.15.46:0/1607128675 shutdown_connections 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d5c40 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.394 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 --2- 172.21.15.46:0/1607128675 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48ca730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 -- 172.21.15.46:0/1607128675 >> 172.21.15.46:0/1607128675 conn(0x5591c48c4a80 msgr2=0x5591c48c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 -- 172.21.15.46:0/1607128675 shutdown_connections 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.389+0000 7fa274192740 1 -- 172.21.15.46:0/1607128675 wait complete. 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 -- start start 2024-03-08T23:18:07.395 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 Processor -- start 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48ca350 0x5591c48d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d04a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c48d65e0 con 0x5591c48d3850 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c47eb0c0 con 0x5591c48ca350 2024-03-08T23:18:07.399 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x5591c48d6760 con 0x5591c48cac70 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.390+0000 7fa274192740 10 client.0 _ll_register_callbacks cb 0x5591c4825ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa263e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55632/0 (socket says 172.21.15.46:55632) 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 -- 172.21.15.46:0/2084333241 learned_addr learned my addr 172.21.15.46:0/2084333241 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26a800640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48ca350 0x5591c48d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 -- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 msgr2=0x5591c48d04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.400 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 -- 172.21.15.46:0/2084333241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48ca350 msgr2=0x5591c48d3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48ca350 0x5591c48d3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 -- 172.21.15.46:0/2084333241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591c45433f0 con 0x5591c48d3850 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa26b200640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa264009fd0 tx=0x7fa264004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f908df9f87031430 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.391+0000 7fa24fe00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26400f810 con 0x5591c48d3850 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fa26400f9b0 con 0x5591c48d3850 2024-03-08T23:18:07.401 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa25402c6e0 con 0x5591c48d3850 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa26400fbb0 con 0x5591c48d3850 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa274192740 10 client.17550 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5591c460d7d0 con 0x5591c48d3850 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa25402c6e0 con 0x5591c48d3850 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.392+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa264027070 con 0x5591c48d3850 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 10 client.17550 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 10 client.17550 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:18:07.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 20 client.17550 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c48e3880 0x5591c48e5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 4 client.17550 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5591c48034f0 con 0x5591c48e3880 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.393+0000 7fa26a800640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c48e3880 0x5591c48e5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa26a800640 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c48e3880 0x5591c48e5cb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=52b0f0fd7278bd32 server_cookie=c108d573d1cb2461 in_seq=0 out_seq=0 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa24fe00640 10 client.17550 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa24fe00640 1 -- 172.21.15.46:0/2084333241 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5591c48034f0 con 0x5591c48e3880 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa24fe00640 10 client.17550 handle_command_reply: tid=0 2024-03-08T23:18:07.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa274192740 1 client.17550 shutdown 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 msgr2=0x5591c48d09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.394+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fa264009fd0 tx=0x7fa264004ab0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 shutdown_connections 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c48e3880 0x5591c48e5cb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48d3850 0x5591c48d09e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 --2- 172.21.15.46:0/2084333241 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48ca350 0x5591c48d3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.404 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 >> 172.21.15.46:0/2084333241 conn(0x5591c48c4a80 msgr2=0x5591c48c67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa24fe00640 0 client.17550 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:18:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 shutdown_connections 2024-03-08T23:18:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 1 -- 172.21.15.46:0/2084333241 wait complete. 2024-03-08T23:18:07.405 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.395+0000 7fa274192740 20 client.17550 trim_cache size 0 max 16384 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.407+0000 7fa274192740 1 Processor -- start 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 -- start start 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.411 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b71f0 con 0x5591c48ca350 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b4d90 con 0x5591c48d3850 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa274192740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b2e50 con 0x5591c48cac70 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa263e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa263e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48618/0 (socket says 172.21.15.46:48618) 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa263e00640 1 -- 172.21.15.46:0/3423069411 learned_addr learned my addr 172.21.15.46:0/3423069411 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55638/0 (socket says 172.21.15.46:55638) 2024-03-08T23:18:07.412 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.408+0000 7fa26b200640 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 -- 172.21.15.46:0/3423069411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 msgr2=0x5591c48da9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48da9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 -- 172.21.15.46:0/3423069411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 msgr2=0x5591c48d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 -- 172.21.15.46:0/3423069411 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5591c48e0040 con 0x5591c48cac70 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa26a800640 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.413 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa263e00640 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa254009900 tx=0x7fa25402bcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d9e203c3a361016 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa25aa00640 1 -- 172.21.15.46:0/3423069411 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254050070 con 0x5591c48cac70 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa25aa00640 1 -- 172.21.15.46:0/3423069411 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2540046c0 con 0x5591c48cac70 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.409+0000 7fa25aa00640 1 -- 172.21.15.46:0/3423069411 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa254043070 con 0x5591c48cac70 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- 172.21.15.46:0/3423069411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 msgr2=0x5591c48d3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa254009900 tx=0x7fa25402bcc0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.417 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- 172.21.15.46:0/3423069411 shutdown_connections 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48da9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d3110 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 --2- 172.21.15.46:0/3423069411 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- 172.21.15.46:0/3423069411 >> 172.21.15.46:0/3423069411 conn(0x5591c48c4a80 msgr2=0x5591c48cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- 172.21.15.46:0/3423069411 shutdown_connections 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- 172.21.15.46:0/3423069411 wait complete. 2024-03-08T23:18:07.418 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.410+0000 7fa274192740 1 -- start start 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 Processor -- start 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d9470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b3450 con 0x5591c48ca350 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b4730 con 0x5591c48d3850 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c1b3a70 con 0x5591c48cac70 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa274192740 10 client.0 _ll_register_callbacks cb 0x7fa2640093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.419 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.411+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55640/0 (socket says 172.21.15.46:55640) 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26a800640 1 -- 172.21.15.46:0/2517272454 learned_addr learned my addr 172.21.15.46:0/2517272454 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa263e00640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 -- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 msgr2=0x5591c48d99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 -- 172.21.15.46:0/2517272454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 msgr2=0x5591c48d9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.420 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 -- 172.21.15.46:0/2517272454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5591c4939dd0 con 0x5591c48d3850 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa263e00640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa26b200640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d9ef0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x5591c48ceca0 tx=0x7fa26400fb20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cebe1df80c4df63e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.412+0000 7fa24fe00640 10 client.0 ms_handle_connect on v2:172.21.15.62:3300/0 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa26403c060 con 0x5591c48d3850 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mon.2 v2:172.21.15.62:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fa264004d20 con 0x5591c48d3850 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2540046c0 con 0x5591c48d3850 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa264043050 con 0x5591c48d3850 2024-03-08T23:18:07.421 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa274192740 10 client.27725 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.413+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5591c493b0b0 con 0x5591c48d3850 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mon.2 v2:172.21.15.62:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2540046c0 con 0x5591c48d3850 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mon.2 v2:172.21.15.62:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa264027040 con 0x5591c48d3850 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 10 client.27725 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 10 client.27725 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 20 client.27725 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c493b750 0x5591c493db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.422 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 4 client.27725 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:18:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 --> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5591c483c310 con 0x5591c493b750 2024-03-08T23:18:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.414+0000 7fa26a800640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c493b750 0x5591c493db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa26a800640 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c493b750 0x5591c493db70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b04077b6efe801e1 server_cookie=6ae9088926bc4882 in_seq=0 out_seq=0 2024-03-08T23:18:07.423 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa24fe00640 10 client.27725 ms_handle_connect on v2:172.21.15.192:6838/441315746 2024-03-08T23:18:07.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa24fe00640 1 -- 172.21.15.46:0/2517272454 <== mds.2 v2:172.21.15.192:6838/441315746 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5591c483c310 con 0x5591c493b750 2024-03-08T23:18:07.424 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa24fe00640 10 client.27725 handle_command_reply: tid=0 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 client.27725 shutdown 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 msgr2=0x5591c48d9ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d9ef0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x5591c48ceca0 tx=0x7fa26400fb20 comp rx=0 tx=0).stop 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 shutdown_connections 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:6838/441315746,v1:172.21.15.192:6839/441315746] conn(0x5591c493b750 0x5591c493db70 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x5591c48d3850 0x5591c48d9ef0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5591c48cac70 0x5591c48d99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.425 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa24fe00640 0 client.27725 ms_handle_reset on v2:172.21.15.192:6838/441315746 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 --2- 172.21.15.46:0/2517272454 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x5591c48ca350 0x5591c48d9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.415+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 >> 172.21.15.46:0/2517272454 conn(0x5591c48c4a80 msgr2=0x5591c48cb8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 shutdown_connections 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa274192740 1 -- 172.21.15.46:0/2517272454 wait complete. 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa274192740 20 client.27725 trim_cache size 0 max 16384 2024-03-08T23:18:07.426 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:18:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 msgr2=0x7fa2500a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 0x7fa2500a4eb0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa26c06ab90 tx=0x7fa264009500 comp rx=0 tx=0).stop 2024-03-08T23:18:07.427 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 msgr2=0x7fa26c19e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00e9f0 tx=0x7fa25c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 shutdown_connections 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa2500a29f0 0x7fa2500a4eb0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa26c112290 0x7fa26c1a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa26c073fb0 0x7fa26c19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 --2- 172.21.15.46:0/1247540440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa26c073690 0x7fa26c19e1e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.416+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 >> 172.21.15.46:0/1247540440 conn(0x7fa26c0786f0 msgr2=0x7fa26c10dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.428 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.417+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 shutdown_connections 2024-03-08T23:18:07.429 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.417+0000 7fa271400640 1 -- 172.21.15.46:0/1247540440 wait complete. 2024-03-08T23:18:07.436 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:18:07.436 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-08T23:18:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 -- 172.21.15.46:0/3348869731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c1086b0 msgr2=0x7f755c110280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 --2- 172.21.15.46:0/3348869731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c1086b0 0x7f755c110280 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7554009ca0 tx=0x7f755402c100 comp rx=0 tx=0).stop 2024-03-08T23:18:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 -- 172.21.15.46:0/3348869731 shutdown_connections 2024-03-08T23:18:07.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 --2- 172.21.15.46:0/3348869731 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c1086b0 0x7f755c110280 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 --2- 172.21.15.46:0/3348869731 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c103f90 0x7f755c107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 --2- 172.21.15.46:0/3348869731 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1035e0 0x7f755c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.646+0000 7f7562200640 1 -- 172.21.15.46:0/3348869731 >> 172.21.15.46:0/3348869731 conn(0x7f755c0fee20 msgr2=0x7f755c101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.647+0000 7f7562200640 1 -- 172.21.15.46:0/3348869731 shutdown_connections 2024-03-08T23:18:07.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.647+0000 7f7562200640 1 -- 172.21.15.46:0/3348869731 wait complete. 2024-03-08T23:18:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.648+0000 7f7562200640 1 Processor -- start 2024-03-08T23:18:07.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.648+0000 7f7562200640 1 -- start start 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 0x7f755c19fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 0x7f755c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1137c0 con 0x7f755c199aa0 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c113640 con 0x7f755c1035e0 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7562200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c113940 con 0x7f755c1086b0 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7561800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 0x7f755c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:32902/0 (socket says 172.21.15.46:32902) 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 -- 172.21.15.46:0/3250921909 learned_addr learned my addr 172.21.15.46:0/3250921909 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f755be00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 0x7f755c19fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 -- 172.21.15.46:0/3250921909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 msgr2=0x7f755c19fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 0x7f755c19fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 -- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 msgr2=0x7f755c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.649+0000 7f7560e00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 0x7f755c199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7560e00640 1 -- 172.21.15.46:0/3250921909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755c19a5c0 con 0x7f755c1035e0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7561800640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 0x7f755c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f755be00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 0x7f755c19fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7560e00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f755000b4a0 tx=0x7f755000b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7559600640 1 -- 172.21.15.46:0/3250921909 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550014070 con 0x7f755c1035e0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7559600640 1 -- 172.21.15.46:0/3250921909 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7550010af0 con 0x7f755c1035e0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f755c19a8b0 con 0x7f755c1035e0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7559600640 1 -- 172.21.15.46:0/3250921909 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550017d10 con 0x7f755c1035e0 2024-03-08T23:18:07.655 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.650+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f755c1a6b00 con 0x7f755c1035e0 2024-03-08T23:18:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.654+0000 7f75665ad740 1 Processor -- start 2024-03-08T23:18:07.656 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 -- start start 2024-03-08T23:18:07.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 0x55c879f6f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.657 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6fe50 0x55c879f784f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f7559600640 1 -- 172.21.15.46:0/3250921909 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7550017070 con 0x7f755c1035e0 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879ed3dd0 con 0x55c879f78a30 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879e7cc70 con 0x55c879f6fe50 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f75665ad740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879ecfe70 con 0x55c879f6f530 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f755be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f7561800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 0x55c879f6f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f755be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55654/0 (socket says 172.21.15.46:55654) 2024-03-08T23:18:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f755be00640 1 -- 172.21.15.46:0/1690252154 learned_addr learned my addr 172.21.15.46:0/1690252154 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.659 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.655+0000 7f7560e00640 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6fe50 0x55c879f784f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 -- 172.21.15.46:0/1690252154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6fe50 msgr2=0x55c879f784f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.663 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6fe50 0x55c879f784f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 -- 172.21.15.46:0/1690252154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 msgr2=0x55c879f7ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 -- 172.21.15.46:0/1690252154 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c879f362b0 con 0x55c879f6f530 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f755be00640 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.656+0000 7f7561800640 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 0x55c879f6f910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7554009bf0 tx=0x7f75540098c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dd941f96d9070f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f754e000640 1 -- 172.21.15.46:0/1690252154 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554043070 con 0x55c879f6f530 2024-03-08T23:18:07.664 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f754e000640 1 -- 172.21.15.46:0/1690252154 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7554003970 con 0x55c879f6f530 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f754e000640 1 -- 172.21.15.46:0/1690252154 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554039040 con 0x55c879f6f530 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f7559600640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 0x7f75440a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f7559600640 1 -- 172.21.15.46:0/3250921909 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f75500ce500 con 0x7f755c1035e0 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f7560e00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 0x7f75440a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 -- 172.21.15.46:0/1690252154 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 msgr2=0x55c879f6f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 0x55c879f6f910 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7554009bf0 tx=0x7f75540098c0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 -- 172.21.15.46:0/1690252154 shutdown_connections 2024-03-08T23:18:07.665 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6fe50 0x55c879f784f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 --2- 172.21.15.46:0/1690252154 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6f530 0x55c879f6f910 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.657+0000 7f75665ad740 1 -- 172.21.15.46:0/1690252154 >> 172.21.15.46:0/1690252154 conn(0x55c879f69c60 msgr2=0x55c879f6a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.658+0000 7f75665ad740 1 -- 172.21.15.46:0/1690252154 shutdown_connections 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.658+0000 7f75665ad740 1 -- 172.21.15.46:0/1690252154 wait complete. 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.658+0000 7f7560e00640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 0x7f75440a4eb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75500d1fd0 tx=0x7f75500d4060 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.658+0000 7f75665ad740 1 -- start start 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.658+0000 7f75665ad740 1 Processor -- start 2024-03-08T23:18:07.666 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f75750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f78a30 0x55c879f75cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879f7b7c0 con 0x55c879f6f530 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879e900c0 con 0x55c879f78a30 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c879f7b940 con 0x55c879f6fe50 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f75665ad740 10 client.0 _ll_register_callbacks cb 0x55c879ecaec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f7561800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.667 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f7561800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55668/0 (socket says 172.21.15.46:55668) 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.659+0000 7f7561800640 1 -- 172.21.15.46:0/2463599594 learned_addr learned my addr 172.21.15.46:0/2463599594 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f78a30 0x55c879f75cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 -- 172.21.15.46:0/2463599594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 msgr2=0x55c879f75750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f75750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 -- 172.21.15.46:0/2463599594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f78a30 msgr2=0x55c879f75cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f78a30 0x55c879f75cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 -- 172.21.15.46:0/2463599594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c879be83f0 con 0x55c879f6f530 2024-03-08T23:18:07.668 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7561800640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7554037040 tx=0x7f7554009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac15c79e63748d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7542a00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7554053060 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f75540329a0 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.660+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7554003970 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7554039040 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 10 client.17568 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c879cb27d0 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7554003970 con 0x55c879f6f530 2024-03-08T23:18:07.669 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7554004560 con 0x55c879f6f530 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 10 client.17568 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 10 client.17568 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 20 client.17568 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879f888d0 0x55c879f8acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 4 client.17568 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c879ea84f0 con 0x55c879f888d0 2024-03-08T23:18:07.670 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.661+0000 7f755be00640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879f888d0 0x55c879f8acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.662+0000 7f755be00640 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879f888d0 0x55c879f8acc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=612bbd39787a8964 server_cookie=b27cedbd05c733e8 in_seq=0 out_seq=0 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.662+0000 7f7542a00640 10 client.17568 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f7542a00640 1 -- 172.21.15.46:0/2463599594 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c879ea84f0 con 0x55c879f888d0 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f7542a00640 10 client.17568 handle_command_reply: tid=0 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 client.17568 shutdown 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 msgr2=0x55c879f783c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7554037040 tx=0x7f7554009580 comp rx=0 tx=0).stop 2024-03-08T23:18:07.672 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 shutdown_connections 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f78a30 0x55c879f75cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879f888d0 0x55c879f8acc0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f75750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 --2- 172.21.15.46:0/2463599594 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f6f530 0x55c879f783c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 >> 172.21.15.46:0/2463599594 conn(0x55c879f69c60 msgr2=0x55c879f6b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f7542a00640 0 client.17568 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 shutdown_connections 2024-03-08T23:18:07.673 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 1 -- 172.21.15.46:0/2463599594 wait complete. 2024-03-08T23:18:07.674 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.663+0000 7f75665ad740 20 client.17568 trim_cache size 0 max 16384 2024-03-08T23:18:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.676+0000 7f75665ad740 1 Processor -- start 2024-03-08T23:18:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.676+0000 7f75665ad740 1 -- start start 2024-03-08T23:18:07.679 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f8f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f77e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1b2c60 con 0x55c879f78a30 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1b0800 con 0x55c879f6f530 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f75665ad740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1ae8c0 con 0x55c879f6fe50 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f7561800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.680 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f755be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f77e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f7561800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55670/0 (socket says 172.21.15.46:55670) 2024-03-08T23:18:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f7561800640 1 -- 172.21.15.46:0/3591160684 learned_addr learned my addr 172.21.15.46:0/3591160684 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.681 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f7561800640 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f8f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f755be00640 1 -- 172.21.15.46:0/3591160684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 msgr2=0x55c879f8f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.677+0000 7f755be00640 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f8f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f755be00640 1 -- 172.21.15.46:0/3591160684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 msgr2=0x55c879f783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f755be00640 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f755be00640 1 -- 172.21.15.46:0/3591160684 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c879f6e620 con 0x55c879f6fe50 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f7561800640 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.685 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f755be00640 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f77e80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55c879f7ba90 tx=0x7f754801af40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ffc4700f743c467 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f754d600640 1 -- 172.21.15.46:0/3591160684 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75480142d0 con 0x55c879f6fe50 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f754d600640 1 -- 172.21.15.46:0/3591160684 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75480149a0 con 0x55c879f6fe50 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.678+0000 7f754d600640 1 -- 172.21.15.46:0/3591160684 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f754801a7d0 con 0x55c879f6fe50 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- 172.21.15.46:0/3591160684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 msgr2=0x55c879f77e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f77e80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55c879f7ba90 tx=0x7f754801af40 comp rx=0 tx=0).stop 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- 172.21.15.46:0/3591160684 shutdown_connections 2024-03-08T23:18:07.686 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f77e80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 --2- 172.21.15.46:0/3591160684 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f8f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- 172.21.15.46:0/3591160684 >> 172.21.15.46:0/3591160684 conn(0x55c879f69c60 msgr2=0x55c879f6a040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- 172.21.15.46:0/3591160684 shutdown_connections 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- 172.21.15.46:0/3591160684 wait complete. 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 -- start start 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.679+0000 7f75665ad740 1 Processor -- start 2024-03-08T23:18:07.687 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f7e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f7ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1aeec0 con 0x55c879f78a30 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1b01a0 con 0x55c879f6f530 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755c1af4e0 con 0x55c879f6fe50 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f75665ad740 10 client.0 _ll_register_callbacks cb 0x7f7548005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f7561800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f7e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.688 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f755be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f7ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f7561800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f7e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:32938/0 (socket says 172.21.15.46:32938) 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f7561800640 1 -- 172.21.15.46:0/283258862 learned_addr learned my addr 172.21.15.46:0/283258862 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.680+0000 7f7561800640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 -- 172.21.15.46:0/283258862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 msgr2=0x55c879f7e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f7e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 -- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 msgr2=0x55c879f7f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 -- 172.21.15.46:0/283258862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c879fdeb60 con 0x55c879f6fe50 2024-03-08T23:18:07.689 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f755be00640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f7ec50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55c879f6af80 tx=0x7f7548005da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9f4f6bd1b161dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f7542a00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75480142d0 con 0x55c879f6fe50 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75480149a0 con 0x55c879f6fe50 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.681+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f7548014b50 con 0x55c879f6fe50 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7548035050 con 0x55c879f6fe50 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 10 client.27679 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.690 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c879fdfe90 con 0x55c879f6fe50 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75480149a0 con 0x55c879f6fe50 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f754800f040 con 0x55c879f6fe50 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 10 client.27679 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 10 client.27679 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.a 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 20 client.27679 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879fe0510 0x55c879fe2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 4 client.27679 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 --> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c879ee1310 con 0x55c879fe0510 2024-03-08T23:18:07.691 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.682+0000 7f7561800640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879fe0510 0x55c879fe2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f7561800640 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879fe0510 0x55c879fe2900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e39d8825684c479 server_cookie=166e6507799c6354 in_seq=0 out_seq=0 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f7542a00640 10 client.27679 ms_handle_connect on v2:172.21.15.46:6840/2979513578 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f7542a00640 1 -- 172.21.15.46:0/283258862 <== mds.1 v2:172.21.15.46:6840/2979513578 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c879ee1310 con 0x55c879fe0510 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f7542a00640 10 client.27679 handle_command_reply: tid=0 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 client.27679 shutdown 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 msgr2=0x55c879f7ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f7ec50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55c879f6af80 tx=0x7f7548005da0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.693 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 shutdown_connections 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x55c879f78a30 0x55c879f7f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.46:6840/2979513578,v1:172.21.15.46:6841/2979513578] conn(0x55c879fe0510 0x55c879fe2900 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.683+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55c879f6fe50 0x55c879f7ec50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f7542a00640 0 client.27679 ms_handle_reset on v2:172.21.15.46:6840/2979513578 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f75665ad740 1 --2- 172.21.15.46:0/283258862 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x55c879f6f530 0x55c879f7e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 >> 172.21.15.46:0/283258862 conn(0x55c879f69c60 msgr2=0x55c879f8f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.694 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 shutdown_connections 2024-03-08T23:18:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f75665ad740 1 -- 172.21.15.46:0/283258862 wait complete. 2024-03-08T23:18:07.695 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f75665ad740 20 client.27679 trim_cache size 0 max 16384 2024-03-08T23:18:07.695 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:18:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 msgr2=0x7f75440a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 0x7f75440a4eb0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f75500d1fd0 tx=0x7f75500d4060 comp rx=0 tx=0).stop 2024-03-08T23:18:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 msgr2=0x7f755c19f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.696 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.684+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f755000b4a0 tx=0x7f755000b960 comp rx=0 tx=0).stop 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 shutdown_connections 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f75440a29f0 0x7f75440a4eb0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f755c199aa0 0x7f755c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f755c1086b0 0x7f755c19fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 --2- 172.21.15.46:0/3250921909 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f755c1035e0 0x7f755c19f8f0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 >> 172.21.15.46:0/3250921909 conn(0x7f755c0fee20 msgr2=0x7f755c1010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 shutdown_connections 2024-03-08T23:18:07.697 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.685+0000 7f7562200640 1 -- 172.21.15.46:0/3250921909 wait complete. 2024-03-08T23:18:07.704 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-08T23:18:07.705 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-08T23:18:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 -- 172.21.15.46:0/3666673426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c1035c0 msgr2=0x7f340c1039a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 --2- 172.21.15.46:0/3666673426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c1035c0 0x7f340c1039a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f33fc009900 tx=0x7f33fc02bc40 comp rx=0 tx=0).stop 2024-03-08T23:18:07.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 -- 172.21.15.46:0/3666673426 shutdown_connections 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 --2- 172.21.15.46:0/3666673426 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c110260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 --2- 172.21.15.46:0/3666673426 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c103f70 0x7f340c107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 --2- 172.21.15.46:0/3666673426 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c1035c0 0x7f340c1039a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.923+0000 7f340be00640 1 -- 172.21.15.46:0/3666673426 >> 172.21.15.46:0/3666673426 conn(0x7f340c0fee00 msgr2=0x7f340c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.924+0000 7f340be00640 1 -- 172.21.15.46:0/3666673426 shutdown_connections 2024-03-08T23:18:07.927 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.924+0000 7f340be00640 1 -- 172.21.15.46:0/3666673426 wait complete. 2024-03-08T23:18:07.929 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.925+0000 7f340be00640 1 Processor -- start 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.925+0000 7f340be00640 1 -- start start 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.925+0000 7f340be00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340be00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c19fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340be00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 0x7f340c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340be00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c113770 con 0x7f340c108690 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340be00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1135f0 con 0x7f340c199b50 2024-03-08T23:18:07.930 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340be00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1138f0 con 0x7f340c1035c0 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48692/0 (socket says 172.21.15.46:48692) 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 -- 172.21.15.46:0/4151651184 learned_addr learned my addr 172.21.15.46:0/4151651184 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f3402000640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c19fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340b400640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 0x7f340c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 -- 172.21.15.46:0/4151651184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 msgr2=0x7f340c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 0x7f340c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 -- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 msgr2=0x7f340c19fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.931 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c19fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340aa00640 1 -- 172.21.15.46:0/4151651184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f340c19a670 con 0x7f340c1035c0 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f340b400640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 0x7f340c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.926+0000 7f3402000640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c19fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f340aa00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005ab0 tx=0x7f33fc004360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f3403e00640 1 -- 172.21.15.46:0/4151651184 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc041070 con 0x7f340c1035c0 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f3403e00640 1 -- 172.21.15.46:0/4151651184 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33fc02cca0 con 0x7f340c1035c0 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f3403e00640 1 -- 172.21.15.46:0/4151651184 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc003d90 con 0x7f340c1035c0 2024-03-08T23:18:07.932 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f340c19a900 con 0x7f340c1035c0 2024-03-08T23:18:07.933 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.927+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f340c1a6b00 con 0x7f340c1035c0 2024-03-08T23:18:07.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f3403e00640 1 -- 172.21.15.46:0/4151651184 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f33fc044070 con 0x7f340c1035c0 2024-03-08T23:18:07.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 Processor -- start 2024-03-08T23:18:07.934 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 -- start start 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f70a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f665dd0 con 0x557d2f709d40 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f60ec70 con 0x557d2f70a660 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.931+0000 7f34147a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f661e70 con 0x557d2f713240 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f340b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f70a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.935 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f340aa00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48696/0 (socket says 172.21.15.46:48696) 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 -- 172.21.15.46:0/1023862657 learned_addr learned my addr 172.21.15.46:0/1023862657 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 -- 172.21.15.46:0/1023862657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 msgr2=0x557d2f712d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.936 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 -- 172.21.15.46:0/1023862657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 msgr2=0x557d2f70a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f70a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.937 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.932+0000 7f3402000640 1 -- 172.21.15.46:0/1023862657 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d2f6c82b0 con 0x557d2f713240 2024-03-08T23:18:07.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f3402000640 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f33f4010450 tx=0x7f33f400bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12c61fb34f7e594c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f3403e00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 0x7f33f00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f33fbe00640 1 -- 172.21.15.46:0/1023862657 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f4011760 con 0x557d2f713240 2024-03-08T23:18:07.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f33fbe00640 1 -- 172.21.15.46:0/1023862657 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f4011de0 con 0x557d2f713240 2024-03-08T23:18:07.941 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f33fbe00640 1 -- 172.21.15.46:0/1023862657 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f401a380 con 0x557d2f713240 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f3403e00640 1 -- 172.21.15.46:0/4151651184 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f33fc0f0b60 con 0x7f340c1035c0 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.933+0000 7f340b400640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 0x7f33f00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- 172.21.15.46:0/1023862657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 msgr2=0x557d2f715630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f33f4010450 tx=0x7f33f400bf90 comp rx=0 tx=0).stop 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f340b400640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 0x7f33f00a4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3404005fd0 tx=0x7f3404009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- 172.21.15.46:0/1023862657 shutdown_connections 2024-03-08T23:18:07.942 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f715630 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 --2- 172.21.15.46:0/1023862657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f70a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- 172.21.15.46:0/1023862657 >> 172.21.15.46:0/1023862657 conn(0x557d2f704470 msgr2=0x557d2f704850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- 172.21.15.46:0/1023862657 shutdown_connections 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- 172.21.15.46:0/1023862657 wait complete. 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.934+0000 7f34147a4740 1 -- start start 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 Processor -- start 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.943 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f70a660 0x557d2f70ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f710510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f716030 con 0x557d2f70a660 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f6220c0 con 0x557d2f713240 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d2f7161b0 con 0x557d2f709d40 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.935+0000 7f34147a4740 10 client.0 _ll_register_callbacks cb 0x557d2f65cec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f3402000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f70a660 0x557d2f70ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.944 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340b400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f710510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48712/0 (socket says 172.21.15.46:48712) 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 -- 172.21.15.46:0/1056278413 learned_addr learned my addr 172.21.15.46:0/1056278413 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 -- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 msgr2=0x557d2f710510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f710510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 -- 172.21.15.46:0/1056278413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f70a660 msgr2=0x557d2f70ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f70a660 0x557d2f70ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 -- 172.21.15.46:0/1056278413 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d2f37a3f0 con 0x557d2f709d40 2024-03-08T23:18:07.945 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f340aa00640 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f33fc006010 tx=0x7f33fc004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74fa2543d16251e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f33f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.192:3300/0 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.936+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc04b050 con 0x557d2f709d40 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mon.1 v2:172.21.15.192:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f33fc0ee0c0 con 0x557d2f709d40 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33f4011de0 con 0x557d2f709d40 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33fc0bcc30 con 0x557d2f709d40 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 10 client.27691 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557d2f4447d0 con 0x557d2f709d40 2024-03-08T23:18:07.946 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mon.1 v2:172.21.15.192:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33f4011de0 con 0x557d2f709d40 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mon.1 v2:172.21.15.192:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f33fc0d4070 con 0x557d2f709d40 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 10 client.27691 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 10 client.27691 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 20 client.27691 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f723310 0x557d2f725760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 4 client.27691 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.937+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d2f63a4f0 con 0x557d2f723310 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.938+0000 7f3402000640 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f723310 0x557d2f725760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.947 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.939+0000 7f3402000640 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f723310 0x557d2f725760 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=541c64ecd7dbd062 server_cookie=fc3ff6a1a5b605b6 in_seq=0 out_seq=0 2024-03-08T23:18:07.948 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.939+0000 7f33f8c00640 10 client.27691 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.939+0000 7f33f8c00640 1 -- 172.21.15.46:0/1056278413 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d2f63a4f0 con 0x557d2f723310 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.939+0000 7f33f8c00640 10 client.27691 handle_command_reply: tid=0 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 client.27691 shutdown 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 msgr2=0x557d2f712be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f33fc006010 tx=0x7f33fc004550 comp rx=0 tx=0).stop 2024-03-08T23:18:07.949 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 shutdown_connections 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f723310 0x557d2f725760 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f710510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f70a660 0x557d2f70ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f33f8c00640 0 client.27691 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 --2- 172.21.15.46:0/1056278413 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f709d40 0x557d2f712be0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 >> 172.21.15.46:0/1056278413 conn(0x557d2f704470 msgr2=0x557d2f7061c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 shutdown_connections 2024-03-08T23:18:07.950 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 1 -- 172.21.15.46:0/1056278413 wait complete. 2024-03-08T23:18:07.951 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.940+0000 7f34147a4740 20 client.27691 trim_cache size 0 max 16384 2024-03-08T23:18:07.955 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.953+0000 7f34147a4740 1 Processor -- start 2024-03-08T23:18:07.956 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 -- start start 2024-03-08T23:18:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.957 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f70d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1ae7c0 con 0x557d2f709d40 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1ac360 con 0x557d2f70a660 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f34147a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1aa420 con 0x557d2f713240 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f340aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f340aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55734/0 (socket says 172.21.15.46:55734) 2024-03-08T23:18:07.958 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f340aa00640 1 -- 172.21.15.46:0/8694868 learned_addr learned my addr 172.21.15.46:0/8694868 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f340b400640 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f70d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.954+0000 7f3402000640 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 -- 172.21.15.46:0/8694868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 msgr2=0x557d2f70d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f70d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 -- 172.21.15.46:0/8694868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 msgr2=0x557d2f712b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.959 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 -- 172.21.15.46:0/8694868 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d2f72a920 con 0x557d2f709d40 2024-03-08T23:18:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f340aa00640 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f33fc02e040 tx=0x7f33fc02bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566f23e35b7f2985 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f33fb400640 1 -- 172.21.15.46:0/8694868 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc04b070 con 0x557d2f709d40 2024-03-08T23:18:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f33fb400640 1 -- 172.21.15.46:0/8694868 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33fc004c80 con 0x557d2f709d40 2024-03-08T23:18:07.960 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.955+0000 7f33fb400640 1 -- 172.21.15.46:0/8694868 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc0bc7c0 con 0x557d2f709d40 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- 172.21.15.46:0/8694868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 msgr2=0x557d2f712630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f33fc02e040 tx=0x7f33fc02bc40 comp rx=0 tx=0).stop 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- 172.21.15.46:0/8694868 shutdown_connections 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f713240 0x557d2f70d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f70a660 0x557d2f712b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 --2- 172.21.15.46:0/8694868 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f712630 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.964 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- 172.21.15.46:0/8694868 >> 172.21.15.46:0/8694868 conn(0x557d2f704470 msgr2=0x557d2f704850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- 172.21.15.46:0/8694868 shutdown_connections 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- 172.21.15.46:0/8694868 wait complete. 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.956+0000 7f34147a4740 1 -- start start 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 Processor -- start 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 0x557d2f7187f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f718d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1aaa20 con 0x557d2f709d40 2024-03-08T23:18:07.965 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1abd00 con 0x557d2f713240 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f340c1ab040 con 0x557d2f70a660 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f34147a4740 10 client.0 _ll_register_callbacks cb 0x7f33f400bae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f340aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f3402000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 0x557d2f7187f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f340aa00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55740/0 (socket says 172.21.15.46:55740) 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f340aa00640 1 -- 172.21.15.46:0/3824621060 learned_addr learned my addr 172.21.15.46:0/3824621060 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f3402000640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 0x557d2f7187f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48730/0 (socket says 172.21.15.46:48730) 2024-03-08T23:18:07.966 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.957+0000 7f340b400640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f718d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 -- 172.21.15.46:0/3824621060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 msgr2=0x557d2f7187f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 0x557d2f7187f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 -- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 msgr2=0x557d2f718d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f718d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 -- 172.21.15.46:0/3824621060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d2f719270 con 0x557d2f709d40 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f340aa00640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005820 tx=0x7f33fc02cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb36b1319aa6de8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f33f8c00640 10 client.0 ms_handle_connect on v2:172.21.15.46:3300/0 2024-03-08T23:18:07.967 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33fc0ca060 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mon.0 v2:172.21.15.46:3300/0 2 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f33fc0c5170 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.958+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33fc004c80 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33fc04b040 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 10 client.17592 fetch_fsmap learned FSMap version 31 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557d2f719270 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mon.0 v2:172.21.15.46:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33fc004c80 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mon.0 v2:172.21.15.46:3300/0 5 ==== fsmap(e 31) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f33fc0d4070 con 0x557d2f709d40 2024-03-08T23:18:07.968 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 10 client.17592 fetch_fsmap finished waiting for FSMap version 31 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 10 client.17592 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 20 client.17592 populate_metadata read hostname 'smithi046' 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f77a740 0x557d2f77cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 4 client.17592 mds_command: new command op to 24578 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.959+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 --> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557d2f673310 con 0x557d2f77a740 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.960+0000 7f3402000640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f77a740 0x557d2f77cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.960+0000 7f3402000640 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f77a740 0x557d2f77cb30 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58958dafa2bd6657 server_cookie=cc4072b28555e119 in_seq=0 out_seq=0 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.960+0000 7f33f8c00640 10 client.17592 ms_handle_connect on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:07.969 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.960+0000 7f33f8c00640 1 -- 172.21.15.46:0/3824621060 <== mds.0 v2:172.21.15.62:6838/1938633866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557d2f673310 con 0x557d2f77a740 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.960+0000 7f33f8c00640 10 client.17592 handle_command_reply: tid=0 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 client.17592 shutdown 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 msgr2=0x557d2f71a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005820 tx=0x7f33fc02cf90 comp rx=0 tx=0).stop 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 shutdown_connections 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:6838/1938633866,v1:172.21.15.62:6839/1938633866] conn(0x557d2f77a740 0x557d2f77cb30 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x557d2f713240 0x557d2f718d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.970 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x557d2f70a660 0x557d2f7187f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 --2- 172.21.15.46:0/3824621060 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x557d2f709d40 0x557d2f71a450 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f33f8c00640 0 client.17592 ms_handle_reset on v2:172.21.15.62:6838/1938633866 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 >> 172.21.15.46:0/3824621060 conn(0x557d2f704470 msgr2=0x557d2f70e840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 shutdown_connections 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 1 -- 172.21.15.46:0/3824621060 wait complete. 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.961+0000 7f34147a4740 20 client.17592 trim_cache size 0 max 16384 2024-03-08T23:18:07.971 INFO:teuthology.orchestra.run.smithi046.stdout:[] 2024-03-08T23:18:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 msgr2=0x7f33f00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.972 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 0x7f33f00a4eb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3404005fd0 tx=0x7f3404009500 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 msgr2=0x7f340c19f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f33fc005ab0 tx=0x7f33fc004360 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 shutdown_connections 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f33f00a29f0 0x7f33f00a4eb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f340c199b50 0x7f340c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f340c108690 0x7f340c19fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 --2- 172.21.15.46:0/4151651184 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f340c1035c0 0x7f340c19f950 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:07.973 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 >> 172.21.15.46:0/4151651184 conn(0x7f340c0fee00 msgr2=0x7f340c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:07.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 shutdown_connections 2024-03-08T23:18:07.974 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:07.962+0000 7f340be00640 1 -- 172.21.15.46:0/4151651184 wait complete. 2024-03-08T23:18:07.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:07 smithi046 ceph-mon[31859]: pgmap v321: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:07.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:07 smithi046 ceph-mon[31859]: osdmap e103: 12 total, 12 up, 12 in 2024-03-08T23:18:07.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:07 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2453872855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:08.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:07 smithi062 ceph-mon[35761]: pgmap v321: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:07 smithi062 ceph-mon[35761]: osdmap e103: 12 total, 12 up, 12 in 2024-03-08T23:18:08.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:07 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2453872855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:08.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:07 smithi192 ceph-mon[36700]: pgmap v321: 129 pgs: 129 active+clean; 78 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-03-08T23:18:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:07 smithi192 ceph-mon[36700]: osdmap e103: 12 total, 12 up, 12 in 2024-03-08T23:18:08.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:07 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2453872855' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:08.984 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-08T23:18:08.985 INFO:tasks.fwd_scrub:done joining 2024-03-08T23:18:08.985 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-08T23:18:08.997 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:18:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 -- 172.21.15.46:0/1206498024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0440103520 msgr2=0x7f0440103900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.231 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 --2- 172.21.15.46:0/1206498024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0440103520 0x7f0440103900 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0430009ca0 tx=0x7f043002c100 comp rx=0 tx=0).stop 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 -- 172.21.15.46:0/1206498024 shutdown_connections 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 --2- 172.21.15.46:0/1206498024 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f04401085f0 0x7f04401101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 --2- 172.21.15.46:0/1206498024 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440103ed0 0x7f0440107f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 --2- 172.21.15.46:0/1206498024 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0440103520 0x7f0440103900 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 -- 172.21.15.46:0/1206498024 >> 172.21.15.46:0/1206498024 conn(0x7f04400fed60 msgr2=0x7f0440101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 -- 172.21.15.46:0/1206498024 shutdown_connections 2024-03-08T23:18:09.232 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.228+0000 7f0445c00640 1 -- 172.21.15.46:0/1206498024 wait complete. 2024-03-08T23:18:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.229+0000 7f0445c00640 1 Processor -- start 2024-03-08T23:18:09.233 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.229+0000 7f0445c00640 1 -- start start 2024-03-08T23:18:09.234 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0440103520 0x7f044019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04401085f0 0x7f044019fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04401136b0 con 0x7f0440199ab0 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440113530 con 0x7f0440103520 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445c00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440113830 con 0x7f04401085f0 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.235 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55760/0 (socket says 172.21.15.46:55760) 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 -- 172.21.15.46:0/1805029563 learned_addr learned my addr 172.21.15.46:0/1805029563 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f043fe00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0440103520 0x7f044019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 -- 172.21.15.46:0/1805029563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04401085f0 msgr2=0x7f044019fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04401085f0 0x7f044019fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 -- 172.21.15.46:0/1805029563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0440103520 msgr2=0x7f044019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0440103520 0x7f044019f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.230+0000 7f0445200640 1 -- 172.21.15.46:0/1805029563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044019a5d0 con 0x7f0440199ab0 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f0445200640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f043800e970 tx=0x7f043800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:09.236 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800ccc0 con 0x7f0440199ab0 2024-03-08T23:18:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044019a8c0 con 0x7f0440199ab0 2024-03-08T23:18:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f043800ce60 con 0x7f0440199ab0 2024-03-08T23:18:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0438010450 con 0x7f0440199ab0 2024-03-08T23:18:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.231+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04401a6b60 con 0x7f0440199ab0 2024-03-08T23:18:09.237 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.232+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0440068450 con 0x7f0440199ab0 2024-03-08T23:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.234+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0438005020 con 0x7f0440199ab0 2024-03-08T23:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.235+0000 7f043d600640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 0x7f04240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.236+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f04380cf740 con 0x7f0440199ab0 2024-03-08T23:18:09.238 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.236+0000 7f043fe00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 0x7f04240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.239 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.237+0000 7f043fe00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 0x7f04240a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0430000f80 tx=0x7f0430009360 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:09.242 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.240+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f043809c030 con 0x7f0440199ab0 2024-03-08T23:18:09.446 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.444+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0440103ed0 con 0x7f0440199ab0 2024-03-08T23:18:09.449 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.447+0000 7f043d600640 1 -- 172.21.15.46:0/1805029563 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v103) v1 ==== 74+0+25393 (secure 0 0 0) 0x7f043809b930 con 0x7f0440199ab0 2024-03-08T23:18:09.449 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:09.450 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":103,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:18:06.846879+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6801/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/1737358861":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1465757142":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1832130111":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1962446760":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:18:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 msgr2=0x7f04240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.452 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 0x7f04240a4e30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0430000f80 tx=0x7f0430009360 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 msgr2=0x7f0440199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f043800e970 tx=0x7f043800ed70 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 shutdown_connections 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f04240a2970 0x7f04240a4e30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0440199ab0 0x7f0440199f60 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04401085f0 0x7f044019fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 --2- 172.21.15.46:0/1805029563 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0440103520 0x7f044019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.453 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 >> 172.21.15.46:0/1805029563 conn(0x7f04400fed60 msgr2=0x7f04401010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 shutdown_connections 2024-03-08T23:18:09.454 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.450+0000 7f0445c00640 1 -- 172.21.15.46:0/1805029563 wait complete. 2024-03-08T23:18:09.472 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:18:09.473 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:18:09.706 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 -- 172.21.15.46:0/426051854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 msgr2=0x7fe8a8111bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 --2- 172.21.15.46:0/426051854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a8111bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe890009890 tx=0x7fe89002bb70 comp rx=0 tx=0).stop 2024-03-08T23:18:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 -- 172.21.15.46:0/426051854 shutdown_connections 2024-03-08T23:18:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 --2- 172.21.15.46:0/426051854 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 0x7fe8a8114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 --2- 172.21.15.46:0/426051854 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a8111bf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.707 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 --2- 172.21.15.46:0/426051854 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a8075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 -- 172.21.15.46:0/426051854 >> 172.21.15.46:0/426051854 conn(0x7fe8a80fefa0 msgr2=0x7fe8a81013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 -- 172.21.15.46:0/426051854 shutdown_connections 2024-03-08T23:18:09.708 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.704+0000 7fe8ae600640 1 -- 172.21.15.46:0/426051854 wait complete. 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.705+0000 7fe8ae600640 1 Processor -- start 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 -- start start 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a819e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 0x7fe8a81a2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.710 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8117c90 con 0x7fe8a8075820 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8117b10 con 0x7fe8a8112130 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8ae600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a8117e10 con 0x7fe8a8076140 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8a7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a819e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8a6a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48754/0 (socket says 172.21.15.46:48754) 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8a6a00640 1 -- 172.21.15.46:0/4085052089 learned_addr learned my addr 172.21.15.46:0/4085052089 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:09.711 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.706+0000 7fe8a7400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a819e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:55780/0 (socket says 172.21.15.46:55780) 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a7e00640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 0x7fe8a81a2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 -- 172.21.15.46:0/4085052089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 msgr2=0x7fe8a81a2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 0x7fe8a81a2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 -- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 msgr2=0x7fe8a819e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a819e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 -- 172.21.15.46:0/4085052089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a81a3240 con 0x7fe8a8076140 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe8a6a00640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe890000c00 tx=0x7fe890004550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:09.712 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.707+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890041020 con 0x7fe8a8076140 2024-03-08T23:18:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.708+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe89002c9f0 con 0x7fe8a8076140 2024-03-08T23:18:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.708+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a81a34d0 con 0x7fe8a8076140 2024-03-08T23:18:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.708+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890039040 con 0x7fe8a8076140 2024-03-08T23:18:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.708+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a81aadc0 con 0x7fe8a8076140 2024-03-08T23:18:09.713 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.709+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a8068530 con 0x7fe8a8076140 2024-03-08T23:18:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.711+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe890044020 con 0x7fe8a8076140 2024-03-08T23:18:09.714 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.712+0000 7fe89fe00640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 0x7fe88c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.713+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fe8900f0da0 con 0x7fe8a8076140 2024-03-08T23:18:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.713+0000 7fe8a7400640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 0x7fe88c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:09.715 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.713+0000 7fe8a7400640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 0x7fe88c0a4eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe898005fd0 tx=0x7fe898010040 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:09.716 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.715+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe89004b020 con 0x7fe8a8076140 2024-03-08T23:18:09.920 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.918+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8a810cd30 con 0x7fe8a8076140 2024-03-08T23:18:09.921 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.919+0000 7fe89fe00640 1 -- 172.21.15.46:0/4085052089 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v103) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8900bd820 con 0x7fe8a8076140 2024-03-08T23:18:09.921 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:18:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 msgr2=0x7fe88c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 0x7fe88c0a4eb0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe898005fd0 tx=0x7fe898010040 comp rx=0 tx=0).stop 2024-03-08T23:18:09.924 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 msgr2=0x7fe8a819e790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe890000c00 tx=0x7fe890004550 comp rx=0 tx=0).stop 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 shutdown_connections 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fe88c0a29f0 0x7fe88c0a4eb0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fe8a8112130 0x7fe8a81a2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe8a8076140 0x7fe8a819e790 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 --2- 172.21.15.46:0/4085052089 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fe8a8075820 0x7fe8a819e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 >> 172.21.15.46:0/4085052089 conn(0x7fe8a80fefa0 msgr2=0x7fe8a8101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:09.925 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 shutdown_connections 2024-03-08T23:18:09.926 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:09.922+0000 7fe8ae600640 1 -- 172.21.15.46:0/4085052089 wait complete. 2024-03-08T23:18:09.945 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:18:10.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:09 smithi062 ceph-mon[35761]: pgmap v323: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:10.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:09 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1805029563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 -- 172.21.15.46:0/3667344104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 msgr2=0x7fa1b8103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 --2- 172.21.15.46:0/3667344104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8103930 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa1a4009890 tx=0x7fa1a402bb70 comp rx=0 tx=0).stop 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 -- 172.21.15.46:0/3667344104 shutdown_connections 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 --2- 172.21.15.46:0/3667344104 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8103e70 0x7fa1b8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 --2- 172.21.15.46:0/3667344104 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8103930 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.176 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 --2- 172.21.15.46:0/3667344104 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8109570 0x7fa1b8109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 -- 172.21.15.46:0/3667344104 >> 172.21.15.46:0/3667344104 conn(0x7fa1b80fef60 msgr2=0x7fa1b8101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 -- 172.21.15.46:0/3667344104 shutdown_connections 2024-03-08T23:18:10.177 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.173+0000 7fa1be000640 1 -- 172.21.15.46:0/3667344104 wait complete. 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.174+0000 7fa1be000640 1 Processor -- start 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.174+0000 7fa1be000640 1 -- start start 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 0x7fa1b819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 0x7fa1b819e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.179 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8113850 con 0x7fa1b8103e70 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81136d0 con 0x7fa1b8109570 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1be000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81139d0 con 0x7fa1b81034b0 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 0x7fa1b819e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1b6a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 0x7fa1b819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1b7e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 0x7fa1b819e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:33012/0 (socket says 172.21.15.46:33012) 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1b7e00640 1 -- 172.21.15.46:0/2128836834 learned_addr learned my addr 172.21.15.46:0/2128836834 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.175+0000 7fa1b7400640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 -- 172.21.15.46:0/2128836834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 msgr2=0x7fa1b8199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.180 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 -- 172.21.15.46:0/2128836834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 msgr2=0x7fa1b819e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 0x7fa1b819e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 -- 172.21.15.46:0/2128836834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b819ed20 con 0x7fa1b8103e70 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1b6a00640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 0x7fa1b819a270 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40057c0 tx=0x7fa1a4004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4041020 con 0x7fa1b8103e70 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1a402c9f0 con 0x7fa1b8103e70 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b819efb0 con 0x7fa1b8103e70 2024-03-08T23:18:10.181 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.176+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a4039040 con 0x7fa1b8103e70 2024-03-08T23:18:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.177+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b81a68a0 con 0x7fa1b8103e70 2024-03-08T23:18:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.178+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b8068530 con 0x7fa1b8103e70 2024-03-08T23:18:10.182 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.180+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa1a4044020 con 0x7fa1b8103e70 2024-03-08T23:18:10.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.181+0000 7fa1afe00640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 0x7fa19c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.182+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fa1a40f0a50 con 0x7fa1b8103e70 2024-03-08T23:18:10.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.182+0000 7fa1b7400640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 0x7fa19c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.184 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.183+0000 7fa1b7400640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 0x7fa19c0a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8005fd0 tx=0x7fa1a8009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:10.188 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.186+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa1a404b020 con 0x7fa1b8103e70 2024-03-08T23:18:10.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:09 smithi046 ceph-mon[31859]: pgmap v323: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:10.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:09 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1805029563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:10.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:09 smithi192 ceph-mon[36700]: pgmap v323: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:10.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:09 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1805029563' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:10.402 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.399+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa1b810b330 con 0x7fa1b8103e70 2024-03-08T23:18:10.403 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.400+0000 7fa1afe00640 1 -- 172.21.15.46:0/2128836834 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v103) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa1a40bd450 con 0x7fa1b8103e70 2024-03-08T23:18:10.404 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:18:10.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 msgr2=0x7fa19c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 0x7fa19c0a4e30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8005fd0 tx=0x7fa1a8009260 comp rx=0 tx=0).stop 2024-03-08T23:18:10.406 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 msgr2=0x7fa1b819a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 0x7fa1b819a270 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40057c0 tx=0x7fa1a4004360 comp rx=0 tx=0).stop 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 shutdown_connections 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fa19c0a2970 0x7fa19c0a4e30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fa1b8109570 0x7fa1b819e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fa1b8103e70 0x7fa1b819a270 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 --2- 172.21.15.46:0/2128836834 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa1b81034b0 0x7fa1b8199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 >> 172.21.15.46:0/2128836834 conn(0x7fa1b80fef60 msgr2=0x7fa1b8100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:10.407 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 shutdown_connections 2024-03-08T23:18:10.408 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.403+0000 7fa1be000640 1 -- 172.21.15.46:0/2128836834 wait complete. 2024-03-08T23:18:10.425 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:18:10.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 -- 172.21.15.46:0/1475707863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474103e70 msgr2=0x7f7474110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 --2- 172.21.15.46:0/1475707863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474103e70 0x7f7474110330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f746c009cd0 tx=0x7f746c02c0e0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 -- 172.21.15.46:0/1475707863 shutdown_connections 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 --2- 172.21.15.46:0/1475707863 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474103e70 0x7f7474110330 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 --2- 172.21.15.46:0/1475707863 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 --2- 172.21.15.46:0/1475707863 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474109570 0x7f7474109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.642+0000 7f747a000640 1 -- 172.21.15.46:0/1475707863 >> 172.21.15.46:0/1475707863 conn(0x7f74740fefa0 msgr2=0x7f74741013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.643+0000 7f747a000640 1 -- 172.21.15.46:0/1475707863 shutdown_connections 2024-03-08T23:18:10.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.643+0000 7f747a000640 1 -- 172.21.15.46:0/1475707863 wait complete. 2024-03-08T23:18:10.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 Processor -- start 2024-03-08T23:18:10.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 -- start start 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 0x7f747419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109570 0x7f747419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74741138b0 con 0x7f7474103e70 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7474113730 con 0x7f7474109570 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f747a000640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7474113a30 con 0x7f74741034b0 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f7472a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 0x7f747419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f7473400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f7473400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48790/0 (socket says 172.21.15.46:48790) 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.644+0000 7f7473400640 1 -- 172.21.15.46:0/2096374428 learned_addr learned my addr 172.21.15.46:0/2096374428 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7473e00640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109570 0x7f747419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 -- 172.21.15.46:0/2096374428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 msgr2=0x7f7474199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 -- 172.21.15.46:0/2096374428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109570 msgr2=0x7f747419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109570 0x7f747419e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 -- 172.21.15.46:0/2096374428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747419ed60 con 0x7f7474103e70 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7473400640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.645+0000 7f7472a00640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 0x7f747419a2b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b3f0 tx=0x7f745c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.646+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c015020 con 0x7f7474103e70 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.646+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747419f050 con 0x7f7474103e70 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.646+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f745c011d10 con 0x7f7474103e70 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.646+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745c00f2e0 con 0x7f7474103e70 2024-03-08T23:18:10.652 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.646+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74741a68e0 con 0x7f7474103e70 2024-03-08T23:18:10.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.647+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7474068530 con 0x7f7474103e70 2024-03-08T23:18:10.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.649+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f745c018020 con 0x7f7474103e70 2024-03-08T23:18:10.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.650+0000 7f746be00640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 0x7f74580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:10.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.650+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f745c0ce950 con 0x7f7474103e70 2024-03-08T23:18:10.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.651+0000 7f7473400640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 0x7f74580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:10.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.651+0000 7f7473400640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 0x7f74580a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7464011550 tx=0x7f7464005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:10.658 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.655+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f745c01f020 con 0x7f7474103e70 2024-03-08T23:18:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.878+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f747402ce70 con 0x7f7474103e70 2024-03-08T23:18:10.882 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.879+0000 7f746be00640 1 -- 172.21.15.46:0/2096374428 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v103) v1 ==== 100+0+11 (secure 0 0 0) 0x7f745c09b350 con 0x7f7474103e70 2024-03-08T23:18:10.883 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:18:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 msgr2=0x7f74580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.885 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 0x7f74580a4e30 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f7464011550 tx=0x7f7464005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 msgr2=0x7f747419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 0x7f747419a2b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f745c00b3f0 tx=0x7f745c00b8b0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 shutdown_connections 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f74580a2970 0x7f74580a4e30 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f7474109570 0x7f747419e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f7474103e70 0x7f747419a2b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.886 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 --2- 172.21.15.46:0/2096374428 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f74741034b0 0x7f7474199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 >> 172.21.15.46:0/2096374428 conn(0x7f74740fefa0 msgr2=0x7f74741008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 shutdown_connections 2024-03-08T23:18:10.887 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:10.882+0000 7f747a000640 1 -- 172.21.15.46:0/2096374428 wait complete. 2024-03-08T23:18:10.903 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:18:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.103+0000 7f82d6400640 1 -- 172.21.15.46:0/3893906821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0112130 msgr2=0x7f82d0114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.103+0000 7f82d6400640 1 --2- 172.21.15.46:0/3893906821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0112130 0x7f82d0114520 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009c10 tx=0x7f82c802bee0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.107 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 -- 172.21.15.46:0/3893906821 shutdown_connections 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 --2- 172.21.15.46:0/3893906821 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0112130 0x7f82d0114520 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 --2- 172.21.15.46:0/3893906821 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0076140 0x7f82d0111bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 --2- 172.21.15.46:0/3893906821 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0075820 0x7f82d0075c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 -- 172.21.15.46:0/3893906821 >> 172.21.15.46:0/3893906821 conn(0x7f82d00fefa0 msgr2=0x7f82d01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 -- 172.21.15.46:0/3893906821 shutdown_connections 2024-03-08T23:18:11.108 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.104+0000 7f82d6400640 1 -- 172.21.15.46:0/3893906821 wait complete. 2024-03-08T23:18:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 Processor -- start 2024-03-08T23:18:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 -- start start 2024-03-08T23:18:11.110 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 0x7f82d019e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0112130 0x7f82d01a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0117ce0 con 0x7f82d0112130 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0117b60 con 0x7f82d0076140 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82d6400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d0117e60 con 0x7f82d0075820 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82cf400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 0x7f82d019e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82cea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82cf400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 0x7f82d019e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48800/0 (socket says 172.21.15.46:48800) 2024-03-08T23:18:11.111 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82cf400640 1 -- 172.21.15.46:0/1648159414 learned_addr learned my addr 172.21.15.46:0/1648159414 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.105+0000 7f82cea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:33052/0 (socket says 172.21.15.46:33052) 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cfe00640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0112130 0x7f82d01a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 -- 172.21.15.46:0/1648159414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 msgr2=0x7f82d019e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 0x7f82d019e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 -- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0112130 msgr2=0x7f82d01a2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0112130 0x7f82d01a2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 -- 172.21.15.46:0/1648159414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d01a31e0 con 0x7f82d0076140 2024-03-08T23:18:11.112 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.106+0000 7f82cea00640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d720 tx=0x7f82bc00dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.107+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc017070 con 0x7f82d0076140 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.107+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82bc00de30 con 0x7f82d0076140 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.107+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0047a0 con 0x7f82d0076140 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.107+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d01a34d0 con 0x7f82d0076140 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.107+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d01aad60 con 0x7f82d0076140 2024-03-08T23:18:11.113 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.108+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d0068530 con 0x7f82d0076140 2024-03-08T23:18:11.114 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.110+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f82bc0099d0 con 0x7f82d0076140 2024-03-08T23:18:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.112+0000 7f82c7e00640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 0x7f82b40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.112+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f82bc0cedb0 con 0x7f82d0076140 2024-03-08T23:18:11.115 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.112+0000 7f82cf400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 0x7f82b40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.116 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.113+0000 7f82cf400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 0x7f82b40a4eb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f82c0005c80 tx=0x7f82c0009260 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:11.119 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.116+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f82bc0d3070 con 0x7f82d0076140 2024-03-08T23:18:11.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4085052089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:18:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2128836834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:10 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4085052089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:18:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2128836834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:10 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4085052089' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2128836834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:10 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:11.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.324+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f82d002ce70 con 0x7f82d0076140 2024-03-08T23:18:11.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.325+0000 7f82c7e00640 1 -- 172.21.15.46:0/1648159414 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v103) v1 ==== 103+0+11 (secure 0 0 0) 0x7f82bc020020 con 0x7f82d0076140 2024-03-08T23:18:11.328 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:18:11.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 msgr2=0x7f82b40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 0x7f82b40a4eb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f82c0005c80 tx=0x7f82c0009260 comp rx=0 tx=0).stop 2024-03-08T23:18:11.332 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 msgr2=0x7f82d019e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d720 tx=0x7f82bc00dbe0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 shutdown_connections 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f82b40a29f0 0x7f82b40a4eb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f82d0112130 0x7f82d01a2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f82d0076140 0x7f82d019e780 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 --2- 172.21.15.46:0/1648159414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f82d0075820 0x7f82d019e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 >> 172.21.15.46:0/1648159414 conn(0x7f82d00fefa0 msgr2=0x7f82d0101390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:11.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 shutdown_connections 2024-03-08T23:18:11.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.329+0000 7f82d6400640 1 -- 172.21.15.46:0/1648159414 wait complete. 2024-03-08T23:18:11.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 -- 172.21.15.46:0/1814058503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 msgr2=0x7f37c41101e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 --2- 172.21.15.46:0/1814058503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c41101e0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009be0 tx=0x7f37bc02bef0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 -- 172.21.15.46:0/1814058503 shutdown_connections 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 --2- 172.21.15.46:0/1814058503 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c41101e0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 --2- 172.21.15.46:0/1814058503 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c4103ef0 0x7f37c4107f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 --2- 172.21.15.46:0/1814058503 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 -- 172.21.15.46:0/1814058503 >> 172.21.15.46:0/1814058503 conn(0x7f37c40fed80 msgr2=0x7f37c41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:11.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 -- 172.21.15.46:0/1814058503 shutdown_connections 2024-03-08T23:18:11.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.566+0000 7f37c3e00640 1 -- 172.21.15.46:0/1814058503 wait complete. 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.567+0000 7f37c3e00640 1 Processor -- start 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.567+0000 7f37c3e00640 1 -- start start 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 0x7f37c4199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4113720 con 0x7f37c41999a0 2024-03-08T23:18:11.573 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41135a0 con 0x7f37c4108610 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41138a0 con 0x7f37c4103540 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:48820/0 (socket says 172.21.15.46:48820) 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c2a00640 1 -- 172.21.15.46:0/2375809949 learned_addr learned my addr 172.21.15.46:0/2375809949 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37ba000640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.568+0000 7f37c3400640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 0x7f37c4199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 -- 172.21.15.46:0/2375809949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 msgr2=0x7f37c419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 -- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 msgr2=0x7f37c4199e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 0x7f37c4199e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 -- 172.21.15.46:0/2375809949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c419a4c0 con 0x7f37c4103540 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c3400640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 0x7f37c4199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c2a00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e9f0 tx=0x7f37b400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b400ccc0 con 0x7f37c4103540 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37b4005500 con 0x7f37c4103540 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37b4010690 con 0x7f37c4103540 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c419a7b0 con 0x7f37c4103540 2024-03-08T23:18:11.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.569+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c41a6a00 con 0x7f37c4103540 2024-03-08T23:18:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.570+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c4068450 con 0x7f37c4103540 2024-03-08T23:18:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.572+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f37b40046c0 con 0x7f37c4103540 2024-03-08T23:18:11.576 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.573+0000 7f37bbe00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 0x7f37a40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:11.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.574+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f37b40cf570 con 0x7f37c4103540 2024-03-08T23:18:11.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.574+0000 7f37ba000640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 0x7f37a40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:11.577 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.574+0000 7f37ba000640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 0x7f37a40a4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f37c4102480 tx=0x7f37ac005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:11.580 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.577+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f37b409bec0 con 0x7f37c4103540 2024-03-08T23:18:11.843 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.840+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f37c419aed0 con 0x7f37c4103540 2024-03-08T23:18:11.852 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.849+0000 7f37bbe00640 1 -- 172.21.15.46:0/2375809949 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) v1 ==== 81+0+740363 (secure 0 0 0) 0x7f37b409b7c0 con 0x7f37c4103540 2024-03-08T23:18:11.863 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "epoch": 27, 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "active_gid": 25559, 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "active_name": "x", 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "active_addrs": { 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:11.864 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6800", 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2505290629 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v1", 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:6801", 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2505290629 2024-03-08T23:18:11.865 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: "active_addr": "172.21.15.46:6801/2505290629", 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: "active_change": "2024-03-08T23:07:46.417572+0000", 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: "active_mgr_features": 4540701547738038271, 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: "available": true, 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: "standbys": [ 2024-03-08T23:18:11.866 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "gid": 25556, 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "y", 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr_features": 4540701547738038271, 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:11.867 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.868 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:11.869 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.870 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.871 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.872 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.873 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:11.874 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.875 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:11.876 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.877 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-08T23:18:11.878 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-08T23:18:11.879 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.880 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.881 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.882 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.883 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.884 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-08T23:18:11.885 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.886 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.887 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.888 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.889 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.890 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.891 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.892 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:11.893 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.894 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-08T23:18:11.895 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-08T23:18:11.896 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.897 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:11.898 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-08T23:18:11.899 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.900 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.901 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:11.902 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-08T23:18:11.903 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.904 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.905 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:11.906 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:11.907 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.908 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.909 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.910 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.911 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.912 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.913 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.914 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.915 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.916 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.918 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.919 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.920 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.921 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.922 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.923 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.924 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.925 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.926 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:11.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:11.928 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.929 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.930 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.931 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.932 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-08T23:18:11.933 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.934 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.935 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.936 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-08T23:18:11.937 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.938 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.939 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.940 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.941 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-08T23:18:11.942 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.943 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.944 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.945 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.946 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.948 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-08T23:18:11.949 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.950 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.951 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.952 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-08T23:18:11.953 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.955 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-08T23:18:11.956 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.957 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.958 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.959 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.960 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-08T23:18:11.961 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.962 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-08T23:18:11.963 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.964 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.965 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.967 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-08T23:18:11.968 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.969 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.970 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-08T23:18:11.971 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.972 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.973 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.974 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.975 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-08T23:18:11.976 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.977 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.978 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-08T23:18:11.979 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-08T23:18:11.980 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.981 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.983 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-08T23:18:11.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.985 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.986 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.987 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.988 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:11.990 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.991 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-08T23:18:11.992 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.993 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:11.994 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-08T23:18:11.995 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.996 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:11.997 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:11.998 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:11.999 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.000 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.001 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.002 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-08T23:18:12.003 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.004 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.005 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.007 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.008 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.009 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-08T23:18:12.010 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-08T23:18:12.011 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-08T23:18:12.012 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.013 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.015 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-08T23:18:12.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.017 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-08T23:18:12.018 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-08T23:18:12.019 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.020 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.021 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.022 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.023 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.024 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.025 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.026 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.027 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.028 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.029 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.030 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.031 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.032 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.033 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.034 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.035 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.036 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.037 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.038 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.039 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.040 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.041 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.042 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.043 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.044 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.045 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.046 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.047 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.048 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.049 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.050 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.051 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-08T23:18:12.052 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.053 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.054 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.055 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.056 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.057 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.058 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.059 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.060 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.061 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.062 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.063 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-08T23:18:12.064 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.065 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.066 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.067 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.068 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.069 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.070 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-08T23:18:12.071 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.072 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.073 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.074 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.075 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-08T23:18:12.076 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.077 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.078 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.080 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-08T23:18:12.081 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.082 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.083 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.084 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.085 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.086 INFO:teuthology.orchestra.run.smithi046.stdout: "MULTICLUSTER_CONFIG": { 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "{}", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.087 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-08T23:18:12.088 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.089 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.090 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.091 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.092 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.093 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.094 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-08T23:18:12.095 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.096 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.097 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.098 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.099 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.100 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.101 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.102 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.103 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.104 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.105 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.106 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-08T23:18:12.107 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.108 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.109 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.110 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-08T23:18:12.111 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.112 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.113 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.114 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.115 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.116 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.118 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "UNSAFE_TLS_v1_2": { 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-08T23:18:12.119 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.120 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.121 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.122 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.123 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-08T23:18:12.124 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.125 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.126 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.127 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.128 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-08T23:18:12.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.130 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-08T23:18:12.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.132 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.133 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.134 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.135 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.136 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.138 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.139 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.140 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.141 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.142 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-08T23:18:12.143 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.144 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.145 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.146 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:12.147 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.148 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.149 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.150 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.151 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.152 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.153 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.154 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.155 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:12.156 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-08T23:18:12.157 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.158 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.159 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.160 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.161 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.162 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.163 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.164 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.165 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.166 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.167 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.168 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.169 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.170 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.171 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-08T23:18:12.172 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-08T23:18:12.173 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.174 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.175 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-08T23:18:12.176 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.177 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.178 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.179 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.180 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.181 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.182 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.183 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.184 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.185 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.186 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.187 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.188 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.189 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.190 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.191 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.192 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-08T23:18:12.193 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.194 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.195 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.196 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-08T23:18:12.197 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.198 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-08T23:18:12.199 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.200 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.201 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.202 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-08T23:18:12.203 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.204 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.205 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.206 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.207 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.208 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.209 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.210 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.211 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.212 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.213 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.214 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.215 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-08T23:18:12.217 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.218 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:12.219 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-08T23:18:12.220 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.221 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.222 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:12.223 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-08T23:18:12.224 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.225 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.226 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.227 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.228 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.229 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.230 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.231 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.232 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.234 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.235 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.236 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.237 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.238 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.239 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.240 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.241 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.242 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.243 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.244 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.245 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-08T23:18:12.246 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.247 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.248 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.249 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.250 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.251 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.252 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.253 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.254 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.255 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.256 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.257 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.258 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.259 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.260 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.261 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.262 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.263 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.264 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.265 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.266 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.267 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.268 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.269 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-08T23:18:12.270 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.271 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.272 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.273 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.274 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.275 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-08T23:18:12.276 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-08T23:18:12.276 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.276 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.276 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.277 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.278 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.279 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.280 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.281 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.282 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.283 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.284 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.285 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.286 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.287 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.288 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.289 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.290 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.291 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.292 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.293 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.294 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.295 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.296 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.297 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.298 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.299 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.300 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.301 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.302 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.303 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.304 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.305 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.306 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.307 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.308 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.309 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.310 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.311 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.312 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.313 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.314 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.315 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-08T23:18:12.316 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.317 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.318 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.319 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.320 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.321 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.322 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.323 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.324 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.325 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.326 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.327 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.328 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.329 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.330 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.331 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.332 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.333 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.334 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.335 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.336 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.337 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.338 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.339 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.340 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.341 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.342 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.343 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.344 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.345 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.346 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.347 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.348 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.349 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.350 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.351 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:12.352 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.353 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.354 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.355 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.356 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.357 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.358 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.359 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.360 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.361 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.362 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.363 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.364 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.365 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.366 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.367 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.368 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-08T23:18:12.369 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.370 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.371 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.372 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.373 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.374 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.375 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.376 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.377 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.378 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.379 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-08T23:18:12.380 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.381 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.382 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.383 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.384 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:12.385 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.386 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.387 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.388 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.389 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.390 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.391 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.392 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.393 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.394 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.395 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.396 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.397 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.398 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.399 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.400 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.401 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.402 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-08T23:18:12.403 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.404 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.405 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.406 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-08T23:18:12.407 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.408 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.409 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.410 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.411 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.412 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.413 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.414 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.415 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.416 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.417 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.417 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.417 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.417 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.418 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.420 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:12.421 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.422 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:12.423 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.424 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-08T23:18:12.425 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.426 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.427 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.428 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.429 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.430 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.431 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.432 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.433 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.434 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.435 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.436 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.437 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.438 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.439 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.440 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.441 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.442 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.443 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.444 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.445 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.446 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.447 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.448 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.449 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.450 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.451 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.452 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.453 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.454 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.455 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.457 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.458 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.459 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.460 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.462 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-08T23:18:12.463 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.464 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.465 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.466 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.467 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.468 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.469 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.470 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-08T23:18:12.471 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.472 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-08T23:18:12.473 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.474 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.475 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.476 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-08T23:18:12.477 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.478 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.479 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.480 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.481 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.482 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.483 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.484 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.485 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.486 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.487 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.488 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.489 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.490 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.491 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.492 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.493 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.494 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.495 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.496 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.497 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.498 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.499 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.500 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.501 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.502 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.503 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.504 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.505 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.506 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-08T23:18:12.507 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.508 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.509 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.510 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.511 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.512 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.513 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.514 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.515 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.516 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.518 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.520 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.521 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.522 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.523 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.524 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-08T23:18:12.525 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-08T23:18:12.526 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.527 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.528 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.529 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-08T23:18:12.530 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.531 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.532 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.533 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.534 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.535 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.536 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.537 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.538 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.539 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.540 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.541 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.542 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.543 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.544 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.545 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.546 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.547 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.548 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.549 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.550 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.551 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.552 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.553 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-08T23:18:12.554 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.555 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-08T23:18:12.556 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.557 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.558 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.559 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.560 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.561 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.562 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.563 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.564 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.565 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.566 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.567 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.568 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.569 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.570 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.571 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.572 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.573 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.574 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-08T23:18:12.575 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.576 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.577 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-08T23:18:12.578 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-08T23:18:12.579 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.580 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.581 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.582 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.583 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-08T23:18:12.584 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.585 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.586 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.587 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.588 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.589 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.590 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.591 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.592 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.593 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.594 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.595 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-08T23:18:12.596 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.597 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-08T23:18:12.598 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.599 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.600 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.601 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.602 INFO:teuthology.orchestra.run.smithi046.stdout: "gid": 25573, 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "z", 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "mgr_features": 4540701547738038271, 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.603 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.604 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.605 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.606 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.607 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.608 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.609 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.610 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.611 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.612 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.613 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.614 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.615 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.616 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-08T23:18:12.617 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-08T23:18:12.618 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.619 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.620 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-08T23:18:12.621 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.622 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.623 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.624 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.625 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.626 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.627 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-08T23:18:12.628 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-08T23:18:12.629 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.630 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.631 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:12.632 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-08T23:18:12.633 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-08T23:18:12.634 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.635 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.636 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-08T23:18:12.637 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.638 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.639 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-08T23:18:12.640 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-08T23:18:12.641 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.642 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.643 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.644 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.645 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.646 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.647 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.648 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.649 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-08T23:18:12.650 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.651 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.652 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-08T23:18:12.653 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-08T23:18:12.654 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-08T23:18:12.655 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.656 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.657 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.658 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.659 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.660 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.661 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.662 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.663 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-08T23:18:12.664 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-08T23:18:12.665 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.666 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.667 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.668 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.669 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.670 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.671 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.672 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-08T23:18:12.673 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-08T23:18:12.674 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.675 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.676 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.677 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.678 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-08T23:18:12.679 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.680 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.681 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.682 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.683 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.685 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.686 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-08T23:18:12.687 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.688 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.689 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.690 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.691 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-08T23:18:12.692 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-08T23:18:12.693 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.694 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.695 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.696 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-08T23:18:12.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:12.698 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.699 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.700 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.701 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-08T23:18:12.702 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.703 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.704 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.705 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.706 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-08T23:18:12.707 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-08T23:18:12.708 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.709 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.710 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.711 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.712 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.713 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-08T23:18:12.714 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-08T23:18:12.715 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.716 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.717 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.718 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-08T23:18:12.719 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.720 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.721 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.722 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.723 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.724 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-08T23:18:12.725 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.726 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.727 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.728 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.729 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.730 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.731 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.732 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.733 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-08T23:18:12.734 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-08T23:18:12.735 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.736 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.737 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.738 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-08T23:18:12.739 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.740 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.741 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.742 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.743 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.744 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.745 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-08T23:18:12.746 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-08T23:18:12.747 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.748 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.749 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.750 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-08T23:18:12.751 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.752 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.753 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.754 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.755 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-08T23:18:12.756 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-08T23:18:12.757 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.758 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.759 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.760 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.761 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.762 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.763 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.764 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.765 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.766 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.767 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.768 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.769 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.770 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.771 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.772 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.773 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.774 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-08T23:18:12.775 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-08T23:18:12.776 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.777 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.778 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-08T23:18:12.779 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.780 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.781 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.783 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.784 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.785 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.786 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.787 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.788 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.789 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.790 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.791 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.792 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.793 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.794 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.795 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.796 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.797 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.798 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-08T23:18:12.799 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.800 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.801 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.803 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-08T23:18:12.804 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.805 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.806 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.807 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.808 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.809 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.810 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.811 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-08T23:18:12.812 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.813 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.814 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.815 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "MULTICLUSTER_CONFIG": { 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "{}", 2024-03-08T23:18:12.816 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.817 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.818 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.819 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.820 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.821 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-08T23:18:12.822 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.823 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.824 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.825 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.826 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-08T23:18:12.827 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.828 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.829 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.830 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.831 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.832 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.833 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-08T23:18:12.834 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-08T23:18:12.835 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.836 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-08T23:18:12.837 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.838 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.839 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.840 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.841 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.842 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.843 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.844 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-08T23:18:12.845 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.846 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "UNSAFE_TLS_v1_2": { 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.847 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.848 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.849 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-08T23:18:12.850 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.851 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.852 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.853 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.854 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-08T23:18:12.855 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.856 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.857 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.858 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.859 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:12.860 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.861 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.862 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.863 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.864 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.865 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.866 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.867 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.868 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.869 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.870 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.871 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.872 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:12.873 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.874 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.875 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.876 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.877 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-08T23:18:12.878 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.879 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.880 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.881 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.882 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.883 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.884 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.885 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.886 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.887 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.888 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.889 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.890 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.891 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.892 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.893 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.895 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.896 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.897 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-08T23:18:12.898 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-08T23:18:12.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.900 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.901 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.902 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.903 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.904 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-08T23:18:12.905 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.906 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.907 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.908 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.909 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.910 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.911 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.912 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.913 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.914 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.915 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.917 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-08T23:18:12.918 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.919 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.920 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-08T23:18:12.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-08T23:18:12.923 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.924 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.926 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:12.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-08T23:18:12.928 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.929 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.930 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.931 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.932 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.933 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.934 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.935 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.936 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-08T23:18:12.937 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.938 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-08T23:18:12.939 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.940 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.941 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-08T23:18:12.942 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.943 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.944 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.945 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.946 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.948 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.949 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.950 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.951 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.952 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.953 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.955 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.956 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.957 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.958 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.959 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.960 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.961 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.962 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.963 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.964 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.965 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.966 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.967 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-08T23:18:12.968 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.969 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.970 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.971 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.972 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.973 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.974 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.975 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.976 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.977 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.978 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.979 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.980 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.981 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.982 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.983 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.984 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:12.985 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:12.986 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:12.987 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.988 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.989 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.990 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.991 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:12.992 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.993 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.994 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.995 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.996 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:12.997 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:12.998 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:12.999 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.000 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.001 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.002 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.003 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.004 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.005 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.006 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.007 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.008 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.009 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.010 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.011 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.012 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.013 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.014 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.015 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.017 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.019 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.020 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.021 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.022 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.023 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.024 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.025 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.026 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.027 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.028 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.029 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.030 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.031 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.032 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.033 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.034 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.035 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.036 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.037 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.038 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.039 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.040 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.041 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.042 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.043 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.044 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.046 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.047 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.048 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.049 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.050 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.051 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.052 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.053 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.054 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.055 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.056 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.057 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.058 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.059 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.060 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.061 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.062 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.063 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.064 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.065 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.066 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.067 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.068 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-08T23:18:13.069 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.070 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.071 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.072 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.073 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.074 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.075 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.076 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.077 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.078 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.079 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.080 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.081 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.082 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.083 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.084 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-08T23:18:13.085 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.086 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-08T23:18:13.087 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-08T23:18:13.088 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.089 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.090 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.091 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.092 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.093 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.094 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.095 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.097 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.097 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.097 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.097 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.097 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.098 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.099 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.100 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.101 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.102 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.103 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:13.104 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.105 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.106 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.107 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.108 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.109 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.110 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:13.111 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.112 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.113 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.114 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.115 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.116 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.117 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.118 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.119 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.120 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.121 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.122 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.123 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.124 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.125 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.126 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-08T23:18:13.127 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.128 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.129 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.130 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.131 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.132 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.133 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.134 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.135 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.136 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.137 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.138 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.139 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.140 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.141 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.142 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.143 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.144 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.145 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.146 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.147 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.148 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.149 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.150 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-08T23:18:13.151 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.152 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.153 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.154 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.155 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.156 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.157 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.158 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.159 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-08T23:18:13.160 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.161 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.162 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.163 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.164 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.165 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.166 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.167 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.168 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.169 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.170 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.171 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.172 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.173 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.174 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.175 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.176 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.177 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.178 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-08T23:18:13.179 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.180 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.181 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.182 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.183 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-08T23:18:13.184 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.185 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.186 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.187 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.188 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.189 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.190 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.191 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.192 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-08T23:18:13.193 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-08T23:18:13.194 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.195 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.196 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.197 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.198 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.199 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.200 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.201 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.202 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.203 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.204 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.205 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.206 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.207 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.208 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.209 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.210 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.211 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.212 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.213 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.214 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.215 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.216 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.217 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.218 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.219 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.220 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.221 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.222 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.223 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.224 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.225 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.226 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.227 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.228 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.229 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.230 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.231 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.232 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.233 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.234 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-08T23:18:13.235 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-08T23:18:13.236 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.237 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.238 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.239 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-08T23:18:13.240 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.241 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.242 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.243 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.244 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-08T23:18:13.245 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.246 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.247 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.248 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.249 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.250 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.251 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.252 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.253 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.254 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.255 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.256 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.257 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.258 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.259 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.260 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.261 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.262 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.263 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.264 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-08T23:18:13.265 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.266 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.267 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.268 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.269 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.270 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.271 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.272 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.273 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.274 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.275 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.276 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.277 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.278 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.279 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.280 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.281 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.282 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.283 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.284 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.285 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.286 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.287 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.288 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.289 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-08T23:18:13.290 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-08T23:18:13.291 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.292 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.293 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.294 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.295 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-08T23:18:13.296 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.297 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.298 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.299 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.301 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.302 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.303 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.304 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.305 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.306 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.307 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.308 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.309 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.310 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.311 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.312 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.313 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-08T23:18:13.314 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.315 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: "modules": [ 2024-03-08T23:18:13.316 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "dashboard", 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "iostat", 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "nfs", 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "restful", 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "snap_schedule" 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "available_modules": [ 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.317 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "alerts", 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.318 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How frequently to reexamine health status", 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.319 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.320 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.321 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.322 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.323 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.324 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.325 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.326 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.327 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_destination": { 2024-03-08T23:18:13.328 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_destination", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email address to send alerts to", 2024-03-08T23:18:13.329 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_from_name": { 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_from_name", 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.330 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "Ceph", 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Email From: name", 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.331 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_host": { 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_host", 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.332 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP server", 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_password": { 2024-03-08T23:18:13.333 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_password", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.334 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Password to authenticate with", 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_port": { 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_port", 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.335 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "465", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP port", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.336 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_sender": { 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_sender", 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.337 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SMTP envelope sender", 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.338 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_ssl": { 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_ssl", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.339 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "smtp_user": { 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "smtp_user", 2024-03-08T23:18:13.340 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "User to authenticate as", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.341 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "balancer", 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.342 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "active": { 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "active", 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.343 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically balance PGs across cluster", 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_time": { 2024-03-08T23:18:13.344 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_time", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0000", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.345 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "beginning time of day to automatically balance", 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "begin_weekday": { 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "begin_weekday", 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.346 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.347 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_max_iterations": { 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_max_iterations", 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "25", 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:13.348 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "250", 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_metrics": { 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_metrics", 2024-03-08T23:18:13.349 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "pgs,objects,bytes", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-08T23:18:13.350 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "crush_compat_step": { 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crush_compat_step", 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.5", 2024-03-08T23:18:13.351 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0.001", 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "0.999", 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "aggressiveness of optimization", 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "end_time": { 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_time", 2024-03-08T23:18:13.352 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2359", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ending time of day to automatically balance", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-08T23:18:13.353 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "end_weekday": { 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "end_weekday", 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.354 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "0", 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "6", 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.356 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.357 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.358 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.359 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.360 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.361 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.362 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.363 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "min_score": { 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_score", 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.364 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.365 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "upmap", 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.366 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "crush-compat", 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "none", 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "read", 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap", 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap-read" 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.367 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Balancer mode", 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_ids": { 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_ids", 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.368 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.369 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.370 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_deviation": { 2024-03-08T23:18:13.371 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_deviation", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-08T23:18:13.372 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "upmap_max_optimizations": { 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "upmap_max_optimizations", 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.373 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.374 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm", 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.375 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_down_multiplier": { 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_down_multiplier", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.376 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_refresh_rate": { 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_refresh_rate", 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.377 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "20", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.378 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "agent_starting_port": { 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "agent_starting_port", 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4721", 2024-03-08T23:18:13.379 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_ptrace": { 2024-03-08T23:18:13.380 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_ptrace", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.381 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_interval": { 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_interval", 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.382 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.383 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "autotune_memory_target_ratio": { 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "autotune_memory_target_ratio", 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0.7", 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.384 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm_log_destination": { 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cephadm_log_destination", 2024-03-08T23:18:13.385 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "file", 2024-03-08T23:18:13.386 INFO:teuthology.orchestra.run.smithi046.stdout: "file,syslog", 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "syslog" 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.387 INFO:teuthology.orchestra.run.smithi046.stdout: "cgroups_split": { 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cgroups_split", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.388 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "config_checks_enabled": { 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_checks_enabled", 2024-03-08T23:18:13.389 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-08T23:18:13.390 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "config_dashboard": { 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_dashboard", 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.391 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.392 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_alertmanager": { 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_alertmanager", 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.393 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_base": { 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_base", 2024-03-08T23:18:13.394 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Container image name, without the tag", 2024-03-08T23:18:13.395 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_elasticsearch": { 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_elasticsearch", 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.396 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "elasticsearch container image", 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.397 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_grafana": { 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_grafana", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.398 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_haproxy": { 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_haproxy", 2024-03-08T23:18:13.399 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "HAproxy container image", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.400 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_agent": { 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_agent", 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.401 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger agent container image", 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.402 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_collector": { 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_collector", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.403 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger collector container image", 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_jaeger_query": { 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_jaeger_query", 2024-03-08T23:18:13.404 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Jaeger query container image", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.405 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_keepalived": { 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_keepalived", 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.406 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Keepalived container image", 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.407 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_loki": { 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_loki", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.408 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Loki container image", 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_node_exporter": { 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_node_exporter", 2024-03-08T23:18:13.409 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.410 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_nvmeof": { 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_nvmeof", 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.411 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Nvme-of container image", 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.412 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_prometheus": { 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_prometheus", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.413 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Prometheus container image", 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_promtail": { 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_promtail", 2024-03-08T23:18:13.414 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Promtail container image", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.415 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "container_image_snmp_gateway": { 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_image_snmp_gateway", 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.416 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "SNMP Gateway container image", 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.417 INFO:teuthology.orchestra.run.smithi046.stdout: "container_init": { 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "container_init", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.418 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Run podman/docker with `--init`", 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "daemon_cache_timeout": { 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "daemon_cache_timeout", 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.419 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.420 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "default_cephadm_command_timeout": { 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_cephadm_command_timeout", 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "900", 2024-03-08T23:18:13.421 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.422 INFO:teuthology.orchestra.run.smithi046.stdout: "default_registry": { 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "default_registry", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "docker.io", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.423 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "device_cache_timeout": { 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_cache_timeout", 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.424 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1800", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache device inventory", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.425 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "device_enhanced_scan": { 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_enhanced_scan", 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.426 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.427 INFO:teuthology.orchestra.run.smithi046.stdout: "facts_cache_timeout": { 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "facts_cache_timeout", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.428 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "seconds to cache host facts data", 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "host_check_interval": { 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "host_check_interval", 2024-03-08T23:18:13.429 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to perform a host check", 2024-03-08T23:18:13.430 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "hw_monitoring": { 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hw_monitoring", 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.431 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.432 INFO:teuthology.orchestra.run.smithi046.stdout: "inventory_list_all": { 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "inventory_list_all", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.433 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.434 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.435 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.436 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "log_refresh_metadata": { 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_refresh_metadata", 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.437 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.438 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.439 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.440 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.441 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.442 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.443 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-08T23:18:13.444 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-08T23:18:13.445 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.446 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "*", 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.447 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "max_count_per_host": { 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_count_per_host", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.448 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of daemons per service per host", 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "max_osd_draining_count": { 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_osd_draining_count", 2024-03-08T23:18:13.449 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-08T23:18:13.450 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "migration_current": { 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "migration_current", 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.451 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "internal - do not modify", 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.452 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "mode": { 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mode", 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "root", 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.453 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm-package", 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "root" 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "mode for remote execution of cephadm", 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.454 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "oob_default_addr": { 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "oob_default_addr", 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "169.254.1.1", 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.455 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "prometheus_alerts_path": { 2024-03-08T23:18:13.456 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus_alerts_path", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-08T23:18:13.457 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_insecure": { 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_insecure", 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.458 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.459 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_password": { 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_password", 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.460 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_url": { 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_url", 2024-03-08T23:18:13.461 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-08T23:18:13.462 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "registry_username": { 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "registry_username", 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.463 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.464 INFO:teuthology.orchestra.run.smithi046.stdout: "secure_monitoring_stack": { 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "secure_monitoring_stack", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.465 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "service_discovery_port": { 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "service_discovery_port", 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.466 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8765", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "cephadm service discovery port", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.467 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "ssh_config_file": { 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssh_config_file", 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.468 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.469 INFO:teuthology.orchestra.run.smithi046.stdout: "use_agent": { 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_agent", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.470 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "use_repo_digest": { 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "use_repo_digest", 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.471 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.472 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_failed_host_check": { 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_failed_host_check", 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.473 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if the host check fails", 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.474 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_daemons": { 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_daemons", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.475 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_on_stray_hosts": { 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_on_stray_hosts", 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.476 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.477 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crash", 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.478 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.479 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.480 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.481 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.482 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.483 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.484 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.485 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.486 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "retain_interval": { 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retain_interval", 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "31536000", 2024-03-08T23:18:13.487 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.488 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_recent_interval": { 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_recent_interval", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1209600", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.489 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.490 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dashboard", 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.491 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.492 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_HOST": { 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.493 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.494 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.495 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_ENABLED": { 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_ENABLED", 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.496 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.497 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.498 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "ENABLE_BROWSABLE_API": { 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-08T23:18:13.499 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.500 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.501 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-08T23:18:13.502 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.503 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-08T23:18:13.505 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.506 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.507 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.508 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.509 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-08T23:18:13.510 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.511 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.512 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.513 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.514 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_PASSWORD": { 2024-03-08T23:18:13.515 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.516 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.517 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.518 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_URL": { 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_URL", 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.519 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.520 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_API_USERNAME": { 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.521 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.522 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.523 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.524 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.525 INFO:teuthology.orchestra.run.smithi046.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.526 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-08T23:18:13.527 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.528 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "MULTICLUSTER_CONFIG": { 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.529 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "{}", 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.530 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_HOST": { 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.531 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-08T23:18:13.532 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.533 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.534 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.535 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.536 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.537 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.538 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.539 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-08T23:18:13.540 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.541 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.542 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.543 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.544 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_ENABLED": { 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-08T23:18:13.545 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.546 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.547 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.548 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.549 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-08T23:18:13.550 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.551 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.552 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "45", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.553 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ACCESS_KEY": { 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.554 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-08T23:18:13.555 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "admin", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.556 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SECRET_KEY": { 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.557 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.558 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "RGW_API_SSL_VERIFY": { 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.559 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "UNSAFE_TLS_v1_2": { 2024-03-08T23:18:13.560 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.561 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.562 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.563 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.564 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.565 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.566 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "cross_origin_url": { 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cross_origin_url", 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.567 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.568 INFO:teuthology.orchestra.run.smithi046.stdout: "crt_file": { 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "crt_file", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.569 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "debug": { 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "debug", 2024-03-08T23:18:13.570 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Enable/disable debug options", 2024-03-08T23:18:13.571 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "jwt_token_ttl": { 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "jwt_token_ttl", 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "28800", 2024-03-08T23:18:13.572 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.573 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.574 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.575 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.576 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.577 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.578 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.579 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.580 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.581 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.582 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.583 INFO:teuthology.orchestra.run.smithi046.stdout: "motd": { 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "motd", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.584 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "The message of the day", 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect_resolve_ip_addr": { 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "redirect_resolve_ip_addr", 2024-03-08T23:18:13.585 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.586 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.587 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.588 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8080", 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.589 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:13.590 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.591 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl_server_port": { 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl_server_port", 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.592 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8443", 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.593 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "redirect", 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.594 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "redirect" 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.595 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:13.596 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "url_prefix": { 2024-03-08T23:18:13.597 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url_prefix", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.598 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.599 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_monitoring": { 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_monitoring", 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.600 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "monitor device health metrics", 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.601 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.602 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.603 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.604 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.605 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.606 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.607 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.608 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.609 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "mark_out_threshold": { 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mark_out_threshold", 2024-03-08T23:18:13.610 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "2419200", 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.611 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "pool_name": { 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pool_name", 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.612 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "device_health_metrics", 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.613 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "retention_period": { 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "retention_period", 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15552000", 2024-03-08T23:18:13.614 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long to retain device health metrics", 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.615 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_frequency": { 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_frequency", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.616 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "self_heal": { 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "self_heal", 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.617 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.618 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.619 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how frequently to wake up and check device health", 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.620 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "warn_threshold": { 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "warn_threshold", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7257600", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.621 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.622 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "diskprediction_local", 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.623 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.624 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.625 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.626 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.627 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.628 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.629 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.630 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "predict_interval": { 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predict_interval", 2024-03-08T23:18:13.631 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "86400", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.632 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "predictor_model": { 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "predictor_model", 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.633 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "prophetstor", 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.634 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "600", 2024-03-08T23:18:13.635 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.636 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "influx", 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": false, 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "influxdb python module not found", 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "batch_size": { 2024-03-08T23:18:13.637 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "batch_size", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5000", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.638 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "database": { 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "database", 2024-03-08T23:18:13.639 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "ceph", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-08T23:18:13.640 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "hostname": { 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "hostname", 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.641 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server hostname", 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.642 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "30", 2024-03-08T23:18:13.643 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "5", 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.644 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.645 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.646 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.647 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.648 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.649 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.650 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.651 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.652 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "password": { 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "password", 2024-03-08T23:18:13.653 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "password of InfluxDB server user", 2024-03-08T23:18:13.654 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "port": { 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "port", 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.655 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "8086", 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "InfluxDB server port", 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.656 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "ssl": { 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ssl", 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "false", 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.657 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "threads": { 2024-03-08T23:18:13.658 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threads", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "32", 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.659 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "username": { 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "username", 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.660 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "username of InfluxDB server user", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.661 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "verify_ssl": { 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "verify_ssl", 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "true", 2024-03-08T23:18:13.662 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.663 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "insights", 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.664 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.665 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.666 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.667 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.668 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.669 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.670 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.671 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.672 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "iostat", 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.673 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.674 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.675 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.676 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.677 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.678 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.679 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.680 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.681 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "k8sevents", 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.682 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_event_retention_days": { 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "ceph_event_retention_days", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "7", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.683 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "config_check_secs": { 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "config_check_secs", 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.684 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "10", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.685 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.686 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.687 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.688 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.689 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.690 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.691 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.692 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.693 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "localpool", 2024-03-08T23:18:13.694 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "failure_domain": { 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "failure_domain", 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.695 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "host", 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "failure domain for any created local pool", 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.696 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.697 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.698 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.699 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.700 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.701 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.702 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.703 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.704 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "min_size": { 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "min_size", 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.705 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default min_size for any created local pool", 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "num_rep": { 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "num_rep", 2024-03-08T23:18:13.706 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default replica count for any created local pool", 2024-03-08T23:18:13.707 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_num": { 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_num", 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.709 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "128", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "default pg_num for any created local pool", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.710 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "prefix": { 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prefix", 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.711 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "name prefix for any created local pool", 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.712 INFO:teuthology.orchestra.run.smithi046.stdout: "subtree": { 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "subtree", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "rack", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.713 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.714 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mds_autoscaler", 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.715 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.716 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.717 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.718 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.719 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.720 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.721 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.722 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.723 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirroring", 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.724 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.725 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.726 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.727 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.728 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.729 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.730 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.731 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.732 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "nfs", 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.733 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.734 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.735 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.736 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.737 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.738 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.739 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.740 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.741 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.742 INFO:teuthology.orchestra.run.smithi046.stdout: "fail_fs": { 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "fail_fs", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.743 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.744 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.745 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.746 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.747 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.748 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.749 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.750 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.751 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.752 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator": { 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "orchestrator", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.753 INFO:teuthology.orchestra.run.smithi046.stdout: "cephadm", 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "rook", 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "test_orchestrator" 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Orchestrator backend", 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.754 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_perf_query", 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.755 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.756 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.757 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.758 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.759 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.760 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.761 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.762 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.763 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "osd_support", 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.764 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.765 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.766 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.767 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.768 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.769 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.770 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.771 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.772 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "pg_autoscaler", 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.773 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.774 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.775 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.776 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.777 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.778 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.779 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.780 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.781 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.782 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.783 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "threshold": { 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "threshold", 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "3.0", 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1.0", 2024-03-08T23:18:13.784 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "scaling threshold", 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.785 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "progress", 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_pg_recovery_event": { 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_pg_recovery_event", 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.786 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.787 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.788 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.789 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.790 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.791 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.792 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.793 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.794 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.795 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.796 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.797 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "max_completed_events": { 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_completed_events", 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.798 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "50", 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "number of past completed events to remember", 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "sleep_interval": { 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "sleep_interval", 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "5", 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.800 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "how long the module is going to sleep", 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.801 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "prometheus", 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "cache": { 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "cache", 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.802 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.803 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "exclude_perf_counters": { 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "exclude_perf_counters", 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.804 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.805 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.806 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.807 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.808 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.809 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.810 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.811 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.812 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.813 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools": { 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools", 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.814 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-08T23:18:13.815 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "300", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.816 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "scrape_interval": { 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "scrape_interval", 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.817 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15.0", 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.818 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "::", 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.819 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:13.820 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "9283", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-08T23:18:13.821 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "stale_cache_strategy": { 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stale_cache_strategy", 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.822 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "log", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.823 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_behaviour": { 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_behaviour", 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "default", 2024-03-08T23:18:13.824 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "default", 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "error" 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.825 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "standby_error_status_code": { 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "standby_error_status_code", 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "500", 2024-03-08T23:18:13.826 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "400", 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "599", 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.827 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.828 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.829 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.830 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.831 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.832 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.833 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.834 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.835 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.836 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_snap_create": { 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_snap_create", 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.837 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10", 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.838 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "mirror_snapshot_schedule": { 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "mirror_snapshot_schedule", 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.839 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.840 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "trash_purge_schedule": { 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "trash_purge_schedule", 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.841 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.842 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "restful", 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "enable_auth": { 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enable_auth", 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.843 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.844 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "key_file": { 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "key_file", 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.845 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.846 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.847 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.848 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.849 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.850 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.851 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.852 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.853 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.854 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "server_addr": { 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_addr", 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.855 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "server_port": { 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "server_port", 2024-03-08T23:18:13.856 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.857 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rgw", 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.858 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.859 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.860 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.861 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.862 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.863 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.864 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.865 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.866 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.867 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rook", 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.868 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.869 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.870 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.871 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.872 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.873 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.874 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.875 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "storage_class": { 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "storage_class", 2024-03-08T23:18:13.876 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "local", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.877 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "selftest", 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.878 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.879 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.880 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.881 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.882 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.883 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.884 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.885 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.886 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "roption1": { 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption1", 2024-03-08T23:18:13.887 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.888 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "roption2": { 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "roption2", 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.889 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "xyz", 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.890 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption1": { 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption1", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.891 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption2": { 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption2", 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.892 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.893 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption3": { 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption3", 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "float", 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.894 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.895 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption4": { 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption4", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.896 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption5": { 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption5", 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.897 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.898 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption6": { 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption6", 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.899 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "rwoption7": { 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rwoption7", 2024-03-08T23:18:13.900 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "1", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "42", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.901 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "testkey": { 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testkey", 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.902 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.903 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "testlkey": { 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testlkey", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.904 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "testnewline": { 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "testnewline", 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.905 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.906 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.907 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "allow_m_granularity": { 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "allow_m_granularity", 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.908 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "allow minute scheduled snapshots", 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "dump_on_update": { 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "dump_on_update", 2024-03-08T23:18:13.909 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "dump database to debug log on update", 2024-03-08T23:18:13.910 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.913 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.913 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.913 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.913 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.913 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.914 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.915 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.916 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.917 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.918 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.919 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.920 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.921 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "stats", 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.922 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.923 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.924 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.925 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.926 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.927 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.928 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.929 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.930 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "status", 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.931 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.932 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.933 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.934 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.935 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.936 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.937 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.938 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.939 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telegraf", 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.940 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "address": { 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "address", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.941 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.942 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "15", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.943 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.944 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.945 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.946 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.947 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.948 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.949 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.950 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.951 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "telemetry", 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_basic": { 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_basic", 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.953 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.954 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_crash": { 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_crash", 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.955 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.956 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_device": { 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_device", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.957 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_ident": { 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_ident", 2024-03-08T23:18:13.958 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-08T23:18:13.959 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "channel_perf": { 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "channel_perf", 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.960 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.961 INFO:teuthology.orchestra.run.smithi046.stdout: "contact": { 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "contact", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.962 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "description": { 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "description", 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.963 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.964 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "device_url": { 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "device_url", 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-08T23:18:13.965 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.966 INFO:teuthology.orchestra.run.smithi046.stdout: "enabled": { 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "enabled", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.967 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.968 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "24", 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "8", 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.969 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "last_opt_revision": { 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "last_opt_revision", 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "1", 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.970 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard": { 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard", 2024-03-08T23:18:13.971 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.972 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "leaderboard_description": { 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "leaderboard_description", 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.973 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.974 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.975 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.976 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.977 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.978 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.979 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.980 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.981 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.982 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "organization": { 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "organization", 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.983 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "proxy": { 2024-03-08T23:18:13.984 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "proxy", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.985 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "url": { 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "url", 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.986 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "test_orchestrator", 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.988 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.989 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.990 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.991 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.992 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.993 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.994 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:13.995 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:13.996 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:13.997 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:13.998 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:13.999 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.000 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.001 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.002 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.003 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.004 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.005 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max_concurrent_clones": { 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "max_concurrent_clones", 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "4", 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.006 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "periodic_async_work": { 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "periodic_async_work", 2024-03-08T23:18:14.007 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Periodically check for async work", 2024-03-08T23:18:14.008 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_delay": { 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_delay", 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.009 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "0", 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.010 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "snapshot_clone_no_wait": { 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snapshot_clone_no_wait", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "True", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.011 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.012 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix", 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "can_run": true, 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "error_string": "", 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "module_options": { 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "discovery_interval": { 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "discovery_interval", 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "uint", 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.013 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "100", 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.014 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "identifier": { 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "identifier", 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.015 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "interval": { 2024-03-08T23:18:14.016 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "interval", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "secs", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "60", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.017 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "log_level": { 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_level", 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.018 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:14.019 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.020 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster": { 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.021 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_cluster_level": { 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_cluster_level", 2024-03-08T23:18:14.022 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "info", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [ 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "critical", 2024-03-08T23:18:14.023 INFO:teuthology.orchestra.run.smithi046.stdout: "debug", 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "error", 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "info", 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "warning" 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.024 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "log_to_file": { 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "log_to_file", 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "bool", 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 1, 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "False", 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.025 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_host": { 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_host", 2024-03-08T23:18:14.026 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.027 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_port": { 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_port", 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "int", 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.028 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "10051", 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.029 INFO:teuthology.orchestra.run.smithi046.stdout: "zabbix_sender": { 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "zabbix_sender", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "str", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "level": "advanced", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "flags": 0, 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "min": "", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "max": "", 2024-03-08T23:18:14.030 INFO:teuthology.orchestra.run.smithi046.stdout: "enum_allowed": [], 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: "desc": "", 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: "long_desc": "", 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: "tags": [], 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: "see_also": [] 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.031 INFO:teuthology.orchestra.run.smithi046.stdout: "services": { 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "dashboard": "https://172.21.15.46:8443/" 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "always_on_modules": { 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "octopus": [ 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-08T23:18:14.032 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "pacific": [ 2024-03-08T23:18:14.033 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-08T23:18:14.034 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "quincy": [ 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-08T23:18:14.035 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "reef": [ 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-08T23:18:14.036 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: ], 2024-03-08T23:18:14.037 INFO:teuthology.orchestra.run.smithi046.stdout: "squid": [ 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "balancer", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "crash", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "devicehealth", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "orchestrator", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "pg_autoscaler", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "progress", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "rbd_support", 2024-03-08T23:18:14.038 INFO:teuthology.orchestra.run.smithi046.stdout: "status", 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "telemetry", 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "volumes" 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "last_failure_osd_epoch": 94, 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "active_clients": [ 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "devicehealth", 2024-03-08T23:18:14.039 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2917273677 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.040 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "libcephsqlite", 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 2380518622 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.041 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "rbd_support", 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3983816933 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.042 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.043 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3884256512 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "snap_schedule", 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.044 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 4211358908 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: "name": "volumes", 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: "addrvec": [ 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: { 2024-03-08T23:18:14.045 INFO:teuthology.orchestra.run.smithi046.stdout: "type": "v2", 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: "addr": "172.21.15.46:0", 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: "nonce": 3855360746 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout: ] 2024-03-08T23:18:14.046 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:14.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: pgmap v324: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:14.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2096374428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.048 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1648159414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:11 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2375809949' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.050 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:13 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.051 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: pgmap v324: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2096374428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1648159414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:11 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2375809949' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.052 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.054 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.055 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:13 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 msgr2=0x7f37a40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:14.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 0x7f37a40a4eb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f37c4102480 tx=0x7f37ac005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:14.055 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 msgr2=0x7f37c419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f37b400e9f0 tx=0x7f37b400eeb0 comp rx=0 tx=0).stop 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 shutdown_connections 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37a40a29f0 0x7f37a40a4eb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f37c41999a0 0x7f37c4199e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f37c4108610 0x7f37c419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 --2- 172.21.15.46:0/2375809949 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f37c4103540 0x7f37c419f7f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 >> 172.21.15.46:0/2375809949 conn(0x7f37c40fed80 msgr2=0x7f37c40ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:14.056 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 shutdown_connections 2024-03-08T23:18:14.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:11.856+0000 7f37c3e00640 1 -- 172.21.15.46:0/2375809949 wait complete. 2024-03-08T23:18:14.064 DEBUG:tasks.check_counter:Getting stats from a 2024-03-08T23:18:14.064 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: pgmap v324: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 53 KiB/s wr, 26 op/s 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2096374428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1648159414' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:14.065 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:11 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2375809949' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.066 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.067 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.068 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:13 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:14.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:18:14.474 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:14 smithi046 ceph-mon[31859]: osdmap e104: 12 total, 12 up, 12 in 2024-03-08T23:18:14.524 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:18:14.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:14 smithi062 ceph-mon[35761]: osdmap e104: 12 total, 12 up, 12 in 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: pgmap v325: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi192 to 174.1M 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi192 to 182617702: error parsing value: Value '182617702' is below minimum 939524096 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi062 to 174.1M 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi062 to 182642790: error parsing value: Value '182642790' is below minimum 939524096 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:14.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:14 smithi192 ceph-mon[36700]: osdmap e104: 12 total, 12 up, 12 in 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: pgmap v327: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:18:16.642 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:16 smithi062 ceph-mon[35761]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:16.688 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: pgmap v327: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.689 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:18:16.690 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:18:16.690 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:16 smithi046 ceph-mon[31859]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:16.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: pgmap v327: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:18:16.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:18:16.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:18:16.814 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:16 smithi192 ceph-mon[36700]: from='mgr.25559 ' entity='mgr.x' 2024-03-08T23:18:16.910 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 10129, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 10240, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 2708651, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 11947291, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 47, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 8, 2024-03-08T23:18:16.911 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 1.046014550, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.416103133, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.288127714, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.259331768, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 10238, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.688874422, 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000067286 2024-03-08T23:18:16.912 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 908, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000291989, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000321 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 928, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 736 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.913 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 9484, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 10314, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 7288551, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 4302011, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 13, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 1.259082407, 2024-03-08T23:18:16.914 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.520368046, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.638362777, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.176665466, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 10315, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.798963302, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000077456 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.915 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 6515, 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.003781219, 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000580 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1791600, 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 156144 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:16.916 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 20055, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 25447, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 5788943, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 11662815, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 11, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 7, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 3.101642637, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 1.679761993, 2024-03-08T23:18:16.917 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 1.102786285, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.382170149, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 25443, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 2.448464743, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000096233 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 10429, 2024-03-08T23:18:16.918 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.006923078, 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000663 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1758400, 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 6412624 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 1, 2024-03-08T23:18:16.919 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-MDSRank": { 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 20079, 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 7.136941488, 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000355443 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.920 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "queue_len": 0, 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "complete_latency": { 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1050, 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.180378712, 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000171789 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.921 INFO:teuthology.orchestra.run.smithi046.stdout: "mds": { 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "request": 2954, 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "reply": 2888, 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "reply_latency": { 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2888, 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 7.334224169, 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.002539551 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.922 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_reply": 0, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "forward": 66, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_commit": 11, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_split": 0, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_merge": 0, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes": 3256, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_top": 49, 2024-03-08T23:18:16.923 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_bottom": 22, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pin_tail": 3185, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_pinned": 3185, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_expired": 745, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "inodes_with_caps": 326, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "caps": 326, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "subtrees": 65, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse": 5016, 2024-03-08T23:18:16.924 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_hit": 4005, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_forward": 66, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_discover": 29, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "traverse_lock": 3, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "load_cent": 0, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "q": 0, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "exported": 15, 2024-03-08T23:18:16.925 INFO:teuthology.orchestra.run.smithi046.stdout: "exported_inodes": 688, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "imported": 52, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "imported_inodes": 3412, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_backtrace_fetch": 29826, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rfiles": 3452, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rbytes": 26509461, 2024-03-08T23:18:16.926 INFO:teuthology.orchestra.run.smithi046.stdout: "root_rsnaps": 6, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_fetch": 29826, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_inodes": 446, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_dirfrag_rstats": 446, 2024-03-08T23:18:16.927 INFO:teuthology.orchestra.run.smithi046.stdout: "scrub_file_inodes": 29380, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_inode_file_caps": 0, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_revoke": 54, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_grant": 1341, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_trunc": 2, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps": 1441, 2024-03-08T23:18:16.928 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_caps_dirty": 1377, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_cap_release": 31, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "process_request_cap_release": 1454 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_cache": { 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update": 65, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_update_receipt": 271, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_try_discover": 139, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_send_discover": 206, 2024-03-08T23:18:16.929 INFO:teuthology.orchestra.run.smithi046.stdout: "dir_handle_discover": 79, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays": 0, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_created": 640, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_enqueued": 640, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_reintegrated": 0, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "strays_migrated": 0, 2024-03-08T23:18:16.930 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_started": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "recovery_completed": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_exportdir": 15, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_flush": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragmentdir": 0, 2024-03-08T23:18:16.931 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "ireq_inodestats": 0 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_log": { 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "evlrg": 1, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "evadd": 4552, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "evex": 0, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "evtrm": 0, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "ev": 4552, 2024-03-08T23:18:16.932 INFO:teuthology.orchestra.run.smithi046.stdout: "evexg": 0, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "evexd": 0, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segadd": 8, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segex": 0, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segtrm": 0, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "seg": 8, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segmjr": 2, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segexg": 0, 2024-03-08T23:18:16.933 INFO:teuthology.orchestra.run.smithi046.stdout: "segexd": 0, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "expos": 4194304, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "wrpos": 13074237, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "rdpos": 0, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "jlat": { 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 193, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.910027621, 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.004715169 2024-03-08T23:18:16.934 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "replayed": 0 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_mem": { 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "ino": 2612, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "ino+": 3521, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "ino-": 909, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "dir": 215, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "dir+": 249, 2024-03-08T23:18:16.935 INFO:teuthology.orchestra.run.smithi046.stdout: "dir-": 34, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "dn": 3256, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "dn+": 4690, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "dn-": 1434, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "cap": 326, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "cap+": 2985, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "cap-": 2659, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "rss": 76284, 2024-03-08T23:18:16.936 INFO:teuthology.orchestra.run.smithi046.stdout: "heap": 207228 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_server": { 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_client_request": 3209, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "dispatch_server_request": 3, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_request": 2954, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_client_session": 24, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "handle_peer_request": 99, 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "req_create_latency": { 2024-03-08T23:18:16.937 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 454, 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.836932796, 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001843464 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getattr_latency": { 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2, 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.009783464, 2024-03-08T23:18:16.938 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.004891732 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "req_link_latency": { 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.939 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookup_latency": { 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1598, 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 2.338053948, 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001463112 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:16.940 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupino_latency": { 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 1, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000151392, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000151392 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.941 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupname_latency": { 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.942 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "req_lssnap_latency": { 2024-03-08T23:18:16.943 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.944 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.944 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mkdir_latency": { 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 3, 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.009550477, 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.003183492 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.945 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mknod_latency": { 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "req_mksnap_latency": { 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.946 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "req_open_latency": { 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 111, 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.067891068, 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000611631 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "req_readdir_latency": { 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 59, 2024-03-08T23:18:16.947 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.075492158, 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001279528 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rename_latency": { 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 18, 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 2.213845159, 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.122991397 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.948 INFO:teuthology.orchestra.run.smithi046.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.949 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmdir_latency": { 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:16.950 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.951 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setattr_latency": { 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2, 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.016100583, 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.008050291 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.952 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setlayout_latency": { 2024-03-08T23:18:16.953 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "req_setxattr_latency": { 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.954 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "req_symlink_latency": { 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 2, 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.002369787, 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.001184893 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "req_unlink_latency": { 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 638, 2024-03-08T23:18:16.955 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 1.764053337, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.002764973 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.956 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_sessions": { 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "session_count": 1, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "session_add": 2, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "session_remove": 1, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_open": 1, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "sessions_stale": 0, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "total_load": 1994, 2024-03-08T23:18:16.957 INFO:teuthology.orchestra.run.smithi046.stdout: "average_load": 1994, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "avg_session_uptime": 624, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:16.958 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:16.959 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:16.960 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:16.961 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 380117, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 2472, 2024-03-08T23:18:16.962 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 880, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 10, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 9664, 2024-03-08T23:18:16.963 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 187, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 8846842, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 100094, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-08T23:18:16.964 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "objecter": { 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "op_active": 0, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "op_laggy": 0, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send": 31439, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "op_send_bytes": 9285748, 2024-03-08T23:18:16.965 INFO:teuthology.orchestra.run.smithi046.stdout: "op_resend": 27, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "op_reply": 31439, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "op_latency": { 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 31439, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 25.124729707, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000799158 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "op_inflight": 0, 2024-03-08T23:18:16.966 INFO:teuthology.orchestra.run.smithi046.stdout: "oplen_avg": { 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 31439, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 31522 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "op": 31439, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "op_r": 29842, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "op_w": 1597, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "op_rmw": 0, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "op_pg": 0, 2024-03-08T23:18:16.967 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_stat": 0, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_create": 10, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_read": 16, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_write": 209, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writefull": 23, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_writesame": 0, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_append": 0, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_zero": 0, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_truncate": 0, 2024-03-08T23:18:16.968 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_delete": 1294, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_mapext": 0, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_getxattr": 29826, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_setxattr": 20, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:16.969 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_call": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_watch": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_notify": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "osdop_other": 2, 2024-03-08T23:18:16.970 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_active": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_send": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_resend": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "linger_ping": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_active": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_send": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "poolop_resend": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_active": 0, 2024-03-08T23:18:16.971 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_send": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "poolstat_resend": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_active": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_send": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "statfs_resend": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "command_active": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "command_send": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "command_resend": 0, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "map_epoch": 104, 2024-03-08T23:18:16.972 INFO:teuthology.orchestra.run.smithi046.stdout: "map_full": 0, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "map_inc": 15, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_sessions": 12, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_open": 12, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_session_close": 0, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_laggy": 0, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_wr": 100, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_rd": 0, 2024-03-08T23:18:16.973 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_del": 22 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "oft": { 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_objs": 1, 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_updates": 0, 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "omap_total_removes": 0 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.974 INFO:teuthology.orchestra.run.smithi046.stdout: "osd-slow-ops": { 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "slow_ops_count": 0 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "purge_queue": { 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_ops_high_water": 16, 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing": 0, 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executing_high_water": 8, 2024-03-08T23:18:16.975 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed_ops": 1280, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_executed": 640, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:16.976 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 39664, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 11998406, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 39664, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 39664, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 11998406, 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:16.977 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-08T23:18:16.978 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 31439, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 9355508, 2024-03-08T23:18:16.979 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 31439, 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 9355508, 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:16.980 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 1024, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 31439, 2024-03-08T23:18:16.981 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 31439, 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 31439, 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 31439, 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.982 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 640, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 69760, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:16.983 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 640, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 16, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 69760, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.984 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-write_buf_throttle-0x55ffe43b86e0": { 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 3758096384, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 4552, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 8879933, 2024-03-08T23:18:16.985 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 4552, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 193, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 8879933, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:16.986 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:16.987 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:16.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.987 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:16.987 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:17 smithi062 ceph-mon[35761]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:18:17.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:17 smithi062 ceph-mon[35761]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:18:17.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:17 smithi046 ceph-mon[31859]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:18:17.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:17 smithi046 ceph-mon[31859]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:18:17.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:17 smithi192 ceph-mon[36700]: Adjusting osd_memory_target on smithi046 to 177.8M 2024-03-08T23:18:17.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:17 smithi192 ceph-mon[36700]: Unable to set osd_memory_target on smithi046 to 186498457: error parsing value: Value '186498457' is below minimum 939524096 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds.exported=15 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds.imported=52 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=65 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=271 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=6 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-03-08T23:18:17.876 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-03-08T23:18:17.877 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-08T23:18:17.877 DEBUG:tasks.check_counter:Getting stats from d 2024-03-08T23:18:17.877 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-08T23:18:18.343 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:18:18.500 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:18 smithi046 ceph-mon[31859]: pgmap v328: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-08T23:18:18.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:18 smithi062 ceph-mon[35761]: pgmap v328: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-08T23:18:18.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:18 smithi192 ceph-mon[36700]: pgmap v328: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 206 B/s wr, 6 op/s 2024-03-08T23:18:20.582 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:20 smithi046 ceph-mon[31859]: pgmap v329: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:20.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:20 smithi062 ceph-mon[35761]: pgmap v329: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:20.796 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-08T23:18:20.797 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.004048891, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000958261, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:20.798 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:20.799 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 2, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 175, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 320, 2024-03-08T23:18:20.800 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 185856, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 5, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.035901079, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.025781098, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.003998190, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:20.801 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 172, 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.033467710, 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000194579 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:20.802 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 185856 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 244, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 230, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1719840, 2024-03-08T23:18:20.803 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 137712, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.109729802, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.026108771, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.111432896, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.010406949, 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:20.804 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 230, 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.022893328, 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000099536 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:20.805 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1719840, 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 137712 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.806 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:20.807 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:20.808 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:20.809 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:20.810 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 148754, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 31, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:20.811 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:20.812 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:20.813 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:20.814 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 242, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1682989, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 242, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 242, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1682989, 2024-03-08T23:18:20.815 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:20.816 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:20.817 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:20 smithi192 ceph-mon[36700]: pgmap v329: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:21.985 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-08T23:18:21.985 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-08T23:18:21.986 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-08T23:18:21.986 DEBUG:tasks.check_counter:Getting stats from g 2024-03-08T23:18:21.987 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-08T23:18:22.469 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:18:22.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:22 smithi062 ceph-mon[35761]: pgmap v330: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:22.644 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:22 smithi046 ceph-mon[31859]: pgmap v330: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:22.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:22 smithi192 ceph-mon[36700]: pgmap v330: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2024-03-08T23:18:24.645 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:24 smithi046 ceph-mon[31859]: pgmap v331: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:18:24.647 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:24.647 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:24.647 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 4, 2024-03-08T23:18:24.647 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 4, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3024, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 448, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.004682242, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000053718, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.001316078, 2024-03-08T23:18:24.648 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 3, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.001442462, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000480820 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:24.649 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 448 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 2, 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 172, 2024-03-08T23:18:24.650 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 320, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 185328, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 5, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.036206586, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.026042540, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.004259464, 2024-03-08T23:18:24.651 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 169, 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.030745559, 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000181926 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:24.652 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 185328 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:24.653 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 236, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 224, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1715728, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 134768, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.109858305, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.035358929, 2024-03-08T23:18:24.654 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.090242398, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.010433996, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 223, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.023725047, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000106390 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:24.655 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1715728, 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 134768 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-08T23:18:24.656 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:24.657 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:24.658 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:24.659 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:24.659 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:24.660 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:24.661 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 149745, 2024-03-08T23:18:24.662 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 31, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:24.663 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:24.664 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-08T23:18:24.665 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 238, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1682469, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 238, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 238, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1682469, 2024-03-08T23:18:24.666 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:24.667 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:24.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:24 smithi192 ceph-mon[36700]: pgmap v331: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:18:24.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:24 smithi062 ceph-mon[35761]: pgmap v331: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-08T23:18:25.487 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-08T23:18:25.488 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-08T23:18:25.489 DEBUG:tasks.check_counter:Getting stats from j 2024-03-08T23:18:25.489 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-08T23:18:25.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:25 smithi046 ceph-mon[31859]: pgmap v332: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:18:25.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:25 smithi192 ceph-mon[36700]: pgmap v332: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:18:25.890 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:25 smithi062 ceph-mon[35761]: pgmap v332: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-08T23:18:25.974 INFO:teuthology.orchestra.run.smithi046.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:18:27.994 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:27 smithi046 ceph-mon[31859]: pgmap v333: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:28.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:27 smithi062 ceph-mon[35761]: pgmap v333: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:28.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:27 smithi192 ceph-mon[36700]: pgmap v333: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:28.332 INFO:teuthology.orchestra.run.smithi046.stdout:{ 2024-03-08T23:18:28.332 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:28.332 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 0, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 2, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 0, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 0, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:28.333 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.003785998, 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.000000000, 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.000781731, 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:28.334 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:28.335 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 6, 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 179, 2024-03-08T23:18:28.336 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 3344, 2024-03-08T23:18:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 186656, 2024-03-08T23:18:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 5, 2024-03-08T23:18:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.037708008, 2024-03-08T23:18:28.337 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.026721910, 2024-03-08T23:18:28.338 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.004965958, 2024-03-08T23:18:28.338 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:28.338 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:28.338 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 177, 2024-03-08T23:18:28.338 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.028786314, 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000162634 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:28.339 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 186656 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_messages": 243, 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages": 231, 2024-03-08T23:18:28.340 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_bytes": 1717632, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_bytes": 139136, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_total_time": 0.116148225, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_send_time": 0.036220978, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_recv_time": 0.102766753, 2024-03-08T23:18:28.341 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_running_fast_dispatch_time": 0.010699212, 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 230, 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.021900234, 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000095218 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:28.342 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:28.343 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:28.343 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:28.343 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.343 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_recv_encrypted_bytes": 1717632, 2024-03-08T23:18:28.343 INFO:teuthology.orchestra.run.smithi046.stdout: "msgr_send_encrypted_bytes": 139136 2024-03-08T23:18:28.344 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.344 INFO:teuthology.orchestra.run.smithi046.stdout: "cct": { 2024-03-08T23:18:28.344 INFO:teuthology.orchestra.run.smithi046.stdout: "total_workers": 0, 2024-03-08T23:18:28.344 INFO:teuthology.orchestra.run.smithi046.stdout: "unhealthy_workers": 0 2024-03-08T23:18:28.344 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "mempool": { 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:28.345 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:28.346 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:28.347 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:28.348 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:28.349 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:28.349 INFO:teuthology.orchestra.run.smithi046.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:28.349 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_items": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:28.350 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_bytes": 149745, 2024-03-08T23:18:28.351 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_anon_items": 31, 2024-03-08T23:18:28.351 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:28.351 INFO:teuthology.orchestra.run.smithi046.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:28.351 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_bytes": 0, 2024-03-08T23:18:28.352 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_items": 0, 2024-03-08T23:18:28.352 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:28.352 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:28.352 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:28.353 INFO:teuthology.orchestra.run.smithi046.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:28.353 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:28.353 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_items": 0, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "pgmap_items": 0, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:28.354 INFO:teuthology.orchestra.run.smithi046.stdout: "mds_co_items": 2, 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_1_items": 0, 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: "unittest_2_items": 0 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: }, 2024-03-08T23:18:28.355 INFO:teuthology.orchestra.run.smithi046.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "val": 0, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "max": 104857600, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "get_started": 0, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "get": 245, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "get_sum": 1683379, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "get_or_fail_success": 245, 2024-03-08T23:18:28.356 INFO:teuthology.orchestra.run.smithi046.stdout: "take": 0, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "take_sum": 0, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "put": 245, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "put_sum": 1683379, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "wait": { 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "avgcount": 0, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "sum": 0.000000000, 2024-03-08T23:18:28.357 INFO:teuthology.orchestra.run.smithi046.stdout: "avgtime": 0.000000000 2024-03-08T23:18:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:28.358 INFO:teuthology.orchestra.run.smithi046.stdout: } 2024-03-08T23:18:28.358 INFO:teuthology.orchestra.run.smithi046.stdout:} 2024-03-08T23:18:29.224 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-08T23:18:29.225 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-08T23:18:29.225 DEBUG:tasks.check_counter:Getting stats from b 2024-03-08T23:18:29.226 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-08T23:18:29.488 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:18:29.862 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:29 smithi062 ceph-mon[35761]: pgmap v334: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:29.975 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:29 smithi046 ceph-mon[31859]: pgmap v334: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:30.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:29 smithi192 ceph-mon[36700]: pgmap v334: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:31.265 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-08T23:18:31.265 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:31.265 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 5461, 2024-03-08T23:18:31.265 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 5792, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 3220731, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 12968349, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 162, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.531780695, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.172337801, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.233558904, 2024-03-08T23:18:31.266 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.108409196, 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 5792, 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.468298312, 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000080852 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 4373, 2024-03-08T23:18:31.267 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000814410, 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000186 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 1850720, 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 172416 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 19684, 2024-03-08T23:18:31.268 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 38114, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 7546056, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 35596594, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 17, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 1.669787306, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.719889480, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.976797073, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.152361560, 2024-03-08T23:18:31.269 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 38111, 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 1.552022927, 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000040723 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 36965, 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.007049290, 2024-03-08T23:18:31.270 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000190 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 6415888, 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 18666672 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 9716, 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 8257, 2024-03-08T23:18:31.271 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 3695327, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 16584770, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 12, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 7, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.747554308, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.281118075, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.284654359, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.185153739, 2024-03-08T23:18:31.272 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 8255, 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.778442896, 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000094299 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 7527, 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.001340471, 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000178 2024-03-08T23:18:31.273 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 333264, 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 99232 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 1, 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-03-08T23:18:31.274 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "finisher-MDSRank": { 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "queue_len": 0, 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "complete_latency": { 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 6344, 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 8.258417119, 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.001301768 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.275 INFO:teuthology.orchestra.run.smithi062.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "queue_len": 0, 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "complete_latency": { 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 3429, 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.474397046, 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000138348 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.276 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "request": 9753, 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "reply": 9553, 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "reply_latency": { 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 9553, 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 32.860900101, 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.003439851 2024-03-08T23:18:31.277 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "slow_reply": 0, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "forward": 200, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_commit": 194, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_split": 22, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_merge": 0, 2024-03-08T23:18:31.278 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes": 1534, 2024-03-08T23:18:31.279 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_top": 723, 2024-03-08T23:18:31.279 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_bottom": 749, 2024-03-08T23:18:31.279 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_pin_tail": 62, 2024-03-08T23:18:31.279 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_pinned": 502, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_expired": 4376, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_with_caps": 138, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": 139, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "subtrees": 63, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse": 11230, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_hit": 10546, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_forward": 194, 2024-03-08T23:18:31.280 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_discover": 4, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_lock": 2, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "load_cent": 0, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "q": 0, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "exported": 108, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "exported_inodes": 5021, 2024-03-08T23:18:31.281 INFO:teuthology.orchestra.run.smithi062.stdout: "imported": 15, 2024-03-08T23:18:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "imported_inodes": 688, 2024-03-08T23:18:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_backtrace_fetch": 6215, 2024-03-08T23:18:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:31.282 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rfiles": 3452, 2024-03-08T23:18:31.283 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rbytes": 26509461, 2024-03-08T23:18:31.283 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rsnaps": 6, 2024-03-08T23:18:31.283 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_backtrace_fetch": 6215, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dir_inodes": 732, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dir_base_inodes": 19, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dirfrag_rstats": 713, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_file_inodes": 5483, 2024-03-08T23:18:31.284 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_inode_file_caps": 16, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_revoke": 317, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_grant": 8125, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_trunc": 6, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_caps": 5136, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_caps_dirty": 4256, 2024-03-08T23:18:31.285 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_cap_release": 345, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "process_request_cap_release": 2955 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache": { 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_update": 173, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_update_receipt": 9, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_try_discover": 6, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_send_discover": 7, 2024-03-08T23:18:31.286 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_handle_discover": 368, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays": 0, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_created": 1885, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_enqueued": 1883, 2024-03-08T23:18:31.287 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_reintegrated": 2, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_migrated": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "recovery_started": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "recovery_completed": 0, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_enqueue_scrub": 19, 2024-03-08T23:18:31.288 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_exportdir": 109, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_flush": 0, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_fragmentdir": 22, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_inodestats": 0 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log": { 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "evlrg": 0, 2024-03-08T23:18:31.289 INFO:teuthology.orchestra.run.smithi062.stdout: "evadd": 15708, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "evex": 0, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "evtrm": 0, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "ev": 15708, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "evexg": 0, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "evexd": 0, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "segadd": 19, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "segex": 0, 2024-03-08T23:18:31.290 INFO:teuthology.orchestra.run.smithi062.stdout: "segtrm": 0, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "seg": 19, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "segmjr": 3, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "segexg": 0, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "segexd": 0, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "expos": 4194304, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "wrpos": 41289465, 2024-03-08T23:18:31.291 INFO:teuthology.orchestra.run.smithi062.stdout: "rdpos": 0, 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "jlat": { 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 694, 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 8.168270720, 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.011769842 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "replayed": 0 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_mem": { 2024-03-08T23:18:31.292 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 1109, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "ino+": 6454, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "ino-": 5345, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dir": 206, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dir+": 290, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dir-": 84, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dn": 1534, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dn+": 7859, 2024-03-08T23:18:31.293 INFO:teuthology.orchestra.run.smithi062.stdout: "dn-": 6325, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "cap": 139, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "cap+": 6066, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "cap-": 5927, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "rss": 80464, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "heap": 215420 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_server": { 2024-03-08T23:18:31.294 INFO:teuthology.orchestra.run.smithi062.stdout: "dispatch_client_request": 10141, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "dispatch_server_request": 102, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_request": 9753, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_session": 146, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_peer_request": 192, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "req_create_latency": { 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 5710, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 20.277185662, 2024-03-08T23:18:31.295 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.003551170 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getattr_latency": { 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 238, 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 1.328120851, 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.005580339 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:31.296 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "req_link_latency": { 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 2, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.003935558, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.001967779 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.297 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookup_latency": { 2024-03-08T23:18:31.298 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 1146, 2024-03-08T23:18:31.298 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 5.256952075, 2024-03-08T23:18:31.298 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.004587218 2024-03-08T23:18:31.298 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.298 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:31.299 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupino_latency": { 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 1, 2024-03-08T23:18:31.300 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000262119, 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000262119 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupname_latency": { 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.301 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.302 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.303 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.303 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lssnap_latency": { 2024-03-08T23:18:31.303 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 11, 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.002452305, 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000222936 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mkdir_latency": { 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 82, 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.232234911, 2024-03-08T23:18:31.304 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.002832133 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mknod_latency": { 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mksnap_latency": { 2024-03-08T23:18:31.305 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 11, 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.080188528, 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.007289866 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "req_open_latency": { 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 334, 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.090426661, 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000270738 2024-03-08T23:18:31.306 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "req_readdir_latency": { 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 71, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.563838300, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.007941384 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rename_latency": { 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 63, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.139290640, 2024-03-08T23:18:31.307 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.002210962 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:31.308 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmdir_latency": { 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 22, 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.838213010, 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.038100591 2024-03-08T23:18:31.309 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 5, 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.026973966, 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.005394793 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.310 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setattr_latency": { 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 8, 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.041198648, 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.005149831 2024-03-08T23:18:31.311 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 8, 2024-03-08T23:18:31.312 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.029149521, 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.003643690 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setlayout_latency": { 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setxattr_latency": { 2024-03-08T23:18:31.313 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 8, 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.024395727, 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.003049465 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "req_symlink_latency": { 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 5, 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.003907802, 2024-03-08T23:18:31.314 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000781560 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "req_unlink_latency": { 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 1828, 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 3.922173817, 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.002145609 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:31.315 INFO:teuthology.orchestra.run.smithi062.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_sessions": { 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "session_count": 2, 2024-03-08T23:18:31.316 INFO:teuthology.orchestra.run.smithi062.stdout: "session_add": 6, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "session_remove": 4, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "sessions_open": 2, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "sessions_stale": 0, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "total_load": 56, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "average_load": 28, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "avg_session_uptime": 644, 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:31.317 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:31.318 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:31.321 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 355594, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 1203, 2024-03-08T23:18:31.322 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 1408, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 16, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:31.323 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 9424, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 181, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 4113904, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 52841, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:31.324 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter": { 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "op_active": 0, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "op_laggy": 0, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "op_send": 11679, 2024-03-08T23:18:31.325 INFO:teuthology.orchestra.run.smithi062.stdout: "op_send_bytes": 40246700, 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "op_resend": 32, 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "op_reply": 11679, 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "op_latency": { 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 11679, 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 52.419142380, 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.004488324 2024-03-08T23:18:31.326 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "op_inflight": 0, 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "oplen_avg": { 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 11679, 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 11988 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "op": 11679, 2024-03-08T23:18:31.327 INFO:teuthology.orchestra.run.smithi062.stdout: "op_r": 6300, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "op_w": 5379, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "op_rmw": 0, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "op_pg": 0, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_stat": 0, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_create": 11, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_read": 66, 2024-03-08T23:18:31.328 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_write": 741, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_writefull": 60, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_writesame": 0, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_append": 0, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_zero": 0, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_truncate": 6, 2024-03-08T23:18:31.329 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_delete": 4287, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_mapext": 0, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_getxattr": 6234, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_setxattr": 22, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:31.330 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_call": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_watch": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_notify": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_pgls": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_other": 6, 2024-03-08T23:18:31.331 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_active": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_send": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_resend": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_ping": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_active": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_send": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_resend": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_active": 0, 2024-03-08T23:18:31.332 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_send": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_resend": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_active": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_send": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_resend": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "command_active": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "command_send": 0, 2024-03-08T23:18:31.333 INFO:teuthology.orchestra.run.smithi062.stdout: "command_resend": 0, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "map_epoch": 104, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "map_full": 0, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "map_inc": 20, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_sessions": 12, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_session_open": 12, 2024-03-08T23:18:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_session_close": 0, 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_laggy": 0, 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_wr": 510, 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_rd": 0, 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_del": 45 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "oft": { 2024-03-08T23:18:31.335 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_objs": 1, 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_updates": 0, 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_removes": 0 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: "osd-slow-ops": { 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: "slow_ops_count": 0 2024-03-08T23:18:31.336 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "purge_queue": { 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_ops_high_water": 17, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing": 0, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_high_water": 10, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executed_ops": 3751, 2024-03-08T23:18:31.337 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executed": 1883, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 34857, 2024-03-08T23:18:31.338 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 10491699, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 34857, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 34857, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 10491699, 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:31.339 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:31.340 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 11679, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 40451867, 2024-03-08T23:18:31.341 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 11679, 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 40451867, 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.342 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 1024, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:31.343 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 11679, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 11679, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 11679, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 11679, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.344 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 3758096384, 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:31.345 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 1883, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 205167, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 1883, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 47, 2024-03-08T23:18:31.346 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 205167, 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-write_buf_throttle-0x561c78a006e0": { 2024-03-08T23:18:31.347 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 3758096384, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 15708, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 37095161, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 15708, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:31.348 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 694, 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 37095161, 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:31.349 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.350 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:31.350 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-08T23:18:32.115 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:31 smithi192 ceph-mon[36700]: pgmap v335: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:32.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:31 smithi062 ceph-mon[35761]: pgmap v335: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:32.179 INFO:tasks.check_counter:Daemon mds.b mds.exported=108 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds.imported=15 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=173 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=9 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-08T23:18:32.180 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=22 2024-03-08T23:18:32.180 DEBUG:tasks.check_counter:Getting stats from e 2024-03-08T23:18:32.181 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-08T23:18:32.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:31 smithi046 ceph-mon[31859]: pgmap v335: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:32.441 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:18:34.041 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:33 smithi062 ceph-mon[35761]: pgmap v336: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:34.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:33 smithi046 ceph-mon[31859]: pgmap v336: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:34.287 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 243, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 231, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 1717632, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 139136, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:34.288 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.052542627, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.015345391, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.038601081, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.006295021, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 230, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.011746309, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000051070 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.289 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 1717632, 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 139136 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.290 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 7, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 181, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 3504, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 264800, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 6, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.012679953, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.008029096, 2024-03-08T23:18:34.291 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.002077914, 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 179, 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.039245408, 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000219248 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:34.292 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 3504, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 264800 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 0, 2024-03-08T23:18:34.293 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 2, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 0, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 0, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.000278111, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000000000, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.000195189, 2024-03-08T23:18:34.294 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:34.295 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-03-08T23:18:34.296 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:34.297 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:34.298 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:34.299 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:34.300 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148692, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 30, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:34.301 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:34.302 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:34.303 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:34.304 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 246, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 1683406, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 246, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 246, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 1683406, 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:34.305 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:34.306 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:34.306 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-08T23:18:34.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:33 smithi192 ceph-mon[36700]: pgmap v336: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:34.979 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-03-08T23:18:34.980 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-03-08T23:18:34.980 DEBUG:tasks.check_counter:Getting stats from h 2024-03-08T23:18:34.981 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-08T23:18:35.236 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:18:36.053 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:35 smithi062 ceph-mon[35761]: pgmap v337: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:36.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:35 smithi046 ceph-mon[31859]: pgmap v337: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:36.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:35 smithi192 ceph-mon[36700]: pgmap v337: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:36.912 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 1947, 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 1949, 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 9448189, 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 619153, 2024-03-08T23:18:36.913 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 6, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.176707143, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.058208665, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.048150058, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.053768085, 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:36.914 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 1947, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.068182035, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000035019 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.915 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 478, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 649, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 119520, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 610670, 2024-03-08T23:18:36.916 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 7, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.057587384, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.018747777, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.013735976, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.018791361, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 647, 2024-03-08T23:18:36.917 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.042863255, 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000066249 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.918 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 460512 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 270, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 250, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 1775692, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 141678, 2024-03-08T23:18:36.919 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 3, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 2, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.053289564, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.015955402, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.041530666, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.005893808, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 249, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.013797207, 2024-03-08T23:18:36.920 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000055410 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.921 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 1775424, 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 141360 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 1, 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-03-08T23:18:36.922 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "finisher-MDSRank": { 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "queue_len": 0, 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "complete_latency": { 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 4062, 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.111371030, 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000027417 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.923 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "queue_len": 0, 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "complete_latency": { 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.924 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": { 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "request": 0, 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "reply": 0, 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "reply_latency": { 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.925 INFO:teuthology.orchestra.run.smithi062.stdout: "slow_reply": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "forward": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_commit": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_split": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_merge": 0, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes": 3818, 2024-03-08T23:18:36.926 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_top": 1, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_bottom": 3817, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_pin_tail": 0, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_pinned": 3816, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_expired": 0, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "inodes_with_caps": 0, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "caps": 0, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "subtrees": 65, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse": 0, 2024-03-08T23:18:36.927 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_hit": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_forward": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_discover": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "traverse_lock": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "load_cent": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "q": 0, 2024-03-08T23:18:36.928 INFO:teuthology.orchestra.run.smithi062.stdout: "exported": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "exported_inodes": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "imported": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "imported_inodes": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_backtrace_fetch": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rfiles": 0, 2024-03-08T23:18:36.929 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rbytes": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "root_rsnaps": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dir_inodes": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T23:18:36.930 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "scrub_file_inodes": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_inode_file_caps": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_revoke": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_grant": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_caps": 0, 2024-03-08T23:18:36.931 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_caps_dirty": 0, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_cap_release": 0, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "process_request_cap_release": 0 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_cache": { 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_update": 0, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_update_receipt": 0, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_try_discover": 0, 2024-03-08T23:18:36.932 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_send_discover": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "dir_handle_discover": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_created": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_enqueued": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_reintegrated": 0, 2024-03-08T23:18:36.933 INFO:teuthology.orchestra.run.smithi062.stdout: "strays_migrated": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "recovery_started": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "recovery_completed": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_exportdir": 0, 2024-03-08T23:18:36.934 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_flush": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_fragmentdir": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "ireq_inodestats": 0 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_log": { 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "evlrg": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "evadd": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "evex": 0, 2024-03-08T23:18:36.935 INFO:teuthology.orchestra.run.smithi062.stdout: "evtrm": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "ev": 4552, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "evexg": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "evexd": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "segadd": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "segex": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "segtrm": 0, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "seg": 8, 2024-03-08T23:18:36.936 INFO:teuthology.orchestra.run.smithi062.stdout: "segmjr": 2, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "segexg": 0, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "segexd": 0, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "expos": 4194304, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "wrpos": 13074237, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "rdpos": 13073663, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "jlat": { 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.937 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "replayed": 4552 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_mem": { 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "ino": 2557, 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "ino+": 3509, 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "ino-": 952, 2024-03-08T23:18:36.938 INFO:teuthology.orchestra.run.smithi062.stdout: "dir": 198, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "dir+": 236, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "dir-": 38, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "dn": 3818, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "dn+": 4466, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "dn-": 648, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "cap": 0, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "cap+": 0, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "cap-": 0, 2024-03-08T23:18:36.939 INFO:teuthology.orchestra.run.smithi062.stdout: "rss": 59800, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "heap": 215420 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_server": { 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "dispatch_client_request": 0, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "dispatch_server_request": 0, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_request": 0, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_client_session": 0, 2024-03-08T23:18:36.940 INFO:teuthology.orchestra.run.smithi062.stdout: "handle_peer_request": 0, 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "req_create_latency": { 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getattr_latency": { 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.941 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "req_link_latency": { 2024-03-08T23:18:36.942 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookup_latency": { 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.943 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupino_latency": { 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.944 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupname_latency": { 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.945 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.946 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "req_lssnap_latency": { 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mkdir_latency": { 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.947 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.948 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.948 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mknod_latency": { 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "req_mksnap_latency": { 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.949 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "req_open_latency": { 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.950 INFO:teuthology.orchestra.run.smithi062.stdout: "req_readdir_latency": { 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rename_latency": { 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.951 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.952 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmdir_latency": { 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:36.953 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.954 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setattr_latency": { 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.955 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setlayout_latency": { 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.956 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "req_setxattr_latency": { 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.957 INFO:teuthology.orchestra.run.smithi062.stdout: "req_symlink_latency": { 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "req_unlink_latency": { 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.958 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.959 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_sessions": { 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "session_count": 1, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "session_add": 2, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "session_remove": 1, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "sessions_open": 1, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "sessions_stale": 0, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "total_load": 0, 2024-03-08T23:18:36.960 INFO:teuthology.orchestra.run.smithi062.stdout: "average_load": 0, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "avg_session_uptime": 2327, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:36.961 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:36.962 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:36.962 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:36.962 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:36.962 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:36.962 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:36.963 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:36.964 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:36.965 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 285480, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 2609, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-03-08T23:18:36.966 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 9424, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 181, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:36.967 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 8762032, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 78013, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:36.968 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "objecter": { 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_active": 0, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_laggy": 0, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_send": 2420, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_send_bytes": 21, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_resend": 3, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_reply": 2420, 2024-03-08T23:18:36.969 INFO:teuthology.orchestra.run.smithi062.stdout: "op_latency": { 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 2420, 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 1.591973642, 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000657840 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "op_inflight": 0, 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "oplen_avg": { 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 2420, 2024-03-08T23:18:36.970 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 2421 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "op": 2420, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "op_r": 2420, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "op_w": 0, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "op_rmw": 0, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "op_pg": 0, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_stat": 1555, 2024-03-08T23:18:36.971 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_create": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_read": 863, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_write": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_writefull": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_writesame": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_append": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_zero": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_truncate": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_delete": 0, 2024-03-08T23:18:36.972 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_mapext": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_getxattr": 1, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_setxattr": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:36.973 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_call": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_watch": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_notify": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_pgls": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osdop_other": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_active": 0, 2024-03-08T23:18:36.974 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_send": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_resend": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "linger_ping": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_active": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_send": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolop_resend": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_active": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_send": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "poolstat_resend": 0, 2024-03-08T23:18:36.975 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_active": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_send": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "statfs_resend": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "command_active": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "command_send": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "command_resend": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "map_epoch": 104, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "map_full": 0, 2024-03-08T23:18:36.976 INFO:teuthology.orchestra.run.smithi062.stdout: "map_inc": 15, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_sessions": 5, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_session_open": 5, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_session_close": 0, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_laggy": 0, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_wr": 0, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_rd": 2, 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_del": 0 2024-03-08T23:18:36.977 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "oft": { 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_objs": 0, 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_updates": 0, 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "omap_total_removes": 0 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "osd-slow-ops": { 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: "slow_ops_count": 0 2024-03-08T23:18:36.978 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "purge_queue": { 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executing_high_water": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executed_ops": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_executed": 0, 2024-03-08T23:18:36.979 INFO:teuthology.orchestra.run.smithi062.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 2691, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 11084454, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:36.980 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 2691, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 2691, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 11084454, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.981 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:36.982 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 2420, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 8882663, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 2420, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 8882663, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.983 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 1024, 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:36.984 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 2420, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 2420, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 2420, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 2420, 2024-03-08T23:18:36.985 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:36.986 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 3758096384, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 0, 2024-03-08T23:18:36.987 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 0, 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-write_buf_throttle-0x560eedc40960": { 2024-03-08T23:18:36.988 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 3758096384, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:36.989 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 0, 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 0, 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:36.990 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-08T23:18:37.692 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-03-08T23:18:37.693 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-08T23:18:37.693 DEBUG:tasks.check_counter:Getting stats from k 2024-03-08T23:18:37.694 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-08T23:18:37.942 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:18:38.064 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:37 smithi062 ceph-mon[35761]: pgmap v338: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:38.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:37 smithi046 ceph-mon[31859]: pgmap v338: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:38.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:37 smithi192 ceph-mon[36700]: pgmap v338: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:39.650 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-03-08T23:18:39.650 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:39.650 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 0, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 2, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 0, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 0, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.001066243, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.000000000, 2024-03-08T23:18:39.651 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.000124870, 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:39.652 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 2, 2024-03-08T23:18:39.653 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 180, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 320, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 186736, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 5, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.013186740, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.009009958, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.001392977, 2024-03-08T23:18:39.654 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 177, 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.016950222, 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000095763 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:39.655 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 186736 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_messages": 251, 2024-03-08T23:18:39.656 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages": 237, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_bytes": 1721744, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_bytes": 142080, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_total_time": 0.050308287, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_send_time": 0.015266187, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_recv_time": 0.039069429, 2024-03-08T23:18:39.657 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_running_fast_dispatch_time": 0.005768361, 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 237, 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.013152347, 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000055495 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:39.658 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_recv_encrypted_bytes": 1721744, 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "msgr_send_encrypted_bytes": 142080 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "cct": { 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "total_workers": 0, 2024-03-08T23:18:39.659 INFO:teuthology.orchestra.run.smithi062.stdout: "unhealthy_workers": 0 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "mempool": { 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:39.660 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:39.661 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:39.662 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:39.663 INFO:teuthology.orchestra.run.smithi062.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_items": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_bytes": 148670, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_anon_items": 30, 2024-03-08T23:18:39.664 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_bytes": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_items": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:39.665 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_items": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "pgmap_items": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "mds_co_items": 2, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:39.666 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_1_items": 0, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "unittest_2_items": 0 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 0, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "max": 104857600, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "get_started": 0, 2024-03-08T23:18:39.667 INFO:teuthology.orchestra.run.smithi062.stdout: "get": 249, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "get_sum": 1683899, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "get_or_fail_success": 249, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "take": 0, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "take_sum": 0, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "put": 249, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "put_sum": 1683899, 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "wait": { 2024-03-08T23:18:39.668 INFO:teuthology.orchestra.run.smithi062.stdout: "avgcount": 0, 2024-03-08T23:18:39.669 INFO:teuthology.orchestra.run.smithi062.stdout: "sum": 0.000000000, 2024-03-08T23:18:39.669 INFO:teuthology.orchestra.run.smithi062.stdout: "avgtime": 0.000000000 2024-03-08T23:18:39.669 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:39.669 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-03-08T23:18:39.669 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-03-08T23:18:40.086 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:39 smithi062 ceph-mon[35761]: pgmap v339: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:40.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:39 smithi046 ceph-mon[31859]: pgmap v339: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:40.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:39 smithi192 ceph-mon[36700]: pgmap v339: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:40.314 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-08T23:18:40.315 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-08T23:18:40.315 DEBUG:tasks.check_counter:Getting stats from c 2024-03-08T23:18:40.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-08T23:18:40.566 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:18:42.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:41 smithi062 ceph-mon[35761]: pgmap v340: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:42.209 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:41 smithi192 ceph-mon[36700]: pgmap v340: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:42.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:41 smithi046 ceph-mon[31859]: pgmap v340: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:42.478 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-08T23:18:42.478 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:42.479 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 2205, 2024-03-08T23:18:42.479 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 2207, 2024-03-08T23:18:42.479 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 12559375, 2024-03-08T23:18:42.479 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 701200, 2024-03-08T23:18:42.479 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 7, 2024-03-08T23:18:42.480 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:42.480 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.208315138, 2024-03-08T23:18:42.480 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.065960043, 2024-03-08T23:18:42.480 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.057482509, 2024-03-08T23:18:42.480 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.065106040, 2024-03-08T23:18:42.481 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:42.481 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2205, 2024-03-08T23:18:42.481 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.085829526, 2024-03-08T23:18:42.481 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000038924 2024-03-08T23:18:42.481 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:42.482 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 90, 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 266, 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 16802783, 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 569634, 2024-03-08T23:18:42.483 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 8, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.025700271, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.009538870, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.011468241, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.002013354, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 263, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.032701576, 2024-03-08T23:18:42.484 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000124340 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.485 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 541584 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 584, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 562, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 10246628, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 241161, 2024-03-08T23:18:42.486 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.086965911, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.026770677, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.053549887, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.015422697, 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:42.487 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 562, 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.026113589, 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000046465 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.488 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1779152, 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 143216 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.489 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4319, 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.121824947, 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000028206 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.490 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.491 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 0, 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 0, 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-08T23:18:42.492 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.493 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.493 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.493 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.493 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-08T23:18:42.493 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-08T23:18:42.494 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 2707, 2024-03-08T23:18:42.495 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 227, 2024-03-08T23:18:42.495 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 2480, 2024-03-08T23:18:42.495 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-08T23:18:42.495 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 2383, 2024-03-08T23:18:42.496 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-08T23:18:42.496 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-08T23:18:42.496 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-08T23:18:42.496 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 11, 2024-03-08T23:18:42.496 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 0, 2024-03-08T23:18:42.497 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 0, 2024-03-08T23:18:42.497 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-08T23:18:42.497 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-08T23:18:42.497 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:42.497 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:42.498 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-08T23:18:42.498 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-08T23:18:42.498 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-08T23:18:42.498 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-08T23:18:42.498 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-08T23:18:42.499 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-08T23:18:42.499 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-08T23:18:42.499 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:42.499 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-08T23:18:42.499 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:42.500 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 3452, 2024-03-08T23:18:42.500 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 26509461, 2024-03-08T23:18:42.500 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 6, 2024-03-08T23:18:42.500 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T23:18:42.500 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:42.501 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:42.501 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:42.501 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-08T23:18:42.501 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T23:18:42.501 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T23:18:42.502 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-08T23:18:42.502 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-08T23:18:42.502 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 0, 2024-03-08T23:18:42.502 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-08T23:18:42.503 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 0 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-08T23:18:42.504 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-08T23:18:42.505 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-08T23:18:42.506 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-08T23:18:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-08T23:18:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 0, 2024-03-08T23:18:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-08T23:18:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-08T23:18:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 15708, 2024-03-08T23:18:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-08T23:18:42.509 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-08T23:18:42.509 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 0, 2024-03-08T23:18:42.509 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-08T23:18:42.509 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-08T23:18:42.509 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 19, 2024-03-08T23:18:42.510 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 3, 2024-03-08T23:18:42.510 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-08T23:18:42.510 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-08T23:18:42.510 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-08T23:18:42.510 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 41289465, 2024-03-08T23:18:42.511 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 41289329, 2024-03-08T23:18:42.511 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-08T23:18:42.511 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.511 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 15708 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-08T23:18:42.512 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 402, 2024-03-08T23:18:42.513 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 6439, 2024-03-08T23:18:42.513 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 6037, 2024-03-08T23:18:42.513 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 65, 2024-03-08T23:18:42.513 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 291, 2024-03-08T23:18:42.513 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 226, 2024-03-08T23:18:42.514 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 2707, 2024-03-08T23:18:42.514 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 7503, 2024-03-08T23:18:42.514 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 4796, 2024-03-08T23:18:42.514 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 0, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 0, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 59236, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 215420 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 0, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-08T23:18:42.515 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 0, 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 0, 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.516 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.517 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.518 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.519 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.520 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.521 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-08T23:18:42.522 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-08T23:18:42.523 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.524 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-08T23:18:42.525 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.526 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.527 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.528 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.529 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.530 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-08T23:18:42.531 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.534 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-08T23:18:42.535 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.536 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.537 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.538 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.538 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-08T23:18:42.538 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.538 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.538 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.539 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.539 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:42.539 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:42.539 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:42.539 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-08T23:18:42.540 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 2, 2024-03-08T23:18:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 6, 2024-03-08T23:18:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 4, 2024-03-08T23:18:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 4, 2024-03-08T23:18:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-08T23:18:42.541 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 0, 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 0, 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 4011, 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-08T23:18:42.542 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:42.543 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:42.543 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:42.543 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:42.543 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:42.544 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:42.545 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:42.545 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:42.545 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:42.545 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:42.545 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:42.546 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:42.546 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:42.546 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:42.546 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:42.546 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:42.547 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-08T23:18:42.548 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 221611, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 475, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:42.549 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:42.550 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 9424, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 181, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:42.551 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 2953447, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 23441, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-08T23:18:42.552 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 2601, 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 26, 2024-03-08T23:18:42.553 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 4, 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 2601, 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2601, 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.655966904, 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000636665 2024-03-08T23:18:42.554 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2601, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2602 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 2601, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2601, 2024-03-08T23:18:42.555 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 1649, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 949, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-08T23:18:42.556 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 2, 2024-03-08T23:18:42.557 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:42.558 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-08T23:18:42.559 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-08T23:18:42.560 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 104, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 15, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 7, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 7, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-08T23:18:42.561 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 0, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 2, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 0 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 0, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-08T23:18:42.562 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T23:18:42.563 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:42.564 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 2875, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 39333129, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 2875, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:42.565 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2875, 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 39333129, 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.566 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:42.567 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2601, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 37100976, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2601, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 37100976, 2024-03-08T23:18:42.568 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.569 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:42.570 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2601, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 2601, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2601, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 2601, 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:42.571 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:42.572 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:42.573 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.574 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x55ede8a5c960": { 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:42.575 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:42.576 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:42.577 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.578 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:42.578 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-08T23:18:43.211 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-03-08T23:18:43.212 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-03-08T23:18:43.213 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-08T23:18:43.213 DEBUG:tasks.check_counter:Getting stats from f 2024-03-08T23:18:43.213 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-08T23:18:43.451 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:18:44.141 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:43 smithi062 ceph-mon[35761]: pgmap v341: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:44.185 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:43 smithi192 ceph-mon[36700]: pgmap v341: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:44.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:43 smithi046 ceph-mon[31859]: pgmap v341: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:45.030 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-08T23:18:45.030 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:45.030 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 317, 2024-03-08T23:18:45.030 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 319, 2024-03-08T23:18:45.030 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 4278689, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 100870, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 6, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 2, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.035721303, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.008949770, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.012520712, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.010612711, 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:45.031 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 317, 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.013760954, 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000043409 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.032 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 769, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 945, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 3525245, 2024-03-08T23:18:45.033 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 707703, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 7, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.094611009, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.030384383, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.025556561, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.028532243, 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:45.034 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 942, 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.053334799, 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000056618 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.035 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 463792 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 1617, 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 1595, 2024-03-08T23:18:45.036 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 2182706, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 570424, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.214717529, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.059972976, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.086121517, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.065221963, 2024-03-08T23:18:45.037 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1595, 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.072606763, 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000045521 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.038 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1779536, 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 144304 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-08T23:18:45.039 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 4048, 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.133675404, 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000033022 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.040 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.041 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 0, 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 0, 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.042 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 2668, 2024-03-08T23:18:45.043 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 2, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 2666, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 2665, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 0, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 0, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 0, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 58, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 0, 2024-03-08T23:18:45.044 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-08T23:18:45.045 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 0, 2024-03-08T23:18:45.046 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T23:18:45.047 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 0, 2024-03-08T23:18:45.048 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 0 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 0, 2024-03-08T23:18:45.049 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-08T23:18:45.050 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-08T23:18:45.051 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 0, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-08T23:18:45.052 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 4466, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 0, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 7, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-08T23:18:45.053 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 11716301, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 11715727, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.054 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 4466 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 1574, 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 2266, 2024-03-08T23:18:45.055 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 692, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 70, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 70, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 0, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 2668, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 2668, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 0, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 0, 2024-03-08T23:18:45.056 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 0, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 0, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 57548, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 215420 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 0, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 0, 2024-03-08T23:18:45.057 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 0, 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 0, 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 0, 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.058 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:45.059 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.060 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.061 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-08T23:18:45.062 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.063 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.064 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.065 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.065 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-08T23:18:45.065 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.065 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.066 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-08T23:18:45.067 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.068 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.069 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.070 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.071 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.072 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.073 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.074 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.075 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:45.076 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 4, 2024-03-08T23:18:45.077 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 3, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 3, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 0, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 0, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 5693, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.078 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:45.079 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:45.080 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:45.081 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:45.082 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 256118, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 1622, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:45.083 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 9424, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 181, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-08T23:18:45.084 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 5545161, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 48443, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-08T23:18:45.085 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 2424, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 21, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 3, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 2424, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2424, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2.155956825, 2024-03-08T23:18:45.086 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000889421 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2424, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 2425 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 2424, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 2424, 2024-03-08T23:18:45.087 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 1571, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 851, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 0, 2024-03-08T23:18:45.088 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:45.089 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 1, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-08T23:18:45.090 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-08T23:18:45.091 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-08T23:18:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 104, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 15, 2024-03-08T23:18:45.093 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 6, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 6, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 0, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 2, 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 0 2024-03-08T23:18:45.094 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 0, 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-08T23:18:45.095 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 0, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 0, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 0, 2024-03-08T23:18:45.096 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 0, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 2699, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 9726765, 2024-03-08T23:18:45.097 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 2699, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2699, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 9726765, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.098 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:45.099 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2424, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 7523014, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2424, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 7523014, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.100 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:45.101 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 2424, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 2424, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 2424, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 2424, 2024-03-08T23:18:45.102 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:45.103 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-08T23:18:45.104 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x55b92f900960": { 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:45.105 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:45.106 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 0, 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 0, 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:45.107 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-08T23:18:45.807 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-08T23:18:45.807 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-08T23:18:45.807 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-03-08T23:18:45.807 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-03-08T23:18:45.808 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-03-08T23:18:45.808 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-03-08T23:18:45.808 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-03-08T23:18:45.808 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-08T23:18:45.808 DEBUG:tasks.check_counter:Getting stats from i 2024-03-08T23:18:45.808 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-08T23:18:46.049 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:18:46.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:45 smithi192 ceph-mon[36700]: pgmap v342: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:46.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:45 smithi062 ceph-mon[35761]: pgmap v342: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:46.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:45 smithi046 ceph-mon[31859]: pgmap v342: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:47.851 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 13335, 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 16639, 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 5743855, 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 10732306, 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 45, 2024-03-08T23:18:47.852 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 4, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.867894655, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.325797038, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.435322744, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.151198641, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 16639, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.461478307, 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000027734 2024-03-08T23:18:47.853 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 12876, 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.002239104, 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000173 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1551232, 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 4298448 2024-03-08T23:18:47.854 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 9046, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 9091, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 2300351, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 6235252, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 15, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 5, 2024-03-08T23:18:47.855 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.575652480, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.231447649, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.123878040, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.180736685, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 9089, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.267558391, 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000029437 2024-03-08T23:18:47.856 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 19, 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000003004, 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000158 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 5040, 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 146592 2024-03-08T23:18:47.857 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 3134, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 3138, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 3168132, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 1600091, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 13, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 7, 2024-03-08T23:18:47.858 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.229584178, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.073989128, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.104721987, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.055548969, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 3135, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.110305494, 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000035185 2024-03-08T23:18:47.859 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 857, 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000157558, 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000183 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1791344, 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 486432 2024-03-08T23:18:47.860 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 1, 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-MDSRank": { 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:47.861 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 12864, 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 3.146987761, 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000244635 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "finisher-PurgeQueue": { 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "queue_len": 0, 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "complete_latency": { 2024-03-08T23:18:47.862 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1280, 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.092349909, 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000072148 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "mds": { 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "request": 2466, 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "reply": 2298, 2024-03-08T23:18:47.863 INFO:teuthology.orchestra.run.smithi192.stdout: "reply_latency": { 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2298, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 1.850478407, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000805256 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_reply": 0, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "forward": 168, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_complete": 0, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_fetch_keys": 0, 2024-03-08T23:18:47.864 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_commit": 11, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_split": 0, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_merge": 0, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes": 2000, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_top": 12, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_bottom": 1988, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pin_tail": 0, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_pinned": 1983, 2024-03-08T23:18:47.865 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_expired": 8, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "inodes_with_caps": 43, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "caps": 43, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "subtrees": 60, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse": 3818, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_hit": 3546, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_forward": 168, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_discover": 69, 2024-03-08T23:18:47.866 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_dir_fetch": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_remote_ino": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "traverse_lock": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "load_cent": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "q": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "exported": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "exported_inodes": 0, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "imported": 56, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "imported_inodes": 1609, 2024-03-08T23:18:47.867 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_dir_fetch": 0, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_backtrace_fetch": 16814, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "openino_peer_discover": 0, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rfiles": 3452, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rbytes": 26509461, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "root_rsnaps": 6, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_fetch": 16814, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_set_tag": 0, 2024-03-08T23:18:47.868 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_backtrace_repaired": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_inotable_repaired": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_inodes": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dir_base_inodes": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_dirfrag_rstats": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "scrub_file_inodes": 16814, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_inode_file_caps": 0, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_revoke": 8, 2024-03-08T23:18:47.869 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_grant": 1208, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_trunc": 0, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps": 1418, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_caps_dirty": 1409, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_cap_release": 17, 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: "process_request_cap_release": 1390 2024-03-08T23:18:47.870 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_cache": { 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update": 45, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_update_receipt": 318, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_try_discover": 170, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_send_discover": 279, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "dir_handle_discover": 45, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays": 0, 2024-03-08T23:18:47.871 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_delayed": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "num_strays_enqueuing": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_created": 692, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_enqueued": 692, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_reintegrated": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "strays_migrated": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_processing": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_enqueued": 0, 2024-03-08T23:18:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: "num_recovering_prioritized": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_started": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "recovery_completed": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_enqueue_scrub": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_exportdir": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_flush": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragmentdir": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_fragstats": 0, 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: "ireq_inodestats": 0 2024-03-08T23:18:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_log": { 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evlrg": 0, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evadd": 4466, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evex": 0, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evtrm": 0, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "ev": 4466, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evexg": 0, 2024-03-08T23:18:47.874 INFO:teuthology.orchestra.run.smithi192.stdout: "evexd": 0, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segadd": 7, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segex": 0, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segtrm": 0, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "seg": 7, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segmjr": 2, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segexg": 0, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "segexd": 0, 2024-03-08T23:18:47.875 INFO:teuthology.orchestra.run.smithi192.stdout: "expos": 4194304, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "wrpos": 11716301, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "rdpos": 0, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "jlat": { 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 86, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.917853582, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.010672716 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.876 INFO:teuthology.orchestra.run.smithi192.stdout: "replayed": 0 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_mem": { 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "ino": 1599, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "ino+": 2296, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "ino-": 697, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "dir": 173, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "dir+": 223, 2024-03-08T23:18:47.877 INFO:teuthology.orchestra.run.smithi192.stdout: "dir-": 50, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "dn": 2000, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "dn+": 2702, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "dn-": 702, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "cap": 43, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "cap+": 1832, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "cap-": 1789, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "rss": 69300, 2024-03-08T23:18:47.878 INFO:teuthology.orchestra.run.smithi192.stdout: "heap": 207228 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_server": { 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_client_request": 2486, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "dispatch_server_request": 1, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_request": 2466, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_client_session": 20, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "handle_peer_request": 6, 2024-03-08T23:18:47.879 INFO:teuthology.orchestra.run.smithi192.stdout: "req_create_latency": { 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 486, 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.453593806, 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000933320 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getattr_latency": { 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.880 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getfilelock_latency": { 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "req_link_latency": { 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.881 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookup_latency": { 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1117, 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.578378483, 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000517796 2024-03-08T23:18:47.882 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookuphash_latency": { 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupino_latency": { 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.883 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupname_latency": { 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupparent_latency": { 2024-03-08T23:18:47.884 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lookupsnap_latency": { 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.885 INFO:teuthology.orchestra.run.smithi192.stdout: "req_lssnap_latency": { 2024-03-08T23:18:47.886 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.886 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mkdir_latency": { 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.887 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mknod_latency": { 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "req_mksnap_latency": { 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.888 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: "req_open_latency": { 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 2, 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000369174, 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000184587 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.889 INFO:teuthology.orchestra.run.smithi192.stdout: "req_readdir_latency": { 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 1, 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.001030649, 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001030649 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rename_latency": { 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.890 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "req_renamesnap_latency": { 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "req_snapdiff_latency": { 2024-03-08T23:18:47.891 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmdir_latency": { 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.892 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmsnap_latency": { 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "req_rmxattr_latency": { 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.893 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.894 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.894 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.894 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setattr_latency": { 2024-03-08T23:18:47.894 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setdirlayout_latency": { 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.895 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setfilelock_latency": { 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setlayout_latency": { 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.896 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "req_setxattr_latency": { 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.897 INFO:teuthology.orchestra.run.smithi192.stdout: "req_symlink_latency": { 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "req_unlink_latency": { 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 692, 2024-03-08T23:18:47.898 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.817106295, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.001180789 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_revoke_eviction": 0, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "cap_acquisition_throttle": 0, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "req_getvxattr_latency": { 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.899 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_sessions": { 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "session_count": 1, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "session_add": 4, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "session_remove": 3, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_open": 1, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "sessions_stale": 0, 2024-03-08T23:18:47.900 INFO:teuthology.orchestra.run.smithi192.stdout: "total_load": 1562, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "average_load": 1562, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "avg_session_uptime": 571, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "mdthresh_evicted": 0 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:47.901 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:47.902 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:47.903 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:47.904 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:47.905 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-08T23:18:47.906 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 346870, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 1363, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 792, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 9, 2024-03-08T23:18:47.907 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 9424, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 181, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:47.908 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 5381263, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 58223, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:47.909 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "objecter": { 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "op_active": 0, 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "op_laggy": 0, 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send": 18400, 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "op_send_bytes": 7859181, 2024-03-08T23:18:47.910 INFO:teuthology.orchestra.run.smithi192.stdout: "op_resend": 27, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "op_reply": 18400, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "op_latency": { 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 18400, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 16.369672891, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000889656 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "op_inflight": 0, 2024-03-08T23:18:47.911 INFO:teuthology.orchestra.run.smithi192.stdout: "oplen_avg": { 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 18400, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 18453 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "op": 18400, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "op_r": 16830, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "op_w": 1570, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "op_rmw": 0, 2024-03-08T23:18:47.912 INFO:teuthology.orchestra.run.smithi192.stdout: "op_pg": 0, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_stat": 0, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_create": 10, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_read": 16, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_write": 102, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writefull": 21, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_writesame": 0, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_append": 0, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_zero": 0, 2024-03-08T23:18:47.913 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_truncate": 0, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_delete": 1397, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_mapext": 0, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_sparse_read": 0, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_clonerange": 0, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_getxattr": 16814, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_setxattr": 20, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_cmpxattr": 0, 2024-03-08T23:18:47.914 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_rmxattr": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_resetxattrs": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_call": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_watch": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_notify": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_src_cmpxattr": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls": 0, 2024-03-08T23:18:47.915 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_pgls_filter": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "osdop_other": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_active": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_send": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_resend": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "linger_ping": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_active": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_send": 0, 2024-03-08T23:18:47.916 INFO:teuthology.orchestra.run.smithi192.stdout: "poolop_resend": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_active": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_send": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "poolstat_resend": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_active": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_send": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "statfs_resend": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "command_active": 0, 2024-03-08T23:18:47.917 INFO:teuthology.orchestra.run.smithi192.stdout: "command_send": 0, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "command_resend": 0, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "map_epoch": 104, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "map_full": 0, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "map_inc": 15, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_sessions": 12, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_open": 12, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_session_close": 0, 2024-03-08T23:18:47.918 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_laggy": 0, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_wr": 60, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_rd": 0, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_del": 13 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "oft": { 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_objs": 1, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_kv_pairs": 0, 2024-03-08T23:18:47.919 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_updates": 0, 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "omap_total_removes": 0 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "osd-slow-ops": { 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "slow_ops_count": 0 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "purge_queue": { 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops": 0, 2024-03-08T23:18:47.920 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_ops_high_water": 16, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing": 0, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executing_high_water": 8, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed_ops": 1384, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_executed": 692, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "pq_item_in_journal": 0 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:47.921 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 25511, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 8721559, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 25511, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:47.922 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 25511, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 8721559, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.923 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_bytes": { 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:47.924 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 18400, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 7934609, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 18400, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 7934609, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.925 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-objecter_ops": { 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 1024, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 0, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 0, 2024-03-08T23:18:47.926 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 0, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 18400, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 18400, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 18400, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 18400, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.927 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle": { 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:47.928 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 692, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 75428, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 692, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 16, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 75428, 2024-03-08T23:18:47.929 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-write_buf_throttle-0x56242f79a6e0": { 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:47.930 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 3758096384, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 4466, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 7521997, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 4466, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:47.931 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 86, 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 7521997, 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:47.932 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-08T23:18:48.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:47 smithi062 ceph-mon[35761]: pgmap v343: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:48.201 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:47 smithi192 ceph-mon[36700]: pgmap v343: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:48.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:47 smithi046 ceph-mon[31859]: pgmap v343: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds.imported=56 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=45 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=318 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-08T23:18:48.648 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-08T23:18:48.649 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-08T23:18:48.649 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-08T23:18:48.649 DEBUG:tasks.check_counter:Getting stats from l 2024-03-08T23:18:48.649 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-08T23:18:48.886 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:18:50.012 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:49 smithi192 ceph-mon[36700]: pgmap v344: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:50.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:49 smithi062 ceph-mon[35761]: pgmap v344: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:50.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:49 smithi046 ceph-mon[31859]: pgmap v344: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:50.631 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-0": { 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 250, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 238, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 1719536, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 143504, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 4, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 0, 2024-03-08T23:18:50.632 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.050499588, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.015065971, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.038573142, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.005996920, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 237, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.011007671, 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000046445 2024-03-08T23:18:50.633 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 1719536, 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 143504 2024-03-08T23:18:50.634 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-1": { 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 6, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 185, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 3344, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 187712, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 5, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 3, 2024-03-08T23:18:50.635 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.012423025, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.008463569, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.001470806, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 183, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.030324340, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000165706 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.636 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 187712 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.637 INFO:teuthology.orchestra.run.smithi192.stdout: "AsyncMessenger::Worker-2": { 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_messages": 0, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages": 2, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_bytes": 0, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_bytes": 0, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_created_connections": 2, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_active_connections": 1, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_total_time": 0.000343737, 2024-03-08T23:18:50.638 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_send_time": 0.000000000, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_recv_time": 0.000286604, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_messages_queue_lat": { 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_handle_ack_lat": { 2024-03-08T23:18:50.639 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "msgr_send_encrypted_bytes": 0 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "cct": { 2024-03-08T23:18:50.640 INFO:teuthology.orchestra.run.smithi192.stdout: "total_workers": 0, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "unhealthy_workers": 0 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "mempool": { 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_bytes": 0, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "bloom_filter_items": 0, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_bytes": 0, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_alloc_items": 0, 2024-03-08T23:18:50.641 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_bytes": 0, 2024-03-08T23:18:50.642 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_data_items": 0, 2024-03-08T23:18:50.642 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-08T23:18:50.642 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_onode_items": 0, 2024-03-08T23:18:50.642 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-08T23:18:50.642 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_meta_items": 0, 2024-03-08T23:18:50.643 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_bytes": 0, 2024-03-08T23:18:50.643 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_other_items": 0, 2024-03-08T23:18:50.643 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-08T23:18:50.643 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_cache_buffer_items": 0, 2024-03-08T23:18:50.643 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_bytes": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_extent_items": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_bytes": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_blob_items": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_shared_blob_items": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_inline_bl_items": 0, 2024-03-08T23:18:50.644 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_bytes": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_fsck_items": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_bytes": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_txc_items": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_deferred_items": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_bytes": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluestore_writing_items": 0, 2024-03-08T23:18:50.645 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_bytes": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_items": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_bytes": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_reader_items": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_bytes": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "bluefs_file_writer_items": 0, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_bytes": 148731, 2024-03-08T23:18:50.646 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_anon_items": 31, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_bytes": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "buffer_meta_items": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_bytes": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_items": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_bytes": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_mapbl_items": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_bytes": 0, 2024-03-08T23:18:50.647 INFO:teuthology.orchestra.run.smithi192.stdout: "osd_pglog_items": 0, 2024-03-08T23:18:50.648 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_bytes": 0, 2024-03-08T23:18:50.648 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_items": 0, 2024-03-08T23:18:50.648 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_bytes": 0, 2024-03-08T23:18:50.648 INFO:teuthology.orchestra.run.smithi192.stdout: "osdmap_mapping_items": 0, 2024-03-08T23:18:50.648 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_bytes": 0, 2024-03-08T23:18:50.649 INFO:teuthology.orchestra.run.smithi192.stdout: "pgmap_items": 0, 2024-03-08T23:18:50.649 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_bytes": 928, 2024-03-08T23:18:50.649 INFO:teuthology.orchestra.run.smithi192.stdout: "mds_co_items": 2, 2024-03-08T23:18:50.649 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_bytes": 0, 2024-03-08T23:18:50.649 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_1_items": 0, 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_bytes": 0, 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "unittest_2_items": 0 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: }, 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "val": 0, 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "max": 104857600, 2024-03-08T23:18:50.650 INFO:teuthology.orchestra.run.smithi192.stdout: "get_started": 0, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "get": 252, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "get_sum": 1684289, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_fail": 0, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "get_or_fail_success": 252, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "take": 0, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "take_sum": 0, 2024-03-08T23:18:50.651 INFO:teuthology.orchestra.run.smithi192.stdout: "put": 252, 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: "put_sum": 1684289, 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: "wait": { 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: "avgcount": 0, 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: "sum": 0.000000000, 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: "avgtime": 0.000000000 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:50.652 INFO:teuthology.orchestra.run.smithi192.stdout: } 2024-03-08T23:18:50.653 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-03-08T23:18:51.268 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-03-08T23:18:51.516 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-03-08T23:18:51.516 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-03-08T23:18:51.516 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-03-08T23:18:51.516 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-08T23:18:51.516 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-08T23:18:51.517 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-08T23:18:51.517 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-08T23:18:51.517 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-08T23:18:51.808 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-08T23:18:51.808 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:18:51.808 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T23:18:51.873 DEBUG:teuthology.orchestra.run.smithi046:> set -ex 2024-03-08T23:18:51.873 DEBUG:teuthology.orchestra.run.smithi046:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-08T23:18:51.928 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:18:52.140 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:51 smithi062 ceph-mon[35761]: pgmap v345: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:52.162 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 -- 172.21.15.46:0/3048588101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 msgr2=0x7f2840103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 --2- 172.21.15.46:0/3048588101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f2840103930 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f2830009900 tx=0x7f283002bc40 comp rx=0 tx=0).stop 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 -- 172.21.15.46:0/3048588101 shutdown_connections 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 --2- 172.21.15.46:0/3048588101 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f28401101f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 --2- 172.21.15.46:0/3048588101 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840103f00 0x7f2840107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 --2- 172.21.15.46:0/3048588101 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f2840103930 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 -- 172.21.15.46:0/3048588101 >> 172.21.15.46:0/3048588101 conn(0x7f28400fed90 msgr2=0x7f28401011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:52.163 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 -- 172.21.15.46:0/3048588101 shutdown_connections 2024-03-08T23:18:52.164 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.159+0000 7f2846400640 1 -- 172.21.15.46:0/3048588101 wait complete. 2024-03-08T23:18:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.160+0000 7f2846400640 1 Processor -- start 2024-03-08T23:18:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.160+0000 7f2846400640 1 -- start start 2024-03-08T23:18:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.166 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840199a40 0x7f2840199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840113700 con 0x7f2840108620 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840113580 con 0x7f2840103550 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2846400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840113880 con 0x7f2840199a40 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f283fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:51388/0 (socket says 172.21.15.46:51388) 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f283fe00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:49766/0 (socket says 172.21.15.46:49766) 2024-03-08T23:18:52.167 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 -- 172.21.15.46:0/2590647712 learned_addr learned my addr 172.21.15.46:0/2590647712 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845a00640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840199a40 0x7f2840199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 -- 172.21.15.46:0/2590647712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840199a40 msgr2=0x7f2840199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840199a40 0x7f2840199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 -- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 msgr2=0x7f284019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f2845000640 1 -- 172.21.15.46:0/2590647712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f284019a560 con 0x7f2840103550 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.161+0000 7f283fe00640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:52.168 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f2845000640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2830005ab0 tx=0x7f28300042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830041070 con 0x7f2840103550 2024-03-08T23:18:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f283002cca0 con 0x7f2840103550 2024-03-08T23:18:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2830003d20 con 0x7f2840103550 2024-03-08T23:18:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f284019a7f0 con 0x7f2840103550 2024-03-08T23:18:52.169 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.162+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28401a6940 con 0x7f2840103550 2024-03-08T23:18:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.164+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2840104a20 con 0x7f2840103550 2024-03-08T23:18:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.166+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2830044070 con 0x7f2840103550 2024-03-08T23:18:52.170 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.167+0000 7f283d600640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 0x7f28240a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.167+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f28300f0a40 con 0x7f2840103550 2024-03-08T23:18:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.167+0000 7f283fe00640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 0x7f28240a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.171 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.168+0000 7f283fe00640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 0x7f28240a4f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2840109b70 tx=0x7f2828009500 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:52.174 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.171+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f28300bd490 con 0x7f2840103550 2024-03-08T23:18:52.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:51 smithi046 ceph-mon[31859]: pgmap v345: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:52.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:51 smithi192 ceph-mon[36700]: pgmap v345: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-08T23:18:52.379 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.375+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f284019af20 con 0x7f2840103550 2024-03-08T23:18:52.381 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.378+0000 7f283d600640 1 -- 172.21.15.46:0/2590647712 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+25258 (secure 0 0 0) 0x7f28300bcd90 con 0x7f2840103550 2024-03-08T23:18:52.382 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:52.382 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:18:14.347594+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6801/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/1737358861":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1465757142":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1832130111":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1962446760":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:18:52.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 msgr2=0x7f28240a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.384 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 0x7f28240a4f30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2840109b70 tx=0x7f2828009500 comp rx=0 tx=0).stop 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 msgr2=0x7f284019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f2830005ab0 tx=0x7f28300042f0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 shutdown_connections 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f28240a2a70 0x7f28240a4f30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2840199a40 0x7f2840199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2840108620 0x7f284019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.385 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 --2- 172.21.15.46:0/2590647712 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2840103550 0x7f284019f840 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 >> 172.21.15.46:0/2590647712 conn(0x7f28400fed90 msgr2=0x7f2840101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 shutdown_connections 2024-03-08T23:18:52.386 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.381+0000 7f2846400640 1 -- 172.21.15.46:0/2590647712 wait complete. 2024-03-08T23:18:52.404 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T22:59:25.735173+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-08T23:04:36.776062+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-08T23:04:38.292398+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-08T23:04:41.925317+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-08T23:18:52.405 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-08T23:18:52.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 -- 172.21.15.46:0/554118219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808103f90 msgr2=0x7f3808107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 --2- 172.21.15.46:0/554118219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808103f90 0x7f3808107fd0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f37f8009900 tx=0x7f37f802bc40 comp rx=0 tx=0).stop 2024-03-08T23:18:52.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 -- 172.21.15.46:0/554118219 shutdown_connections 2024-03-08T23:18:52.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 --2- 172.21.15.46:0/554118219 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 0x7f3808110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.642 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 --2- 172.21.15.46:0/554118219 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808103f90 0x7f3808107fd0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 --2- 172.21.15.46:0/554118219 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f38081039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 -- 172.21.15.46:0/554118219 >> 172.21.15.46:0/554118219 conn(0x7f38080fee20 msgr2=0x7f3808101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:52.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 -- 172.21.15.46:0/554118219 shutdown_connections 2024-03-08T23:18:52.643 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.638+0000 7f380f200640 1 -- 172.21.15.46:0/554118219 wait complete. 2024-03-08T23:18:52.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 Processor -- start 2024-03-08T23:18:52.646 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 -- start start 2024-03-08T23:18:52.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 0x7f380819fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808199ab0 0x7f3808199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808113790 con 0x7f3808199ab0 2024-03-08T23:18:52.647 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808113610 con 0x7f38081086b0 2024-03-08T23:18:52.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.640+0000 7f380f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808113910 con 0x7f38081035e0 2024-03-08T23:18:52.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:43312/0 (socket says 172.21.15.46:43312) 2024-03-08T23:18:52.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 -- 172.21.15.46:0/155789616 learned_addr learned my addr 172.21.15.46:0/155789616 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:52.648 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380d400640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 0x7f380819fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380e800640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808199ab0 0x7f3808199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 -- 172.21.15.46:0/155789616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 msgr2=0x7f380819fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 0x7f380819fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 -- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808199ab0 msgr2=0x7f3808199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.649 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808199ab0 0x7f3808199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 -- 172.21.15.46:0/155789616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380819a5d0 con 0x7f38081035e0 2024-03-08T23:18:52.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.641+0000 7f380de00640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00e9f0 tx=0x7f37fc00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:52.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.642+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc00cd80 con 0x7f38081035e0 2024-03-08T23:18:52.650 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.642+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380819a8c0 con 0x7f38081035e0 2024-03-08T23:18:52.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.642+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37fc005500 con 0x7f38081035e0 2024-03-08T23:18:52.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.642+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37fc010690 con 0x7f38081035e0 2024-03-08T23:18:52.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.642+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38081a6b10 con 0x7f38081035e0 2024-03-08T23:18:52.651 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.643+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3808068450 con 0x7f38081035e0 2024-03-08T23:18:52.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.646+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f37fc0046c0 con 0x7f38081035e0 2024-03-08T23:18:52.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.648+0000 7f3802a00640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 0x7f37f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:52.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.648+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f37fc0cf860 con 0x7f38081035e0 2024-03-08T23:18:52.653 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.648+0000 7f380d400640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 0x7f37f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:52.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.649+0000 7f380d400640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 0x7f37f00a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37f802c250 tx=0x7f37f802cf90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:52.654 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.649+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f37fc053260 con 0x7f38081035e0 2024-03-08T23:18:52.857 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.854+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f380810b8e0 con 0x7f38081035e0 2024-03-08T23:18:52.858 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.856+0000 7f3802a00640 1 -- 172.21.15.46:0/155789616 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f37fc09c1a0 con 0x7f38081035e0 2024-03-08T23:18:52.858 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 1 2024-03-08T23:18:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 msgr2=0x7f37f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 0x7f37f00a4e30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f37f802c250 tx=0x7f37f802cf90 comp rx=0 tx=0).stop 2024-03-08T23:18:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 msgr2=0x7f380819f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:52.861 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f37fc00e9f0 tx=0x7f37fc00edf0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 shutdown_connections 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f37f00a2970 0x7f37f00a4e30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f3808199ab0 0x7f3808199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f38081086b0 0x7f380819fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 --2- 172.21.15.46:0/155789616 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f38081035e0 0x7f380819f900 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.857+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 >> 172.21.15.46:0/155789616 conn(0x7f38080fee20 msgr2=0x7f3808101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:52.862 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.858+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 shutdown_connections 2024-03-08T23:18:52.863 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:52.858+0000 7f380f200640 1 -- 172.21.15.46:0/155789616 wait complete. 2024-03-08T23:18:52.882 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-08T23:18:53.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 -- 172.21.15.46:0/1087418407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c1034b0 msgr2=0x7fef7c103930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 --2- 172.21.15.46:0/1087418407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c1034b0 0x7fef7c103930 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fef64009c10 tx=0x7fef6402bee0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 -- 172.21.15.46:0/1087418407 shutdown_connections 2024-03-08T23:18:53.117 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 --2- 172.21.15.46:0/1087418407 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c103e70 0x7fef7c110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 --2- 172.21.15.46:0/1087418407 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c1034b0 0x7fef7c103930 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 --2- 172.21.15.46:0/1087418407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c109570 0x7fef7c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.113+0000 7fef80e00640 1 -- 172.21.15.46:0/1087418407 >> 172.21.15.46:0/1087418407 conn(0x7fef7c0fef60 msgr2=0x7fef7c101380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:53.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.114+0000 7fef80e00640 1 -- 172.21.15.46:0/1087418407 shutdown_connections 2024-03-08T23:18:53.118 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.114+0000 7fef80e00640 1 -- 172.21.15.46:0/1087418407 wait complete. 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 Processor -- start 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 -- start start 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c103e70 0x7fef7c19a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c109570 0x7fef7c19e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.120 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c113880 con 0x7fef7c109570 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c113700 con 0x7fef7c103e70 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef80e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef7c113a00 con 0x7fef7c1034b0 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef79e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef7a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c109570 0x7fef7c19e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef79400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c103e70 0x7fef7c19a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef79e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57740/0 (socket says 172.21.15.46:57740) 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.115+0000 7fef79e00640 1 -- 172.21.15.46:0/3157630400 learned_addr learned my addr 172.21.15.46:0/3157630400 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:53.121 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.116+0000 7fef79e00640 1 -- 172.21.15.46:0/3157630400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c103e70 msgr2=0x7fef7c19a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.116+0000 7fef79e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c103e70 0x7fef7c19a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.116+0000 7fef79e00640 1 -- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c109570 msgr2=0x7fef7c19e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.116+0000 7fef79e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c109570 0x7fef7c19e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.116+0000 7fef79e00640 1 -- 172.21.15.46:0/3157630400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c19ed50 con 0x7fef7c1034b0 2024-03-08T23:18:53.122 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef79e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef7400ea60 tx=0x7fef7400ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:53.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef7400cd20 con 0x7fef7c1034b0 2024-03-08T23:18:53.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c19f040 con 0x7fef7c1034b0 2024-03-08T23:18:53.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef740035f0 con 0x7fef7c1034b0 2024-03-08T23:18:53.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef74010690 con 0x7fef7c1034b0 2024-03-08T23:18:53.123 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.117+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c1a68d0 con 0x7fef7c1034b0 2024-03-08T23:18:53.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.118+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c068530 con 0x7fef7c1034b0 2024-03-08T23:18:53.124 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.121+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fef740037a0 con 0x7fef7c1034b0 2024-03-08T23:18:53.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.123+0000 7fef72a00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 0x7fef600a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.123+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fef740cf630 con 0x7fef7c1034b0 2024-03-08T23:18:53.126 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.123+0000 7fef79400640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 0x7fef600a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.124+0000 7fef79400640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 0x7fef600a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef64000f80 tx=0x7fef640333e0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:53.127 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.124+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fef7409c000 con 0x7fef7c1034b0 2024-03-08T23:18:53.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:52 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2590647712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:53.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:52 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/155789616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:53.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:52 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2590647712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:53.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:52 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/155789616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:53.333 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.330+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fef7c02ce70 con 0x7fef7c1034b0 2024-03-08T23:18:53.334 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.331+0000 7fef72a00640 1 -- 172.21.15.46:0/3157630400 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7fef7409b900 con 0x7fef7c1034b0 2024-03-08T23:18:53.334 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:18:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 msgr2=0x7fef600a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 0x7fef600a4e30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef64000f80 tx=0x7fef640333e0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 msgr2=0x7fef7c199d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.337 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fef7400ea60 tx=0x7fef7400ee60 comp rx=0 tx=0).stop 2024-03-08T23:18:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 shutdown_connections 2024-03-08T23:18:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fef600a2970 0x7fef600a4e30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fef7c109570 0x7fef7c19e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.338 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fef7c103e70 0x7fef7c19a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 --2- 172.21.15.46:0/3157630400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef7c1034b0 0x7fef7c199d60 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 >> 172.21.15.46:0/3157630400 conn(0x7fef7c0fef60 msgr2=0x7fef7c100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 shutdown_connections 2024-03-08T23:18:53.339 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.334+0000 7fef80e00640 1 -- 172.21.15.46:0/3157630400 wait complete. 2024-03-08T23:18:53.359 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-08T23:18:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:52 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2590647712' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:53.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:52 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/155789616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-08T23:18:53.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.586+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1732876216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc1035e0 msgr2=0x7ffbdc1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.586+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1732876216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc1035e0 0x7ffbdc1039c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7ffbd0009900 tx=0x7ffbd002bc30 comp rx=0 tx=0).stop 2024-03-08T23:18:53.590 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1732876216 shutdown_connections 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1732876216 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 0x7ffbdc110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1732876216 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbdc103f90 0x7ffbdc107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1732876216 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc1035e0 0x7ffbdc1039c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1732876216 >> 172.21.15.46:0/1732876216 conn(0x7ffbdc0fee20 msgr2=0x7ffbdc101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1732876216 shutdown_connections 2024-03-08T23:18:53.591 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.587+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1732876216 wait complete. 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 Processor -- start 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 -- start start 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbdc1035e0 0x7ffbdc199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 0x7ffbdc19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.594 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc113810 con 0x7ffbdc19e650 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc113690 con 0x7ffbdc1086b0 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2a00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbdc113990 con 0x7ffbdc1035e0 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.588+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37604/0 (socket says 172.21.15.46:37604) 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 -- 172.21.15.46:0/1630336678 learned_addr learned my addr 172.21.15.46:0/1630336678 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe0c00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 0x7ffbdc19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 -- 172.21.15.46:0/1630336678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbdc1035e0 msgr2=0x7ffbdc199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.595 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbdc1035e0 0x7ffbdc199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 -- 172.21.15.46:0/1630336678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 msgr2=0x7ffbdc19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 0x7ffbdc19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 -- 172.21.15.46:0/1630336678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbdc19f140 con 0x7ffbdc19e650 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.589+0000 7ffbe2000640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400eb10 tx=0x7ffbd400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.590+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd400ce20 con 0x7ffbdc19e650 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.590+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbdc19f430 con 0x7ffbdc19e650 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.590+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbd4005020 con 0x7ffbdc19e650 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.590+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd4010690 con 0x7ffbdc19e650 2024-03-08T23:18:53.596 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.590+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbdc1a6c20 con 0x7ffbdc19e650 2024-03-08T23:18:53.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.591+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbdc068450 con 0x7ffbdc19e650 2024-03-08T23:18:53.597 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.593+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ffbd4005540 con 0x7ffbdc19e650 2024-03-08T23:18:53.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.594+0000 7ffbda000640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 0x7ffbc40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:53.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.594+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ffbd40cf830 con 0x7ffbdc19e650 2024-03-08T23:18:53.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.595+0000 7ffbe1600640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 0x7ffbc40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:53.598 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.595+0000 7ffbe1600640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 0x7ffbc40a4eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00096c0 tx=0x7ffbd0000bc0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:53.603 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.600+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffbd409c170 con 0x7ffbdc19e650 2024-03-08T23:18:53.804 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.801+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ffbdc10b940 con 0x7ffbdc19e650 2024-03-08T23:18:53.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.802+0000 7ffbda000640 1 -- 172.21.15.46:0/1630336678 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7ffbd409ba70 con 0x7ffbdc19e650 2024-03-08T23:18:53.805 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 64 2024-03-08T23:18:53.808 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 msgr2=0x7ffbc40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 0x7ffbc40a4eb0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ffbd00096c0 tx=0x7ffbd0000bc0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 msgr2=0x7ffbdc19eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ffbd400eb10 tx=0x7ffbd400ef10 comp rx=0 tx=0).stop 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 shutdown_connections 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ffbc40a29f0 0x7ffbc40a4eb0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ffbdc19e650 0x7ffbdc19eb00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ffbdc1086b0 0x7ffbdc19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.809 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 --2- 172.21.15.46:0/1630336678 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ffbdc1035e0 0x7ffbdc199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 >> 172.21.15.46:0/1630336678 conn(0x7ffbdc0fee20 msgr2=0x7ffbdc101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 shutdown_connections 2024-03-08T23:18:53.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:53.805+0000 7ffbe2a00640 1 -- 172.21.15.46:0/1630336678 wait complete. 2024-03-08T23:18:53.829 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-08T23:18:54.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.054+0000 7fb31f200640 1 -- 172.21.15.46:0/1084780342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb320103f90 msgr2=0x7fb320107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.054+0000 7fb31f200640 1 --2- 172.21.15.46:0/1084780342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb320103f90 0x7fb320107fd0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb30c009900 tx=0x7fb30c02bc30 comp rx=0 tx=0).stop 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 -- 172.21.15.46:0/1084780342 shutdown_connections 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 --2- 172.21.15.46:0/1084780342 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb3201086b0 0x7fb320110280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 --2- 172.21.15.46:0/1084780342 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb320103f90 0x7fb320107fd0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 --2- 172.21.15.46:0/1084780342 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201035e0 0x7fb3201039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 -- 172.21.15.46:0/1084780342 >> 172.21.15.46:0/1084780342 conn(0x7fb3200fee20 msgr2=0x7fb320101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 -- 172.21.15.46:0/1084780342 shutdown_connections 2024-03-08T23:18:54.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.055+0000 7fb31f200640 1 -- 172.21.15.46:0/1084780342 wait complete. 2024-03-08T23:18:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 Processor -- start 2024-03-08T23:18:54.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 -- start start 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 0x7fb32019f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 0x7fb32019fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 0x7fb320199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320113790 con 0x7fb3201035e0 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320113610 con 0x7fb3201086b0 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31f200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb320113910 con 0x7fb320199a70 2024-03-08T23:18:54.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.056+0000 7fb31de00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 0x7fb32019f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 0x7fb320199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31e800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 0x7fb320199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57768/0 (socket says 172.21.15.46:57768) 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31e800640 1 -- 172.21.15.46:0/10477900 learned_addr learned my addr 172.21.15.46:0/10477900 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 0x7fb32019fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 -- 172.21.15.46:0/10477900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 msgr2=0x7fb320199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 0x7fb320199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 -- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 msgr2=0x7fb32019f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 0x7fb32019f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.063 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 -- 172.21.15.46:0/10477900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32019a590 con 0x7fb3201086b0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31de00640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 0x7fb32019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.057+0000 7fb31d400640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 0x7fb32019fe00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb30c036040 tx=0x7fb30c005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.058+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c043020 con 0x7fb3201086b0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.058+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32019a820 con 0x7fb3201086b0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.058+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb30c005000 con 0x7fb3201086b0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.058+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c003af0 con 0x7fb3201086b0 2024-03-08T23:18:54.064 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.058+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3201a6ac0 con 0x7fb3201086b0 2024-03-08T23:18:54.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.059+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb320068450 con 0x7fb3201086b0 2024-03-08T23:18:54.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.061+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb30c046020 con 0x7fb3201086b0 2024-03-08T23:18:54.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.062+0000 7fb316a00640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 0x7fb3040a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.062+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fb30c0f0ff0 con 0x7fb3201086b0 2024-03-08T23:18:54.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.062+0000 7fb31de00640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 0x7fb3040a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.066 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.063+0000 7fb31de00640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 0x7fb3040a4eb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb32019abb0 tx=0x7fb310005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:54.069 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.067+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb30c04d020 con 0x7fb3201086b0 2024-03-08T23:18:54.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:53 smithi046 ceph-mon[31859]: pgmap v346: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:54.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:53 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3157630400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:53 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1630336678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.270 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.267+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fb32010b910 con 0x7fb3201086b0 2024-03-08T23:18:54.271 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.269+0000 7fb316a00640 1 -- 172.21.15.46:0/10477900 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) v1 ==== 103+0+11 (secure 0 0 0) 0x7fb30c0bda60 con 0x7fb3201086b0 2024-03-08T23:18:54.271 INFO:teuthology.orchestra.run.smithi046.stdout:pg_num: 32 2024-03-08T23:18:54.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.270+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 msgr2=0x7fb3040a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.270+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 0x7fb3040a4eb0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb32019abb0 tx=0x7fb310005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:54.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.270+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 msgr2=0x7fb32019fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.274 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.270+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 0x7fb32019fe00 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb30c036040 tx=0x7fb30c005e80 comp rx=0 tx=0).stop 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 shutdown_connections 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fb3040a29f0 0x7fb3040a4eb0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb320199a70 0x7fb320199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fb3201086b0 0x7fb32019fe00 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 --2- 172.21.15.46:0/10477900 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fb3201035e0 0x7fb32019f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 >> 172.21.15.46:0/10477900 conn(0x7fb3200fee20 msgr2=0x7fb320101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:54.275 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 shutdown_connections 2024-03-08T23:18:54.276 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.271+0000 7fb31f200640 1 -- 172.21.15.46:0/10477900 wait complete. 2024-03-08T23:18:54.293 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-08T23:18:54.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:53 smithi192 ceph-mon[36700]: pgmap v346: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:54.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:53 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3157630400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:53 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1630336678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:53 smithi062 ceph-mon[35761]: pgmap v346: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-08T23:18:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:53 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3157630400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:53 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1630336678' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-08T23:18:54.522 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1901860119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073690 msgr2=0x7f5b78073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1901860119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073690 0x7f5b78073a70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009ca0 tx=0x7f5b6802c180 comp rx=0 tx=0).stop 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1901860119 shutdown_connections 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1901860119 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b78114560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1901860119 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073fb0 0x7f5b78111c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1901860119 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073690 0x7f5b78073a70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.523 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1901860119 >> 172.21.15.46:0/1901860119 conn(0x7f5b780fcc50 msgr2=0x7f5b780ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:54.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1901860119 shutdown_connections 2024-03-08T23:18:54.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.519+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1901860119 wait complete. 2024-03-08T23:18:54.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.520+0000 7f5b7dc00640 1 Processor -- start 2024-03-08T23:18:54.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.520+0000 7f5b7dc00640 1 -- start start 2024-03-08T23:18:54.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.520+0000 7f5b7dc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7dc00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7dc00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b781a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7dc00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b78117b70 con 0x7f5b78073fb0 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7dc00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b781179f0 con 0x7f5b78112170 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7dc00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b78117cf0 con 0x7f5b78073690 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b781a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7d200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b781a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:54916/0 (socket says 172.21.15.46:54916) 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b7d200640 1 -- 172.21.15.46:0/1021490521 learned_addr learned my addr 172.21.15.46:0/1021490521 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37634/0 (socket says 172.21.15.46:37634) 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57782/0 (socket says 172.21.15.46:57782) 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 -- 172.21.15.46:0/1021490521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 msgr2=0x7f5b781a2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b781a2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 -- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 msgr2=0x7f5b7819e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77e00640 1 -- 172.21.15.46:0/1021490521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b781a3080 con 0x7f5b78073690 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.521+0000 7f5b77400640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.522+0000 7f5b77e00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5b6802c6c0 tx=0x7f5b68035d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.522+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68039040 con 0x7f5b78073690 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.523+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b781a3310 con 0x7f5b78073690 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.523+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b68044070 con 0x7f5b78073690 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.523+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b680328d0 con 0x7f5b78073690 2024-03-08T23:18:54.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.523+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b781aac00 con 0x7f5b78073690 2024-03-08T23:18:54.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.524+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b78074dd0 con 0x7f5b78073690 2024-03-08T23:18:54.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.526+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5b68032a70 con 0x7f5b78073690 2024-03-08T23:18:54.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.527+0000 7f5b74c00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 0x7f5b5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:54.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.527+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f5b680f1e30 con 0x7f5b78073690 2024-03-08T23:18:54.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.528+0000 7f5b77400640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 0x7f5b5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:54.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.528+0000 7f5b77400640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 0x7f5b5c0a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5b780ff350 tx=0x7f5b60005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:54.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.529+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5b680be9a0 con 0x7f5b78073690 2024-03-08T23:18:54.750 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.745+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5b78068450 con 0x7f5b78073690 2024-03-08T23:18:54.751 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.746+0000 7f5b74c00640 1 -- 172.21.15.46:0/1021490521 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) v1 ==== 81+0+942 (secure 0 0 0) 0x7f5b680be2a0 con 0x7f5b78073690 2024-03-08T23:18:54.751 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/757291496 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.751 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1737358861 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.751 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/757291496 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.751 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/3556325343 2024-03-09T22:56:39.034402+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1288083028 2024-03-09T22:56:39.034402+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1166845583 2024-03-09T22:56:05.453593+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1465757142 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6801/2941246107 2024-03-09T22:56:05.453593+0000 2024-03-08T23:18:54.752 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/2084846526 2024-03-09T22:56:05.453593+0000 2024-03-08T23:18:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1832130111 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/300257767 2024-03-09T22:56:05.453593+0000 2024-03-08T23:18:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/1962446760 2024-03-09T23:07:46.416978+0000 2024-03-08T23:18:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:6800/4209254516 2024-03-09T22:56:39.034402+0000 2024-03-08T23:18:54.753 INFO:teuthology.orchestra.run.smithi046.stdout:172.21.15.46:0/256680715 2024-03-09T22:56:39.034402+0000 2024-03-08T23:18:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:listed 16 entries 2024-03-08T23:18:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 msgr2=0x7f5b5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 0x7f5b5c0a4e30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5b780ff350 tx=0x7f5b60005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:54.754 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 msgr2=0x7f5b7819e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5b6802c6c0 tx=0x7f5b68035d80 comp rx=0 tx=0).stop 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 shutdown_connections 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f5b5c0a2970 0x7f5b5c0a4e30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f5b78112170 0x7f5b781a2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f5b78073fb0 0x7f5b7819e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.755 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 --2- 172.21.15.46:0/1021490521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f5b78073690 0x7f5b7819e090 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:54.756 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.749+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 >> 172.21.15.46:0/1021490521 conn(0x7f5b780fcc50 msgr2=0x7f5b780fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.750+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 shutdown_connections 2024-03-08T23:18:54.757 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:54.750+0000 7f5b7dc00640 1 -- 172.21.15.46:0/1021490521 wait complete. 2024-03-08T23:18:54.772 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-08T23:18:54.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:18:54.772 DEBUG:teuthology.orchestra.run.smithi046:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-08T23:18:54.996 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:18:54.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:18:54.997 DEBUG:teuthology.orchestra.run.smithi046:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-08T23:18:55.061 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:18:55.061 DEBUG:teuthology.orchestra.run.smithi046:> ip netns list 2024-03-08T23:18:55.100 INFO:teuthology.orchestra.run.smithi046.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-08T23:18:55.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:18:55.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-08T23:18:55.203 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/10477900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:55.204 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:54 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1021490521' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:18:55.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-08T23:18:55.215 DEBUG:teuthology.orchestra.run.smithi046:> sudo ip link delete ceph-brx 2024-03-08T23:18:55.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/10477900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:55.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:54 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1021490521' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:18:55.335 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-08T23:18:55.346 INFO:tasks.cephadm:Teardown begin 2024-03-08T23:18:55.347 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-08T23:18:55.347 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:18:55.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/10477900' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-08T23:18:55.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:54 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1021490521' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-08T23:18:55.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 -- 172.21.15.46:0/2279170283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8103530 msgr2=0x7f4ba8103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 --2- 172.21.15.46:0/2279170283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8103530 0x7f4ba8103910 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f4b98009ca0 tx=0x7f4b9802c0f0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 -- 172.21.15.46:0/2279170283 shutdown_connections 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 --2- 172.21.15.46:0/2279170283 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8108600 0x7f4ba81101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 --2- 172.21.15.46:0/2279170283 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103ee0 0x7f4ba8107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 --2- 172.21.15.46:0/2279170283 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8103530 0x7f4ba8103910 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 -- 172.21.15.46:0/2279170283 >> 172.21.15.46:0/2279170283 conn(0x7f4ba80fed90 msgr2=0x7f4ba81011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 -- 172.21.15.46:0/2279170283 shutdown_connections 2024-03-08T23:18:55.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.549+0000 7f4bad600640 1 -- 172.21.15.46:0/2279170283 wait complete. 2024-03-08T23:18:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.550+0000 7f4bad600640 1 Processor -- start 2024-03-08T23:18:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.550+0000 7f4bad600640 1 -- start start 2024-03-08T23:18:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 0x7f4ba819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:55.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8108600 0x7f4ba819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba81136c0 con 0x7f4ba8108600 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8113540 con 0x7f4ba8103530 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bad600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba8113840 con 0x7f4ba8199a00 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bacc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bacc00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57794/0 (socket says 172.21.15.46:57794) 2024-03-08T23:18:55.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4bacc00640 1 -- 172.21.15.46:0/1528796882 learned_addr learned my addr 172.21.15.46:0/1528796882 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4ba7400640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8108600 0x7f4ba819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.551+0000 7f4ba7e00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 0x7f4ba819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 -- 172.21.15.46:0/1528796882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 msgr2=0x7f4ba819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 0x7f4ba819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 -- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8108600 msgr2=0x7f4ba819fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8108600 0x7f4ba819fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 -- 172.21.15.46:0/1528796882 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba819a520 con 0x7f4ba8199a00 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4ba7e00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 0x7f4ba819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bacc00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e970 tx=0x7f4ba000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba000ccc0 con 0x7f4ba8199a00 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba819a810 con 0x7f4ba8199a00 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba000ce60 con 0x7f4ba8199a00 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.552+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba0010450 con 0x7f4ba8199a00 2024-03-08T23:18:55.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.553+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba81a6ab0 con 0x7f4ba8199a00 2024-03-08T23:18:55.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.554+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba8068450 con 0x7f4ba8199a00 2024-03-08T23:18:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.556+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4ba0005020 con 0x7f4ba8199a00 2024-03-08T23:18:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.557+0000 7f4ba4c00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 0x7f4b8c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.557+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7f4ba00cf610 con 0x7f4ba8199a00 2024-03-08T23:18:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.557+0000 7f4ba7e00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 0x7f4b8c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:55.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.558+0000 7f4ba7e00640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 0x7f4b8c0a4eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000f80 tx=0x7f4b98009880 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:55.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.559+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4ba009bf50 con 0x7f4ba8199a00 2024-03-08T23:18:55.805 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.803+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba819af30 con 0x7f4ba8199a00 2024-03-08T23:18:55.806 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.804+0000 7f4ba4c00640 1 -- 172.21.15.46:0/1528796882 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7f4ba819af30 con 0x7f4ba8199a00 2024-03-08T23:18:55.807 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:55.807 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:18:55.807 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:18:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.807+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 msgr2=0x7f4b8c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.807+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 0x7f4b8c0a4eb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f4b98000f80 tx=0x7f4b98009880 comp rx=0 tx=0).stop 2024-03-08T23:18:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.807+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 msgr2=0x7f4ba8199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:55.810 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.807+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4ba000e970 tx=0x7f4ba000ed70 comp rx=0 tx=0).stop 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 shutdown_connections 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f4b8c0a29f0 0x7f4b8c0a4eb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4ba8199a00 0x7f4ba8199eb0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f4ba8108600 0x7f4ba819fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 --2- 172.21.15.46:0/1528796882 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f4ba8103530 0x7f4ba819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 >> 172.21.15.46:0/1528796882 conn(0x7f4ba80fed90 msgr2=0x7f4ba81010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:55.811 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 shutdown_connections 2024-03-08T23:18:55.814 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:55.808+0000 7f4bad600640 1 -- 172.21.15.46:0/1528796882 wait complete. 2024-03-08T23:18:55.830 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-08T23:18:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.050+0000 7ff62c600640 1 -- 172.21.15.46:0/2652839657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624103e60 msgr2=0x7ff624110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.050+0000 7ff62c600640 1 --2- 172.21.15.46:0/2652839657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624103e60 0x7ff624110320 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009ca0 tx=0x7ff61c02c100 comp rx=0 tx=0).stop 2024-03-08T23:18:56.053 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 -- 172.21.15.46:0/2652839657 shutdown_connections 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 --2- 172.21.15.46:0/2652839657 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624103e60 0x7ff624110320 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 --2- 172.21.15.46:0/2652839657 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff624103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 --2- 172.21.15.46:0/2652839657 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624109560 0x7ff624109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 -- 172.21.15.46:0/2652839657 >> 172.21.15.46:0/2652839657 conn(0x7ff6240fef50 msgr2=0x7ff624101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 -- 172.21.15.46:0/2652839657 shutdown_connections 2024-03-08T23:18:56.054 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.051+0000 7ff62c600640 1 -- 172.21.15.46:0/2652839657 wait complete. 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.052+0000 7ff62c600640 1 Processor -- start 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.052+0000 7ff62c600640 1 -- start start 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.052+0000 7ff62c600640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62c600640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624103e60 0x7ff62419fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62c600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 0x7ff624199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.057 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62c600640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff624113840 con 0x7ff624109560 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62c600640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6241136c0 con 0x7ff6241034a0 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62c600640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6241139c0 con 0x7ff624103e60 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62a600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 0x7ff624199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62a600640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 0x7ff624199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37658/0 (socket says 172.21.15.46:37658) 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff62a600640 1 -- 172.21.15.46:0/3060521983 learned_addr learned my addr 172.21.15.46:0/3060521983 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:56.058 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:54960/0 (socket says 172.21.15.46:54960) 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629200640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624103e60 0x7ff62419fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 -- 172.21.15.46:0/3060521983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624103e60 msgr2=0x7ff62419fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624103e60 0x7ff62419fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 -- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 msgr2=0x7ff624199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 0x7ff624199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.053+0000 7ff629c00640 1 -- 172.21.15.46:0/3060521983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62419a200 con 0x7ff6241034a0 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff629c00640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff61800e9f0 tx=0x7ff61800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:56.059 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61800ccc0 con 0x7ff6241034a0 2024-03-08T23:18:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff618005020 con 0x7ff6241034a0 2024-03-08T23:18:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6180049e0 con 0x7ff6241034a0 2024-03-08T23:18:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62419a4f0 con 0x7ff6241034a0 2024-03-08T23:18:56.060 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.054+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6241a6790 con 0x7ff6241034a0 2024-03-08T23:18:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.055+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff624068530 con 0x7ff6241034a0 2024-03-08T23:18:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.057+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff618005560 con 0x7ff6241034a0 2024-03-08T23:18:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.059+0000 7ff622000640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 0x7ff60c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.059+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7ff6180cf230 con 0x7ff6241034a0 2024-03-08T23:18:56.061 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.059+0000 7ff629200640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 0x7ff60c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.062 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.060+0000 7ff629200640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 0x7ff60c0a4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff624108b10 tx=0x7ff614005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:56.065 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.063+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff6180051d0 con 0x7ff6241034a0 2024-03-08T23:18:56.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:55 smithi046 ceph-mon[31859]: pgmap v347: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:56.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:55 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1528796882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:56.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.309+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff62419afb0 con 0x7ff6241034a0 2024-03-08T23:18:56.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:55 smithi192 ceph-mon[36700]: pgmap v347: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:56.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:55 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1528796882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:56.313 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.310+0000 7ff622000640 1 -- 172.21.15.46:0/3060521983 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11334 (secure 0 0 0) 0x7ff61809bb70 con 0x7ff6241034a0 2024-03-08T23:18:56.313 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:56.314 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14382,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6835/2033972073","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6834","nonce":2033972073},{"type":"v1","addr":"172.21.15.46:6835","nonce":2033972073}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14448,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6839/2215569681","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6838","nonce":2215569681},{"type":"v1","addr":"172.21.15.46:6839","nonce":2215569681}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24457,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/400907664","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":400907664},{"type":"v1","addr":"172.21.15.192:6833","nonce":400907664}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6833/3990698134","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":3990698134},{"type":"v1","addr":"172.21.15.62:6833","nonce":3990698134}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.46:6837/950462730","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6836","nonce":950462730},{"type":"v1","addr":"172.21.15.46:6837","nonce":950462730}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24500,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.62:6835/213914904","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6834","nonce":213914904},{"type":"v1","addr":"172.21.15.62:6835","nonce":213914904}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-08T23:04:38.798478+0000","modified":"2024-03-08T23:13:33.835701+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24578,"mds_1":24560,"mds_2":24553},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24520":{"gid":24520,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/3258548049","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":3258548049},{"type":"v1","addr":"172.21.15.192:6835","nonce":3258548049}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24524":{"gid":24524,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.62:6837/2774760539","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6836","nonce":2774760539},{"type":"v1","addr":"172.21.15.62:6837","nonce":2774760539}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/3408320280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":3408320280},{"type":"v1","addr":"172.21.15.192:6837","nonce":3408320280}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24553":{"gid":24553,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":13,"addr":"172.21.15.192:6839/441315746","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":441315746},{"type":"v1","addr":"172.21.15.192:6839","nonce":441315746}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24560":{"gid":24560,"name":"a","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.46:6841/2979513578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6840","nonce":2979513578},{"type":"v1","addr":"172.21.15.46:6841","nonce":2979513578}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24578":{"gid":24578,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.62:6839/1938633866","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6838","nonce":1938633866},{"type":"v1","addr":"172.21.15.62:6839","nonce":1938633866}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-08T23:18:56.314 INFO:teuthology.orchestra.run.smithi046.stderr:dumped fsmap epoch 31 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.313+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 msgr2=0x7ff60c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.313+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 0x7ff60c0a4eb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff624108b10 tx=0x7ff614005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 msgr2=0x7ff62419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff61800e9f0 tx=0x7ff61800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 shutdown_connections 2024-03-08T23:18:56.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff60c0a29f0 0x7ff60c0a4eb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff624109560 0x7ff624199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff624103e60 0x7ff62419fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 --2- 172.21.15.46:0/3060521983 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff6241034a0 0x7ff62419f8d0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 >> 172.21.15.46:0/3060521983 conn(0x7ff6240fef50 msgr2=0x7ff624100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 shutdown_connections 2024-03-08T23:18:56.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.314+0000 7ff62c600640 1 -- 172.21.15.46:0/3060521983 wait complete. 2024-03-08T23:18:56.337 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-08T23:18:56.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:55 smithi062 ceph-mon[35761]: pgmap v347: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:56.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:55 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1528796882' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:56.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.555+0000 7fbb0ea00640 1 -- 172.21.15.46:0/3657718930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 msgr2=0x7fbb081018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.555+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/3657718930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb081018d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8009900 tx=0x7fbaf802bc30 comp rx=0 tx=0).stop 2024-03-08T23:18:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 -- 172.21.15.46:0/3657718930 shutdown_connections 2024-03-08T23:18:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/3657718930 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb0810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/3657718930 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08101ea0 0x7fbb08105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/3657718930 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb081018d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 -- 172.21.15.46:0/3657718930 >> 172.21.15.46:0/3657718930 conn(0x7fbb080fcd90 msgr2=0x7fbb080ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:56.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 -- 172.21.15.46:0/3657718930 shutdown_connections 2024-03-08T23:18:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.556+0000 7fbb0ea00640 1 -- 172.21.15.46:0/3657718930 wait complete. 2024-03-08T23:18:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.557+0000 7fbb0ea00640 1 Processor -- start 2024-03-08T23:18:56.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.557+0000 7fbb0ea00640 1 -- start start 2024-03-08T23:18:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.566 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb08073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08074070 0x7fbb081a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb081116a0 con 0x7fbb081014f0 2024-03-08T23:18:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08111520 con 0x7fbb081065c0 2024-03-08T23:18:56.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ea00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08111820 con 0x7fbb08074070 2024-03-08T23:18:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbaffe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb08073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.568 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37684/0 (socket says 172.21.15.46:37684) 2024-03-08T23:18:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 -- 172.21.15.46:0/622477440 learned_addr learned my addr 172.21.15.46:0/622477440 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb0ca00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08074070 0x7fbb081a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 -- 172.21.15.46:0/622477440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08074070 msgr2=0x7fbb081a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.569 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08074070 0x7fbb081a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 -- 172.21.15.46:0/622477440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 msgr2=0x7fbb08073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb08073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbb07e00640 1 -- 172.21.15.46:0/622477440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb081a6450 con 0x7fbb081014f0 2024-03-08T23:18:56.570 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.558+0000 7fbaffe00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb08073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb07e00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbaf802c240 tx=0x7fbaf80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8043020 con 0x7fbb081014f0 2024-03-08T23:18:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaf80054e0 con 0x7fbb081014f0 2024-03-08T23:18:56.571 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb081a66e0 con 0x7fbb081014f0 2024-03-08T23:18:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf8003b20 con 0x7fbb081014f0 2024-03-08T23:18:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.559+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb081a6c80 con 0x7fbb081014f0 2024-03-08T23:18:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.560+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb08068530 con 0x7fbb081014f0 2024-03-08T23:18:56.572 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.562+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbaf8046020 con 0x7fbb081014f0 2024-03-08T23:18:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.566+0000 7fbb05600640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 0x7fbaec0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.566+0000 7fbaffe00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 0x7fbaec0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.566+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fbaf80f1420 con 0x7fbb081014f0 2024-03-08T23:18:56.574 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.567+0000 7fbaffe00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 0x7fbaec0a4eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb08075030 tx=0x7fbaf0005f90 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:56.575 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.571+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbaf804d020 con 0x7fbb081014f0 2024-03-08T23:18:56.777 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.775+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb08074ad0 con 0x7fbb081014f0 2024-03-08T23:18:56.782 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.780+0000 7fbb05600640 1 -- 172.21.15.46:0/622477440 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+25258 (secure 0 0 0) 0x7fbb08074ad0 con 0x7fbb081014f0 2024-03-08T23:18:56.783 INFO:teuthology.orchestra.run.smithi046.stdout: 2024-03-08T23:18:56.783 INFO:teuthology.orchestra.run.smithi046.stdout:{"epoch":104,"fsid":"ced7dd96-dd9e-11ee-95c6-87774f69a715","created":"2024-03-08T22:55:32.140469+0000","modified":"2024-03-08T23:18:14.347594+0000","last_up_change":"2024-03-08T23:03:40.259247+0000","last_in_change":"2024-03-08T23:03:20.618640+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T22:59:25.735173+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-08T23:04:36.776062+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-08T23:04:38.292398+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-08T23:04:41.925317+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a6108879-28e8-496c-9422-46c94c3d7ca1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6802","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6803","nonce":3833351406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6804","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6805","nonce":3833351406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6808","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6809","nonce":3833351406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6806","nonce":3833351406},{"type":"v1","addr":"172.21.15.46:6807","nonce":3833351406}]},"public_addr":"172.21.15.46:6803/3833351406","cluster_addr":"172.21.15.46:6805/3833351406","heartbeat_back_addr":"172.21.15.46:6809/3833351406","heartbeat_front_addr":"172.21.15.46:6807/3833351406","state":["exists","up"]},{"osd":1,"uuid":"5b19e310-9857-4d83-97f8-eab41f626164","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6801","nonce":1681427025}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6803","nonce":1681427025}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6807","nonce":1681427025}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":1681427025},{"type":"v1","addr":"172.21.15.62:6805","nonce":1681427025}]},"public_addr":"172.21.15.62:6801/1681427025","cluster_addr":"172.21.15.62:6803/1681427025","heartbeat_back_addr":"172.21.15.62:6807/1681427025","heartbeat_front_addr":"172.21.15.62:6805/1681427025","state":["exists","up"]},{"osd":2,"uuid":"aeb4bce6-c8ac-445e-a93b-fbd09d6cc2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6801","nonce":2982587479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6803","nonce":2982587479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6807","nonce":2982587479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":2982587479},{"type":"v1","addr":"172.21.15.192:6805","nonce":2982587479}]},"public_addr":"172.21.15.192:6801/2982587479","cluster_addr":"172.21.15.192:6803/2982587479","heartbeat_back_addr":"172.21.15.192:6807/2982587479","heartbeat_front_addr":"172.21.15.192:6805/2982587479","state":["exists","up"]},{"osd":3,"uuid":"0f714570-8e8a-416b-a6f3-b0a42f737b81","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6810","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6811","nonce":2986867677}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6812","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6813","nonce":2986867677}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6816","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6817","nonce":2986867677}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6814","nonce":2986867677},{"type":"v1","addr":"172.21.15.46:6815","nonce":2986867677}]},"public_addr":"172.21.15.46:6811/2986867677","cluster_addr":"172.21.15.46:6813/2986867677","heartbeat_back_addr":"172.21.15.46:6817/2986867677","heartbeat_front_addr":"172.21.15.46:6815/2986867677","state":["exists","up"]},{"osd":4,"uuid":"c5a9bc30-1ba9-4a47-8568-bb143164fc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6809","nonce":170069406}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6811","nonce":170069406}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6815","nonce":170069406}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":170069406},{"type":"v1","addr":"172.21.15.62:6813","nonce":170069406}]},"public_addr":"172.21.15.62:6809/170069406","cluster_addr":"172.21.15.62:6811/170069406","heartbeat_back_addr":"172.21.15.62:6815/170069406","heartbeat_front_addr":"172.21.15.62:6813/170069406","state":["exists","up"]},{"osd":5,"uuid":"0d68a52e-f5cd-4087-8d3d-0289330f4a9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6809","nonce":2546200396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6811","nonce":2546200396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6815","nonce":2546200396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":2546200396},{"type":"v1","addr":"172.21.15.192:6813","nonce":2546200396}]},"public_addr":"172.21.15.192:6809/2546200396","cluster_addr":"172.21.15.192:6811/2546200396","heartbeat_back_addr":"172.21.15.192:6815/2546200396","heartbeat_front_addr":"172.21.15.192:6813/2546200396","state":["exists","up"]},{"osd":6,"uuid":"2b897988-ec0f-4984-bc95-ec8014984db8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6818","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6819","nonce":2295220560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6820","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6821","nonce":2295220560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6824","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6825","nonce":2295220560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6822","nonce":2295220560},{"type":"v1","addr":"172.21.15.46:6823","nonce":2295220560}]},"public_addr":"172.21.15.46:6819/2295220560","cluster_addr":"172.21.15.46:6821/2295220560","heartbeat_back_addr":"172.21.15.46:6825/2295220560","heartbeat_front_addr":"172.21.15.46:6823/2295220560","state":["exists","up"]},{"osd":7,"uuid":"da72fad7-cbc8-48e4-8941-79da443bfc2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6817","nonce":1007444458}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6819","nonce":1007444458}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6823","nonce":1007444458}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":1007444458},{"type":"v1","addr":"172.21.15.62:6821","nonce":1007444458}]},"public_addr":"172.21.15.62:6817/1007444458","cluster_addr":"172.21.15.62:6819/1007444458","heartbeat_back_addr":"172.21.15.62:6823/1007444458","heartbeat_front_addr":"172.21.15.62:6821/1007444458","state":["exists","up"]},{"osd":8,"uuid":"a65ba870-b01f-4b9a-847c-d0aee0440027","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6817","nonce":3623660745}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6819","nonce":3623660745}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6823","nonce":3623660745}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3623660745},{"type":"v1","addr":"172.21.15.192:6821","nonce":3623660745}]},"public_addr":"172.21.15.192:6817/3623660745","cluster_addr":"172.21.15.192:6819/3623660745","heartbeat_back_addr":"172.21.15.192:6823/3623660745","heartbeat_front_addr":"172.21.15.192:6821/3623660745","state":["exists","up"]},{"osd":9,"uuid":"f73f96e3-f0af-44cf-9faa-76a4a469b378","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6826","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6827","nonce":1699784429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6828","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6829","nonce":1699784429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6832","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6833","nonce":1699784429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.46:6830","nonce":1699784429},{"type":"v1","addr":"172.21.15.46:6831","nonce":1699784429}]},"public_addr":"172.21.15.46:6827/1699784429","cluster_addr":"172.21.15.46:6829/1699784429","heartbeat_back_addr":"172.21.15.46:6833/1699784429","heartbeat_front_addr":"172.21.15.46:6831/1699784429","state":["exists","up"]},{"osd":10,"uuid":"dce7acee-a4ac-4a8c-952e-a210943385e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6825","nonce":3230310440}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6827","nonce":3230310440}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6831","nonce":3230310440}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":3230310440},{"type":"v1","addr":"172.21.15.62:6829","nonce":3230310440}]},"public_addr":"172.21.15.62:6825/3230310440","cluster_addr":"172.21.15.62:6827/3230310440","heartbeat_back_addr":"172.21.15.62:6831/3230310440","heartbeat_front_addr":"172.21.15.62:6829/3230310440","state":["exists","up"]},{"osd":11,"uuid":"8a7095bc-14f0-4fe3-b8d4-717280a11bd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6825","nonce":3216803010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6827","nonce":3216803010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6831","nonce":3216803010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":3216803010},{"type":"v1","addr":"172.21.15.192:6829","nonce":3216803010}]},"public_addr":"172.21.15.192:6825/3216803010","cluster_addr":"172.21.15.192:6827/3216803010","heartbeat_back_addr":"172.21.15.192:6831/3216803010","heartbeat_front_addr":"172.21.15.192:6829/3216803010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:32.649417+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:58:57.730748+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:21.335074+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T22:59:54.707006+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:21.535063+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:00:47.020595+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:20.455246+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:01:47.192465+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:12.716898+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:02:46.520426+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:13.435748+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:03:38.841111+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.e","mappings":[{"from":2,"to":4}]},{"pgid":"2.1c","mappings":[{"from":8,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":6}]},{"pgid":"3.21","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.15","mappings":[{"from":5,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.46:6801/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/1737358861":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/757291496":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/3556325343":"2024-03-09T22:56:39.034402+0000","172.21.15.46:6800/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:6801/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1288083028":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/1166845583":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1465757142":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6801/2941246107":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/2084846526":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1832130111":"2024-03-09T23:07:46.416978+0000","172.21.15.46:0/300257767":"2024-03-09T22:56:05.453593+0000","172.21.15.46:0/1962446760":"2024-03-09T23:07:46.416978+0000","172.21.15.46:6800/4209254516":"2024-03-09T22:56:39.034402+0000","172.21.15.46:0/256680715":"2024-03-09T22:56:39.034402+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-08T23:18:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 msgr2=0x7fbaec0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 0x7fbaec0a4eb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbb08075030 tx=0x7fbaf0005f90 comp rx=0 tx=0).stop 2024-03-08T23:18:56.785 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 msgr2=0x7fbb080735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fbaf802c240 tx=0x7fbaf80042f0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 shutdown_connections 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fbaec0a29f0 0x7fbaec0a4eb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb08074070 0x7fbb081a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fbb081065c0 0x7fbb08073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 --2- 172.21.15.46:0/622477440 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fbb081014f0 0x7fbb080735f0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 >> 172.21.15.46:0/622477440 conn(0x7fbb080fcd90 msgr2=0x7fbb080fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 shutdown_connections 2024-03-08T23:18:56.786 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:56.783+0000 7fbb0ea00640 1 -- 172.21.15.46:0/622477440 wait complete. 2024-03-08T23:18:56.806 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-08T23:18:57.031 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.028+0000 7fd63b200640 1 -- 172.21.15.46:0/4033678400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c101d80 msgr2=0x7fd63c105dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.028+0000 7fd63b200640 1 --2- 172.21.15.46:0/4033678400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c101d80 0x7fd63c105dc0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd624009900 tx=0x7fd62402bc30 comp rx=0 tx=0).stop 2024-03-08T23:18:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 -- 172.21.15.46:0/4033678400 shutdown_connections 2024-03-08T23:18:57.032 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 --2- 172.21.15.46:0/4033678400 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 --2- 172.21.15.46:0/4033678400 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c101d80 0x7fd63c105dc0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 --2- 172.21.15.46:0/4033678400 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c1017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 -- 172.21.15.46:0/4033678400 >> 172.21.15.46:0/4033678400 conn(0x7fd63c0fcc90 msgr2=0x7fd63c0ff0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:57.033 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 -- 172.21.15.46:0/4033678400 shutdown_connections 2024-03-08T23:18:57.034 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.029+0000 7fd63b200640 1 -- 172.21.15.46:0/4033678400 wait complete. 2024-03-08T23:18:57.038 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 Processor -- start 2024-03-08T23:18:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 -- start start 2024-03-08T23:18:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 0x7fd63c19eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.039 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c111530 con 0x7fd63c19e6f0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c1113b0 con 0x7fd63c1013d0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.030+0000 7fd63b200640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63c1116b0 con 0x7fd63c1064a0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd63a800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 0x7fd63c19eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55006/0 (socket says 172.21.15.46:55006) 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 -- 172.21.15.46:0/2000059206 learned_addr learned my addr 172.21.15.46:0/2000059206 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 -- 172.21.15.46:0/2000059206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 msgr2=0x7fd63c19a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.040 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c19a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 -- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 msgr2=0x7fd63c19eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 0x7fd63c19eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 -- 172.21.15.46:0/2000059206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63c19f1e0 con 0x7fd63c1013d0 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639400640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd63a800640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 0x7fd63c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.031+0000 7fd639e00640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009d50 tx=0x7fd62c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:57.041 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.032+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c003b70 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.032+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd62c003d10 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.032+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd63c19f4d0 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.032+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c010a20 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.032+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd63c1a6ce0 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.033+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63c068000 con 0x7fd63c1013d0 2024-03-08T23:18:57.042 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.034+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd62c003ec0 con 0x7fd63c1013d0 2024-03-08T23:18:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.036+0000 7fd632a00640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 0x7fd6200a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.036+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9475+0+0 (secure 0 0 0) 0x7fd62c0cf890 con 0x7fd63c1013d0 2024-03-08T23:18:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.036+0000 7fd639400640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 0x7fd6200a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.043 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.037+0000 7fd639400640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 0x7fd6200a4eb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd624036040 tx=0x7fd624000bc0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:57.044 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.040+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd62c09c1d0 con 0x7fd63c1013d0 2024-03-08T23:18:57.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:56 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/3060521983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:56 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/622477440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.287 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.284+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd63c1a6fa0 con 0x7fd63c1013d0 2024-03-08T23:18:57.299 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.296+0000 7fd632a00640 1 -- 172.21.15.46:0/2000059206 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v32) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd62c09bad0 con 0x7fd63c1013d0 2024-03-08T23:18:57.300 INFO:teuthology.orchestra.run.smithi046.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-08T23:18:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.299+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 msgr2=0x7fd6200a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.299+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 0x7fd6200a4eb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd624036040 tx=0x7fd624000bc0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.299+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 msgr2=0x7fd63c199e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.299+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fd62c009d50 tx=0x7fd62c00c6a0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.302 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 shutdown_connections 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7fd6200a29f0 0x7fd6200a4eb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7fd63c19e6f0 0x7fd63c19eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd63c1064a0 0x7fd63c19a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 --2- 172.21.15.46:0/2000059206 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7fd63c1013d0 0x7fd63c199e30 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 >> 172.21.15.46:0/2000059206 conn(0x7fd63c0fcc90 msgr2=0x7fd63c0fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 shutdown_connections 2024-03-08T23:18:57.303 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.300+0000 7fd63b200640 1 -- 172.21.15.46:0/2000059206 wait complete. 2024-03-08T23:18:57.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:56 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/3060521983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:56 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/622477440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.321 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-08T23:18:57.351 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:56 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/3060521983' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.351 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:56 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/622477440' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-08T23:18:57.524 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 -- 172.21.15.46:0/1689907166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c5010d7d0 msgr2=0x7f1c5010dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 --2- 172.21.15.46:0/1689907166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c5010d7d0 0x7f1c5010dbb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f1c44009860 tx=0x7f1c4402bc10 comp rx=0 tx=0).stop 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 -- 172.21.15.46:0/1689907166 shutdown_connections 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 --2- 172.21.15.46:0/1689907166 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c501081d0 0x7f1c50114620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 --2- 172.21.15.46:0/1689907166 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c50107830 0x7f1c50107c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 --2- 172.21.15.46:0/1689907166 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c5010d7d0 0x7f1c5010dbb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 -- 172.21.15.46:0/1689907166 >> 172.21.15.46:0/1689907166 conn(0x7f1c50077110 msgr2=0x7f1c50106110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:57.525 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 -- 172.21.15.46:0/1689907166 shutdown_connections 2024-03-08T23:18:57.526 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.522+0000 7f1c56400640 1 -- 172.21.15.46:0/1689907166 wait complete. 2024-03-08T23:18:57.527 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 Processor -- start 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 -- start start 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c50107830 0x7f1c5019e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5010d7d0 0x7f1c501a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50117b90 con 0x7f1c5010d7d0 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50117a10 con 0x7f1c501081d0 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.523+0000 7f1c56400640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50117d10 con 0x7f1c50107830 2024-03-08T23:18:57.528 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.62:3300/0 says I am v2:172.21.15.46:55034/0 (socket says 172.21.15.46:55034) 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 -- 172.21.15.46:0/812413547 learned_addr learned my addr 172.21.15.46:0/812413547 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c55000640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c50107830 0x7f1c5019e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c55a00640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5010d7d0 0x7f1c501a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 -- 172.21.15.46:0/812413547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c50107830 msgr2=0x7f1c5019e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c50107830 0x7f1c5019e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 -- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5010d7d0 msgr2=0x7f1c501a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5010d7d0 0x7f1c501a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:57.529 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 -- 172.21.15.46:0/812413547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c501a3000 con 0x7f1c501081d0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.524+0000 7f1c4fe00640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00e9f0 tx=0x7f1c3c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.525+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c00cd80 con 0x7f1c501081d0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.525+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c3c010910 con 0x7f1c501081d0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.525+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c501a32f0 con 0x7f1c501081d0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.525+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c3c018990 con 0x7f1c501081d0 2024-03-08T23:18:57.530 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.525+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c501aab80 con 0x7f1c501081d0 2024-03-08T23:18:57.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.526+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c50068450 con 0x7f1c501081d0 2024-03-08T23:18:57.531 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.528+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1c3c010ac0 con 0x7f1c501081d0 2024-03-08T23:18:57.532 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.530+0000 7f1c4d600640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 0x7f1c380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:57.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.530+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9645+0+0 (secure 0 0 0) 0x7f1c3c0cf600 con 0x7f1c501081d0 2024-03-08T23:18:57.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.530+0000 7f1c55000640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 0x7f1c380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:57.533 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.531+0000 7f1c55000640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 0x7f1c380a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c4402c220 tx=0x7f1c440056d0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:57.535 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.533+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1c3c09b8b0 con 0x7f1c501081d0 2024-03-08T23:18:57.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mds[65509]: -- 172.21.15.62:0/3957520137 <== mon.2 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55796a9a3a40 con 0x55796a977000 2024-03-08T23:18:57.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: starting mds.h at 2024-03-08T23:18:57.642 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.642 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.642 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mds[65820]: -- 172.21.15.62:0/1364344470 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d594335a40 con 0x55d594308c00 2024-03-08T23:18:57.642 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: starting mds.b at 2024-03-08T23:18:57.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a[31855]: 2024-03-08T23:18:57.287+0000 7f5a0e400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T23:18:57.725 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.725 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.725 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mds[70677]: -- 172.21.15.46:0/3624693845 <== mon.0 v2:172.21.15.46:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556efc07ba40 con 0x556efc04ec00 2024-03-08T23:18:57.726 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: starting mds.a at 2024-03-08T23:18:57.781 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:57.779+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1c5010fca0 con 0x7f1c501081d0 2024-03-08T23:18:57.812 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.813 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.813 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mds[68116]: -- 172.21.15.192:0/3899127289 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ead59ada40 con 0x55ead5980800 2024-03-08T23:18:57.813 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: starting mds.i at 2024-03-08T23:18:57.813 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.813 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mds[67502]: -- 172.21.15.192:0/4133977307 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8e6aada40 con 0x55a8e6a80c00 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: starting mds.f at 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c[67803]: ignoring --setuser ceph since I am not root 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c[67803]: ignoring --setgroup ceph since I am not root 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mds[67807]: -- 172.21.15.192:0/4022938107 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f01fe9a40 con 0x561f01fbd000 2024-03-08T23:18:57.814 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c[67803]: starting mds.c at 2024-03-08T23:18:58.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: pgmap v348: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:58.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/2000059206' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: osdmap e105: 12 total, 12 up, 12 in 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-08T23:18:58.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/812413547' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:57 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.300 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.298+0000 7f1c4d600640 1 -- 172.21.15.46:0/812413547 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1c3c09f950 con 0x7f1c501081d0 2024-03-08T23:18:58.304 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.302+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 msgr2=0x7f1c380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.302+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 0x7f1c380a4e30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1c4402c220 tx=0x7f1c440056d0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.302+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 msgr2=0x7f1c5019e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:58.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.302+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1c3c00e9f0 tx=0x7f1c3c00eeb0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 shutdown_connections 2024-03-08T23:18:58.306 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f1c380a2970 0x7f1c380a4e30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f1c5010d7d0 0x7f1c501a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f1c501081d0 0x7f1c5019e550 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 --2- 172.21.15.46:0/812413547 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1c50107830 0x7f1c5019e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 >> 172.21.15.46:0/812413547 conn(0x7f1c50077110 msgr2=0x7f1c50078150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:58.307 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 shutdown_connections 2024-03-08T23:18:58.308 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.303+0000 7f1c56400640 1 -- 172.21.15.46:0/812413547 wait complete. 2024-03-08T23:18:58.312 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: pgmap v348: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/2000059206' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: osdmap e105: 12 total, 12 up, 12 in 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-08T23:18:58.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/812413547' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:57 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.328 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-08T23:18:58.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: pgmap v348: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/2000059206' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-08T23:18:58.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: osdmap e105: 12 total, 12 up, 12 in 2024-03-08T23:18:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-03-08T23:18:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/812413547' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:57 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:18:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 -- 172.21.15.46:0/126326556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103eb0 msgr2=0x7f0178107ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 --2- 172.21.15.46:0/126326556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103eb0 0x7f0178107ef0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0164009900 tx=0x7f016402bc30 comp rx=0 tx=0).stop 2024-03-08T23:18:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 -- 172.21.15.46:0/126326556 shutdown_connections 2024-03-08T23:18:58.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 --2- 172.21.15.46:0/126326556 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f01781085d0 0x7f01781101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 --2- 172.21.15.46:0/126326556 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103eb0 0x7f0178107ef0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 --2- 172.21.15.46:0/126326556 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f0178103500 0x7f01781038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 -- 172.21.15.46:0/126326556 >> 172.21.15.46:0/126326556 conn(0x7f01780fed60 msgr2=0x7f0178101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.552+0000 7f0177e00640 1 -- 172.21.15.46:0/126326556 shutdown_connections 2024-03-08T23:18:58.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.553+0000 7f0177e00640 1 -- 172.21.15.46:0/126326556 wait complete. 2024-03-08T23:18:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.554+0000 7f0177e00640 1 Processor -- start 2024-03-08T23:18:58.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.554+0000 7f0177e00640 1 -- start start 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.554+0000 7f0177e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 0x7f017819f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01781085d0 0x7f017819fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 0x7f0178199e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0178113660 con 0x7f01781085d0 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01781134e0 con 0x7f0178103500 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01781137e0 con 0x7f0178199980 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 0x7f0178199e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:58.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 0x7f017819f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177400640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 0x7f0178199e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57898/0 (socket says 172.21.15.46:57898) 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0177400640 1 -- 172.21.15.46:0/580126178 learned_addr learned my addr 172.21.15.46:0/580126178 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176000640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01781085d0 0x7f017819fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 -- 172.21.15.46:0/580126178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 msgr2=0x7f0178199e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 0x7f0178199e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.560 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 -- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01781085d0 msgr2=0x7f017819fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01781085d0 0x7f017819fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.555+0000 7f0176a00640 1 -- 172.21.15.46:0/580126178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017819a4a0 con 0x7f0178103500 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f0176a00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 0x7f017819f7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f016800e9f0 tx=0x7f016800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f016800ccc0 con 0x7f0178103500 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0168010910 con 0x7f0178103500 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0168018960 con 0x7f0178103500 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017819a790 con 0x7f0178103500 2024-03-08T23:18:58.561 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.556+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01781a6a30 con 0x7f0178103500 2024-03-08T23:18:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.557+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0178068450 con 0x7f0178103500 2024-03-08T23:18:58.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.559+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0168010ac0 con 0x7f0178103500 2024-03-08T23:18:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.561+0000 7f016f400640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 0x7f015c0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.561+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9645+0+0 (secure 0 0 0) 0x7f01680cf6f0 con 0x7f0178103500 2024-03-08T23:18:58.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.561+0000 7f0176000640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 0x7f015c0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:58.564 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.562+0000 7f0176000640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 0x7f015c0a4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f0164000bc0 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:58.567 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.565+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f016809bf90 con 0x7f0178103500 2024-03-08T23:18:58.770 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:58.768+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0178103eb0 con 0x7f0178103500 2024-03-08T23:18:59.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-08T23:18:59.225 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: Cluster is now healthy 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6834/2105112732,v1:172.21.15.192:6835/2105112732] up:boot 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6837/2456620740,v1:172.21.15.62:6839/2456620740] up:boot 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:18:59.226 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.62:6836/2852182990,v1:172.21.15.62:6838/2852182990] up:boot 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6836/3479330298,v1:172.21.15.192:6838/3479330298] up:boot 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.192:6837/3147286503,v1:172.21.15.192:6839/3147286503] up:boot 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: mds.? [v2:172.21.15.46:6840/114945365,v1:172.21.15.46:6841/114945365] up:boot 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: fsmap 12 up:standby 2024-03-08T23:18:59.227 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:18:59.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:18:59.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/580126178' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.228 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:18:58 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.305 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.303+0000 7f016f400640 1 -- 172.21.15.46:0/580126178 <== mon.2 v2:172.21.15.62:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v106) v1 ==== 181+0+0 (secure 0 0 0) 0x7f016809b890 con 0x7f0178103500 2024-03-08T23:18:59.306 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_metadata' removed 2024-03-08T23:18:59.311 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.309+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 msgr2=0x7f015c0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.309+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 0x7f015c0a4eb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f0164000bc0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.309+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 msgr2=0x7f017819f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:59.312 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.309+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 0x7f017819f7d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f016800e9f0 tx=0x7f016800eeb0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: Cluster is now healthy 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6834/2105112732,v1:172.21.15.192:6835/2105112732] up:boot 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:18:59.313 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6837/2456620740,v1:172.21.15.62:6839/2456620740] up:boot 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.62:6836/2852182990,v1:172.21.15.62:6838/2852182990] up:boot 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6836/3479330298,v1:172.21.15.192:6838/3479330298] up:boot 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.192:6837/3147286503,v1:172.21.15.192:6839/3147286503] up:boot 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: mds.? [v2:172.21.15.46:6840/114945365,v1:172.21.15.46:6841/114945365] up:boot 2024-03-08T23:18:59.314 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:18:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: fsmap 12 up:standby 2024-03-08T23:18:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:18:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:18:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/580126178' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.315 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:18:58 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 shutdown_connections 2024-03-08T23:18:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f015c0a29f0 0x7f015c0a4eb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.316 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0178199980 0x7f0178199e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f01781085d0 0x7f017819fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 --2- 172.21.15.46:0/580126178 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f0178103500 0x7f017819f7d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 >> 172.21.15.46:0/580126178 conn(0x7f01780fed60 msgr2=0x7f0178101060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.311+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 shutdown_connections 2024-03-08T23:18:59.317 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.312+0000 7f0177e00640 1 -- 172.21.15.46:0/580126178 wait complete. 2024-03-08T23:18:59.342 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: Cluster is now healthy 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6834/2105112732,v1:172.21.15.192:6835/2105112732] up:boot 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-08T23:18:59.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6837/2456620740,v1:172.21.15.62:6839/2456620740] up:boot 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.62:6836/2852182990,v1:172.21.15.62:6838/2852182990] up:boot 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6836/3479330298,v1:172.21.15.192:6838/3479330298] up:boot 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.192:6837/3147286503,v1:172.21.15.192:6839/3147286503] up:boot 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: mds.? [v2:172.21.15.46:6840/114945365,v1:172.21.15.46:6841/114945365] up:boot 2024-03-08T23:18:59.392 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-08T23:18:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: fsmap 12 up:standby 2024-03-08T23:18:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-08T23:18:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='mgr.25559 172.21.15.46:0/2789865130' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-08T23:18:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/580126178' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.393 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:18:58 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:18:59.549 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 -- 172.21.15.46:0/1599181452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c073690 msgr2=0x7ff76c073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 --2- 172.21.15.46:0/1599181452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c073690 0x7ff76c073a70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ff75c009900 tx=0x7ff75c02bcd0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 -- 172.21.15.46:0/1599181452 shutdown_connections 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 --2- 172.21.15.46:0/1599181452 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c112290 0x7ff76c114680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 --2- 172.21.15.46:0/1599181452 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073fb0 0x7ff76c111d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 --2- 172.21.15.46:0/1599181452 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c073690 0x7ff76c073a70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 -- 172.21.15.46:0/1599181452 >> 172.21.15.46:0/1599181452 conn(0x7ff76c0786f0 msgr2=0x7ff76c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:18:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 -- 172.21.15.46:0/1599181452 shutdown_connections 2024-03-08T23:18:59.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.547+0000 7ff771800640 1 -- 172.21.15.46:0/1599181452 wait complete. 2024-03-08T23:18:59.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.548+0000 7ff771800640 1 Processor -- start 2024-03-08T23:18:59.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 -- start start 2024-03-08T23:18:59.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 0x7ff76c19e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:59.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c073fb0 0x7ff76c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:59.553 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 0x7ff76c1a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c117ce0 con 0x7ff76c112290 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c117b60 con 0x7ff76c073fb0 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff771800640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c117e60 con 0x7ff76c073690 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff76aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 0x7ff76c19e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff76b400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 0x7ff76c1a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:59.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff76aa00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 0x7ff76c19e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.46:57916/0 (socket says 172.21.15.46:57916) 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.549+0000 7ff76aa00640 1 -- 172.21.15.46:0/4241543315 learned_addr learned my addr 172.21.15.46:0/4241543315 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff762000640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c073fb0 0x7ff76c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 -- 172.21.15.46:0/4241543315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 msgr2=0x7ff76c19e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 0x7ff76c19e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 -- 172.21.15.46:0/4241543315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c073fb0 msgr2=0x7ff76c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c073fb0 0x7ff76c19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:18:59.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 -- 172.21.15.46:0/4241543315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c1a31b0 con 0x7ff76c112290 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.550+0000 7ff76b400640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 0x7ff76c1a2a80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff764005fd0 tx=0x7ff76400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.551+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764003b70 con 0x7ff76c112290 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.551+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff764003d10 con 0x7ff76c112290 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.551+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c1a34a0 con 0x7ff76c112290 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.551+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff764010830 con 0x7ff76c112290 2024-03-08T23:18:59.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.551+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c1aad30 con 0x7ff76c112290 2024-03-08T23:18:59.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.552+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c068530 con 0x7ff76c112290 2024-03-08T23:18:59.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.554+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff764004020 con 0x7ff76c112290 2024-03-08T23:18:59.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.555+0000 7ff763e00640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 0x7ff7500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:18:59.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.556+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9120+0+0 (secure 0 0 0) 0x7ff7640cf660 con 0x7ff76c112290 2024-03-08T23:18:59.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.556+0000 7ff76aa00640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 0x7ff7500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:18:59.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.557+0000 7ff76aa00640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 0x7ff7500a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff75c006010 tx=0x7ff75c005a20 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:18:59.562 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.560+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff76409c130 con 0x7ff76c112290 2024-03-08T23:18:59.778 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:18:59.775+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff76c1ab010 con 0x7ff76c112290 2024-03-08T23:19:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.316+0000 7ff763e00640 1 -- 172.21.15.46:0/4241543315 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v107) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff76409ba30 con 0x7ff76c112290 2024-03-08T23:19:00.318 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data' removed 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.321+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 msgr2=0x7ff7500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.321+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 0x7ff7500a4e30 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff75c006010 tx=0x7ff75c005a20 comp rx=0 tx=0).stop 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.321+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 msgr2=0x7ff76c1a2a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.321+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 0x7ff76c1a2a80 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff764005fd0 tx=0x7ff76400c6a0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 shutdown_connections 2024-03-08T23:19:00.326 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7ff7500a2970 0x7ff7500a4e30 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7ff76c112290 0x7ff76c1a2a80 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7ff76c073fb0 0x7ff76c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 --2- 172.21.15.46:0/4241543315 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff76c073690 0x7ff76c19e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 >> 172.21.15.46:0/4241543315 conn(0x7ff76c0786f0 msgr2=0x7ff76c10dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 shutdown_connections 2024-03-08T23:19:00.327 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.323+0000 7ff771800640 1 -- 172.21.15.46:0/4241543315 wait complete. 2024-03-08T23:19:00.358 DEBUG:teuthology.orchestra.run.smithi046:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-08T23:19:00.550 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 -- 172.21.15.46:0/2751070940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68101ca0 msgr2=0x7f2e68105ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 --2- 172.21.15.46:0/2751070940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68101ca0 0x7f2e68105ce0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f2e50009900 tx=0x7f2e5002bc60 comp rx=0 tx=0).stop 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 -- 172.21.15.46:0/2751070940 shutdown_connections 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 --2- 172.21.15.46:0/2751070940 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681063c0 0x7f2e6810df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 --2- 172.21.15.46:0/2751070940 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68101ca0 0x7f2e68105ce0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 --2- 172.21.15.46:0/2751070940 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e681012f0 0x7f2e681016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.551 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 -- 172.21.15.46:0/2751070940 >> 172.21.15.46:0/2751070940 conn(0x7f2e680fcb90 msgr2=0x7f2e680fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:19:00.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 -- 172.21.15.46:0/2751070940 shutdown_connections 2024-03-08T23:19:00.552 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.548+0000 7f2e67e00640 1 -- 172.21.15.46:0/2751070940 wait complete. 2024-03-08T23:19:00.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.549+0000 7f2e67e00640 1 Processor -- start 2024-03-08T23:19:00.554 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.549+0000 7f2e67e00640 1 -- start start 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.549+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 0x7f2e680735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.549+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e681063c0 0x7f2e68073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67e00640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67e00640 1 -- --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e68111520 con 0x7f2e68074070 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67e00640 1 -- --> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e681113a0 con 0x7f2e681012f0 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67e00640 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e681116a0 con 0x7f2e681063c0 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:19:00.555 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 --2- >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.46:3300/0 says I am v2:172.21.15.46:37812/0 (socket says 172.21.15.46:37812) 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 -- 172.21.15.46:0/1574700288 learned_addr learned my addr 172.21.15.46:0/1574700288 (peer_addr_for_me v2:172.21.15.46:0/0) 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e66a00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 0x7f2e680735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 -- 172.21.15.46:0/1574700288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e681063c0 msgr2=0x7f2e68073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e681063c0 0x7f2e68073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 -- 172.21.15.46:0/1574700288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 msgr2=0x7f2e680735f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 0x7f2e680735f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e67400640 1 -- 172.21.15.46:0/1574700288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e681a62b0 con 0x7f2e68074070 2024-03-08T23:19:00.556 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.550+0000 7f2e66a00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 0x7f2e680735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e67400640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e970 tx=0x7f2e6000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000ccc0 con 0x7f2e68074070 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e6000ce60 con 0x7f2e68074070 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e681a65a0 con 0x7f2e68074070 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e60010490 con 0x7f2e68074070 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.551+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e681a6aa0 con 0x7f2e68074070 2024-03-08T23:19:00.557 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.552+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e68068000 con 0x7f2e68074070 2024-03-08T23:19:00.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.554+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 4 ==== mgrmap(e 27) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2e60005020 con 0x7f2e68074070 2024-03-08T23:19:00.558 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.556+0000 7f2e5fe00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 0x7f2e4c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:19:00.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.557+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 8582+0+0 (secure 0 0 0) 0x7f2e600cf4a0 con 0x7f2e68074070 2024-03-08T23:19:00.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.557+0000 7f2e66a00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 0x7f2e4c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:19:00.559 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.557+0000 7f2e66a00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 0x7f2e4c0a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800c420 tx=0x7f2e58009570 comp rx=0 tx=0).ready entity=mgr.25559 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:19:00.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:00 smithi192 ceph-mon[36700]: pgmap v350: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 7 op/s 2024-03-08T23:19:00.562 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:00 smithi192 ceph-mon[36700]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:00.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:00 smithi192 ceph-mon[36700]: osdmap e106: 12 total, 12 up, 12 in 2024-03-08T23:19:00.563 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:00 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:00.563 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.561+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2e6009c160 con 0x7f2e68074070 2024-03-08T23:19:00.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:00 smithi062 ceph-mon[35761]: pgmap v350: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 7 op/s 2024-03-08T23:19:00.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:00 smithi062 ceph-mon[35761]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:00.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:00 smithi062 ceph-mon[35761]: osdmap e106: 12 total, 12 up, 12 in 2024-03-08T23:19:00.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:00 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:00.724 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-mon[31859]: pgmap v350: 129 pgs: 129 active+clean; 78 MiB data, 511 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 921 B/s wr, 7 op/s 2024-03-08T23:19:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-mon[31859]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-mon[31859]: osdmap e106: 12 total, 12 up, 12 in 2024-03-08T23:19:00.725 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:00.763 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:00.761+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 --> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e68101ca0 con 0x7f2e68074070 2024-03-08T23:19:01.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:19:00.939+0000 7f239d800700 -1 client.15471: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-08T23:19:01.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:19:00.943+0000 7f239d800700 -1 client.15471: SimpleRADOSStriper: lock: snap_db_v0.db: lock failed: (2) No such file or directory 2024-03-08T23:19:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.320+0000 7f2e5fe00640 1 -- 172.21.15.46:0/1574700288 <== mon.0 v2:172.21.15.46:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v108) v1 ==== 178+0+0 (secure 0 0 0) 0x7f2e6009ba60 con 0x7f2e68074070 2024-03-08T23:19:01.322 INFO:teuthology.orchestra.run.smithi046.stderr:pool 'cephfs_data_ec' removed 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.325+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 msgr2=0x7f2e4c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.325+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 0x7f2e4c0a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800c420 tx=0x7f2e58009570 comp rx=0 tx=0).stop 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.325+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 msgr2=0x7f2e681a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.325+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f2e6000e970 tx=0x7f2e6000ed70 comp rx=0 tx=0).stop 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.325+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 shutdown_connections 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.326+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:6800/2505290629,v1:172.21.15.46:6801/2505290629] conn(0x7f2e4c0a29a0 0x7f2e4c0a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:01.328 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.326+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.46:3300/0,v1:172.21.15.46:6789/0] conn(0x7f2e68074070 0x7f2e681a5bb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.326+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e681063c0 0x7f2e68073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.326+0000 7f2e67e00640 1 --2- 172.21.15.46:0/1574700288 >> [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] conn(0x7f2e681012f0 0x7f2e680735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:19:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.326+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 >> 172.21.15.46:0/1574700288 conn(0x7f2e680fcb90 msgr2=0x7f2e680fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:19:01.329 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.327+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 shutdown_connections 2024-03-08T23:19:01.331 INFO:teuthology.orchestra.run.smithi046.stderr:2024-03-08T23:19:01.329+0000 7f2e67e00640 1 -- 172.21.15.46:0/1574700288 wait complete. 2024-03-08T23:19:01.362 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:19:01.446 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:19:01.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:19:01.521 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-08T23:19:01.522 DEBUG:teuthology.orchestra.run.smithi046:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-08T23:19:01.576 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-08T23:19:01.594 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-08T23:19:01.611 INFO:tasks.cephadm:Stopping all daemons... 2024-03-08T23:19:01.611 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-08T23:19:01.611 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a 2024-03-08T23:19:01.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:01 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:01.640 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:01 smithi062 ceph-mon[35761]: osdmap e107: 12 total, 12 up, 12 in 2024-03-08T23:19:01.641 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:01 smithi062 ceph-mon[35761]: from='client.? 172.21.15.46:0/1574700288' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:01.698 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:01 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:01.698 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:01 smithi046 ceph-mon[31859]: osdmap e107: 12 total, 12 up, 12 in 2024-03-08T23:19:01.699 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:01 smithi046 ceph-mon[31859]: from='client.? 172.21.15.46:0/1574700288' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:01.812 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:01 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/4241543315' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-08T23:19:01.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:01 smithi192 ceph-mon[36700]: osdmap e107: 12 total, 12 up, 12 in 2024-03-08T23:19:01.813 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:01 smithi192 ceph-mon[36700]: from='client.? 172.21.15.46:0/1574700288' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-08T23:19:01.974 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:01 smithi046 systemd[1]: Stopping Ceph mon.a for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a[31855]: 2024-03-08T23:19:02.215+0000 7f5a15200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a[31855]: 2024-03-08T23:19:02.215+0000 7f5a15200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-08T23:19:02.475 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 podman[104260]: 2024-03-08 23:19:02.257002012 +0000 UTC m=+0.194132597 container died 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-08T23:19:02.953 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 podman[104260]: 2024-03-08 23:19:02.698289095 +0000 UTC m=+0.635419733 container cleanup 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:02.953 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 bash[104260]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a 2024-03-08T23:19:03.224 INFO:journalctl@ceph.mon.a.smithi046.stdout:Mar 08 23:19:02 smithi046 podman[104272]: 2024-03-08 23:19:02.949848849 +0000 UTC m=+0.679009914 container remove 874a63e74f7de188a3721b2488ebb390b78cfeda770930108a88879e312b7db8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-a, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-08T23:19:03.506 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.a.service' 2024-03-08T23:19:03.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:03.657 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-08T23:19:03.657 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-08T23:19:03.657 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.b 2024-03-08T23:19:03.979 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:03 smithi062 systemd[1]: Stopping Ceph mon.b for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:04.390 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-b[35757]: 2024-03-08T23:19:03.976+0000 7fc13c2c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:04.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-b[35757]: 2024-03-08T23:19:03.976+0000 7fc13c2c2700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-08T23:19:04.391 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:04 smithi062 podman[71734]: 2024-03-08 23:19:04.212172439 +0000 UTC m=+0.353241982 container died 600f2b51dbe428faf4eac7135a342b5f098d855988640055ac4242ab2d44a023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:19:04.855 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:04 smithi062 podman[71734]: 2024-03-08 23:19:04.602569272 +0000 UTC m=+0.743638829 container cleanup 600f2b51dbe428faf4eac7135a342b5f098d855988640055ac4242ab2d44a023 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:19:04.856 INFO:journalctl@ceph.mon.b.smithi062.stdout:Mar 08 23:19:04 smithi062 bash[71734]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-b 2024-03-08T23:19:05.025 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.b.service' 2024-03-08T23:19:05.082 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:05.082 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-08T23:19:05.084 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-08T23:19:05.084 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.c 2024-03-08T23:19:05.402 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 systemd[1]: Stopping Ceph mon.c for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:05.684 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c[36696]: 2024-03-08T23:19:05.399+0000 7f47f94e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:05.685 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c[36696]: 2024-03-08T23:19:05.399+0000 7f47f94e5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-08T23:19:05.685 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 podman[74160]: 2024-03-08 23:19:05.512451701 +0000 UTC m=+0.230040287 container died 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-08T23:19:06.062 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 podman[74160]: 2024-03-08 23:19:05.885338539 +0000 UTC m=+0.602927118 container cleanup 287209fb75c39753aaed6db5ab14707a42a1439bea6d6cdcbdd92eca57058152 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:19:06.063 INFO:journalctl@ceph.mon.c.smithi192.stdout:Mar 08 23:19:05 smithi192 bash[74160]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mon-c 2024-03-08T23:19:06.365 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mon.c.service' 2024-03-08T23:19:06.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:06.411 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-08T23:19:06.411 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-08T23:19:06.412 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x 2024-03-08T23:19:06.975 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:06 smithi046 systemd[1]: Stopping Ceph mgr.x for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:07.397 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:19:07.092+0000 7f2447e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:07.397 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x[32101]: 2024-03-08T23:19:07.092+0000 7f2447e00700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-08T23:19:07.397 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 podman[104381]: 2024-03-08 23:19:07.205677755 +0000 UTC m=+0.311375295 container died 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:08.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 podman[104381]: 2024-03-08 23:19:07.753312121 +0000 UTC m=+0.859009639 container cleanup 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-08T23:19:08.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 bash[104381]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x 2024-03-08T23:19:08.225 INFO:journalctl@ceph.mgr.x.smithi046.stdout:Mar 08 23:19:07 smithi046 podman[104395]: 2024-03-08 23:19:07.988260011 +0000 UTC m=+0.762567623 container remove 924ab432f56eb39016c32b839e03f07316894d36bdd77d71cd5a58509ae0d8c8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-08T23:19:08.529 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-08T23:19:08.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:08.658 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-08T23:19:08.658 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-08T23:19:08.658 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.y 2024-03-08T23:19:08.996 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:19:08 smithi062 systemd[1]: Stopping Ceph mgr.y for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:09.330 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:19:08 smithi062 podman[71855]: 2024-03-08 23:19:08.994530445 +0000 UTC m=+0.134638470 container died 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-08T23:19:09.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:19:09 smithi062 podman[71855]: 2024-03-08 23:19:09.424663694 +0000 UTC m=+0.564771713 container cleanup 4bbe139a01e345b9601f363e32026f1ca0236befb8f0a69c31d2a3e040be7300 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-08T23:19:09.641 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Mar 08 23:19:09 smithi062 bash[71855]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-y 2024-03-08T23:19:09.862 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-08T23:19:09.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:09.903 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-08T23:19:09.903 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-08T23:19:09.904 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.z 2024-03-08T23:19:10.289 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:19:09 smithi192 systemd[1]: Stopping Ceph mgr.z for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:10.562 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:19:10 smithi192 podman[74295]: 2024-03-08 23:19:10.287329884 +0000 UTC m=+0.197302336 container died 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:19:10.938 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:19:10 smithi192 podman[74295]: 2024-03-08 23:19:10.675823145 +0000 UTC m=+0.585795597 container cleanup 588a77c102637fbecf58928e5f579da2e454f800d117342584609e50575ba88d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True) 2024-03-08T23:19:10.938 INFO:journalctl@ceph.mgr.z.smithi192.stdout:Mar 08 23:19:10 smithi192 bash[74295]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mgr-z 2024-03-08T23:19:11.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-08T23:19:11.099 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:11.100 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-08T23:19:11.100 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-08T23:19:11.100 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.0 2024-03-08T23:19:11.474 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:11 smithi046 systemd[1]: Stopping Ceph osd.0 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:11.974 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0[43885]: 2024-03-08T23:19:11.681+0000 7febea800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:11.975 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0[43885]: 2024-03-08T23:19:11.681+0000 7febea800700 -1 osd.0 108 *** Got signal Terminated *** 2024-03-08T23:19:11.975 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0[43885]: 2024-03-08T23:19:11.681+0000 7febea800700 -1 osd.0 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:17.441 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:17 smithi046 podman[104501]: 2024-03-08 23:19:17.137469213 +0000 UTC m=+5.611996539 container died 22d01cd6bc3f76644b96c5d13f93540e7182d005d705a84ebc955468d3e99eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:19:17.724 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:17 smithi046 podman[104501]: 2024-03-08 23:19:17.575749914 +0000 UTC m=+6.050277236 container cleanup 22d01cd6bc3f76644b96c5d13f93540e7182d005d705a84ebc955468d3e99eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-08T23:19:17.725 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:17 smithi046 bash[104501]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0 2024-03-08T23:19:18.224 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:17 smithi046 podman[104521]: 2024-03-08 23:19:17.876961377 +0000 UTC m=+0.726151598 container remove 22d01cd6bc3f76644b96c5d13f93540e7182d005d705a84ebc955468d3e99eec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-08T23:19:18.974 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:18 smithi046 podman[104613]: 2024-03-08 23:19:18.575124498 +0000 UTC m=+0.036238786 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:19:18.975 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:18 smithi046 podman[104613]: 2024-03-08 23:19:18.702969808 +0000 UTC m=+0.164084133 container create bb781ccbcf2248310afad80d5fe49806d106b81863b9d8b0ba7519bba5ce24e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-08T23:19:19.469 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:19 smithi046 podman[104613]: 2024-03-08 23:19:19.007135371 +0000 UTC m=+0.468249684 container init bb781ccbcf2248310afad80d5fe49806d106b81863b9d8b0ba7519bba5ce24e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-08T23:19:19.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:19 smithi046 podman[104613]: 2024-03-08 23:19:19.018785485 +0000 UTC m=+0.479899840 container start bb781ccbcf2248310afad80d5fe49806d106b81863b9d8b0ba7519bba5ce24e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-08T23:19:19.470 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:19 smithi046 podman[104613]: 2024-03-08 23:19:19.119955983 +0000 UTC m=+0.581070338 container attach bb781ccbcf2248310afad80d5fe49806d106b81863b9d8b0ba7519bba5ce24e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-08T23:19:19.724 INFO:journalctl@ceph.osd.0.smithi046.stdout:Mar 08 23:19:19 smithi046 podman[104613]: 2024-03-08 23:19:19.468604897 +0000 UTC m=+0.929719292 container died bb781ccbcf2248310afad80d5fe49806d106b81863b9d8b0ba7519bba5ce24e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-08T23:19:20.255 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.0.service' 2024-03-08T23:19:20.370 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:20.370 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-08T23:19:20.371 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-08T23:19:20.371 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.1 2024-03-08T23:19:20.682 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:20 smithi062 systemd[1]: Stopping Ceph osd.1 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:21.140 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T23:19:20.678+0000 7f3b3fb71700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:21.141 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T23:19:20.678+0000 7f3b3fb71700 -1 osd.1 108 *** Got signal Terminated *** 2024-03-08T23:19:21.141 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1[40361]: 2024-03-08T23:19:20.678+0000 7f3b3fb71700 -1 osd.1 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:26.532 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:26 smithi062 podman[71991]: 2024-03-08 23:19:26.228094753 +0000 UTC m=+5.658004453 container died d397f1567baa93858a7a3efaa03e1e10415eb61d15878f832ce8ab57d845429a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-08T23:19:26.867 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:26 smithi062 podman[71991]: 2024-03-08 23:19:26.621602126 +0000 UTC m=+6.051511813 container cleanup d397f1567baa93858a7a3efaa03e1e10415eb61d15878f832ce8ab57d845429a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:26.867 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:26 smithi062 bash[71991]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1 2024-03-08T23:19:27.140 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:26 smithi062 podman[72009]: 2024-03-08 23:19:26.864907242 +0000 UTC m=+0.633420010 container remove d397f1567baa93858a7a3efaa03e1e10415eb61d15878f832ce8ab57d845429a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-08T23:19:27.471 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.095519433 +0000 UTC m=+0.018375363 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:19:27.471 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.216035024 +0000 UTC m=+0.138890948 container create c88db30b8a07c6b3c4fa7c731af048c65b8b914b1d20109c937440cd805a3e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1-deactivate, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:27.471 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.468963878 +0000 UTC m=+0.391819810 container init c88db30b8a07c6b3c4fa7c731af048c65b8b914b1d20109c937440cd805a3e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-08T23:19:27.774 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.472193082 +0000 UTC m=+0.395049007 container start c88db30b8a07c6b3c4fa7c731af048c65b8b914b1d20109c937440cd805a3e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:19:27.774 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.560548967 +0000 UTC m=+0.483404904 container attach c88db30b8a07c6b3c4fa7c731af048c65b8b914b1d20109c937440cd805a3e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-08T23:19:28.107 INFO:journalctl@ceph.osd.1.smithi062.stdout:Mar 08 23:19:27 smithi062 podman[72100]: 2024-03-08 23:19:27.771955159 +0000 UTC m=+0.694811090 container died c88db30b8a07c6b3c4fa7c731af048c65b8b914b1d20109c937440cd805a3e3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-08T23:19:28.403 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.1.service' 2024-03-08T23:19:28.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:28.448 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-08T23:19:28.449 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-08T23:19:28.449 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.2 2024-03-08T23:19:28.813 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:28 smithi192 systemd[1]: Stopping Ceph osd.2 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:29.312 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2[41845]: 2024-03-08T23:19:28.855+0000 7f1999fa2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:29.313 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2[41845]: 2024-03-08T23:19:28.855+0000 7f1999fa2700 -1 osd.2 108 *** Got signal Terminated *** 2024-03-08T23:19:29.313 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2[41845]: 2024-03-08T23:19:28.855+0000 7f1999fa2700 -1 osd.2 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:34.676 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:34 smithi192 podman[74419]: 2024-03-08 23:19:34.390859106 +0000 UTC m=+5.747438791 container died 7ff87dc3ac46b66288bceaa253a3c5ed5a6c932595a47982fe44f8e5baea2faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:35.044 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:34 smithi192 podman[74419]: 2024-03-08 23:19:34.765208968 +0000 UTC m=+6.121788642 container cleanup 7ff87dc3ac46b66288bceaa253a3c5ed5a6c932595a47982fe44f8e5baea2faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-08T23:19:35.044 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:34 smithi192 bash[74419]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2 2024-03-08T23:19:35.312 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74436]: 2024-03-08 23:19:35.041828431 +0000 UTC m=+0.649257584 container remove 7ff87dc3ac46b66288bceaa253a3c5ed5a6c932595a47982fe44f8e5baea2faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-08T23:19:35.631 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.281346303 +0000 UTC m=+0.017751116 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:19:35.631 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.409791629 +0000 UTC m=+0.146196423 container create 99634c74ae694b0946b6829597dab8dc02e578ff84ff6a84b52bc859630a24e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True) 2024-03-08T23:19:35.631 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.629058464 +0000 UTC m=+0.365463272 container init 99634c74ae694b0946b6829597dab8dc02e578ff84ff6a84b52bc859630a24e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-08T23:19:35.918 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.632373727 +0000 UTC m=+0.368778530 container start 99634c74ae694b0946b6829597dab8dc02e578ff84ff6a84b52bc859630a24e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-08T23:19:35.918 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.720935969 +0000 UTC m=+0.457340774 container attach 99634c74ae694b0946b6829597dab8dc02e578ff84ff6a84b52bc859630a24e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:19:36.236 INFO:journalctl@ceph.osd.2.smithi192.stdout:Mar 08 23:19:35 smithi192 podman[74532]: 2024-03-08 23:19:35.916225159 +0000 UTC m=+0.652629962 container died 99634c74ae694b0946b6829597dab8dc02e578ff84ff6a84b52bc859630a24e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-08T23:19:36.574 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.2.service' 2024-03-08T23:19:36.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:36.618 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-08T23:19:36.619 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-08T23:19:36.619 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.3 2024-03-08T23:19:36.975 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:36 smithi046 systemd[1]: Stopping Ceph osd.3 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:37.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:37.259+0000 7f9992200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:37.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:37.259+0000 7f9992200700 -1 osd.3 108 *** Got signal Terminated *** 2024-03-08T23:19:37.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:37.259+0000 7f9992200700 -1 osd.3 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:37.553 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:37 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:37.549+0000 7f998da00700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.736731+0000 front 2024-03-08T23:19:13.736788+0000 (oldest deadline 2024-03-08T23:19:37.236275+0000) 2024-03-08T23:19:38.974 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:38 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:38.506+0000 7f998da00700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.736731+0000 front 2024-03-08T23:19:13.736788+0000 (oldest deadline 2024-03-08T23:19:37.236275+0000) 2024-03-08T23:19:39.313 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:38.955+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:39.724 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:39.466+0000 7f998da00700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.736731+0000 front 2024-03-08T23:19:13.736788+0000 (oldest deadline 2024-03-08T23:19:37.236275+0000) 2024-03-08T23:19:39.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:39.549+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:40.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:39 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:39.939+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:40.252 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:39.923+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:40.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:40.250+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:40.724 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:40 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:40.417+0000 7f998da00700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.736731+0000 front 2024-03-08T23:19:13.736788+0000 (oldest deadline 2024-03-08T23:19:37.236275+0000) 2024-03-08T23:19:40.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:40.506+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:41.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:40 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:40.908+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:41.288 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:40.935+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:41.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:41.286+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:41.724 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:41 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3[50554]: 2024-03-08T23:19:41.436+0000 7f998da00700 -1 osd.3 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.736731+0000 front 2024-03-08T23:19:13.736788+0000 (oldest deadline 2024-03-08T23:19:37.236275+0000) 2024-03-08T23:19:41.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:41 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:41.484+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:42.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:41 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:41.869+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:42.300 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:41.939+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:42.300 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:41.854+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:42.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:42.298+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:42.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:42 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:42.511+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:42.954 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:42 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:42.909+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:42.954 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:42 smithi046 podman[104730]: 2024-03-08 23:19:42.679474445 +0000 UTC m=+5.558564410 container stop d92eab0fe036fe359152455978d00d98a87a97f85945f8e686eca93e958c2397 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:42.954 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:42 smithi046 podman[104730]: 2024-03-08 23:19:42.705598246 +0000 UTC m=+5.584688225 container died d92eab0fe036fe359152455978d00d98a87a97f85945f8e686eca93e958c2397 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, ceph=True) 2024-03-08T23:19:43.224 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:43 smithi046 podman[104730]: 2024-03-08 23:19:43.085863756 +0000 UTC m=+5.964953759 container cleanup d92eab0fe036fe359152455978d00d98a87a97f85945f8e686eca93e958c2397 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-08T23:19:43.225 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:43 smithi046 bash[104730]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3 2024-03-08T23:19:43.304 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:42.931+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:43.304 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:42.834+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:43.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:43.301+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:43.724 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:43 smithi046 podman[104748]: 2024-03-08 23:19:43.303814335 +0000 UTC m=+0.601966966 container remove d92eab0fe036fe359152455978d00d98a87a97f85945f8e686eca93e958c2397 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-08T23:19:43.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:43 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:43.551+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:44.173 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:43 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:43.911+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:44.278 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:43.919+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:44.278 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:43.834+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:44.442 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.077780671 +0000 UTC m=+0.038881459 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:19:44.443 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.17951045 +0000 UTC m=+0.140611259 container create a11a65330a663909c932f0b66cdeb85519875b1dfbcb2a5dceb2d8d19c45453b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-08T23:19:44.443 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.439716228 +0000 UTC m=+0.400817101 container init a11a65330a663909c932f0b66cdeb85519875b1dfbcb2a5dceb2d8d19c45453b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0) 2024-03-08T23:19:44.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:44.276+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:44.724 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.451395696 +0000 UTC m=+0.412496487 container start a11a65330a663909c932f0b66cdeb85519875b1dfbcb2a5dceb2d8d19c45453b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=) 2024-03-08T23:19:44.725 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.590278425 +0000 UTC m=+0.551379252 container attach a11a65330a663909c932f0b66cdeb85519875b1dfbcb2a5dceb2d8d19c45453b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:44.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:44 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:44.524+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:45.224 INFO:journalctl@ceph.osd.3.smithi046.stdout:Mar 08 23:19:44 smithi046 podman[104837]: 2024-03-08 23:19:44.914635985 +0000 UTC m=+0.875736840 container died a11a65330a663909c932f0b66cdeb85519875b1dfbcb2a5dceb2d8d19c45453b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:19:45.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:44 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:44.873+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:45.235 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:44.950+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:45.236 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:44.823+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:45.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:45.233+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:45.683 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.3.service' 2024-03-08T23:19:45.801 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:45.801 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-08T23:19:45.802 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-08T23:19:45.802 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.4 2024-03-08T23:19:45.836 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:45 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:45.543+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:45.893 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:45 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:45.848+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:46.120 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:45 smithi062 systemd[1]: Stopping Ceph osd.4 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:46.254 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:45.997+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:46.254 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:45.837+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:46.391 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:46 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:46.117+0000 7f95b1a65700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:46.391 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:46 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:46.117+0000 7f95b1a65700 -1 osd.4 108 *** Got signal Terminated *** 2024-03-08T23:19:46.391 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:46 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:46.117+0000 7f95b1a65700 -1 osd.4 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:46.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:46.251+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:46.829 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:46 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:46.498+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:47.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:46.980+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:47.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:46.792+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:47.140 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:46 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:46.826+0000 7f95aa051700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:22.083971+0000 front 2024-03-08T23:19:22.084008+0000 (oldest deadline 2024-03-08T23:19:46.784176+0000) 2024-03-08T23:19:47.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:46 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:46.828+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:47.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:47.231+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:47.867 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:47 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:47.458+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:47.869 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:47 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:47.865+0000 7f95aa051700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:22.083971+0000 front 2024-03-08T23:19:22.084008+0000 (oldest deadline 2024-03-08T23:19:46.784176+0000) 2024-03-08T23:19:48.062 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:47.944+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:48.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:47.750+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:48.126 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:47.804+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:48.126 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:47 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:47.804+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:48.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:48.268+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:48.867 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:48 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:48.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:48.868 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:48 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:48.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:49.062 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:48.720+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:49.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:48.896+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:49.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:48.896+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:49.140 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:48 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:48.865+0000 7f95aa051700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:22.083971+0000 front 2024-03-08T23:19:22.084008+0000 (oldest deadline 2024-03-08T23:19:46.784176+0000) 2024-03-08T23:19:49.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:48 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:48.775+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:49.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:48 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:48.775+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:49.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:49.297+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:49.825 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:49 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:49.497+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:49.826 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:49 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:49.497+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:49.826 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:49 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:49.455+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:50.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:49.696+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:50.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:49.942+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:50.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:49.942+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:50.140 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:49 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:49.820+0000 7f95aa051700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:22.083971+0000 front 2024-03-08T23:19:22.084008+0000 (oldest deadline 2024-03-08T23:19:46.784176+0000) 2024-03-08T23:19:50.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:49 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:49.984+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:50.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:49 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:49.810+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:50.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:49 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:49.810+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:50.716 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:50 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:50.334+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:50.806 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:50 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:50.420+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:50.806 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:50 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:50.468+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:50.806 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:50 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:50.468+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:51.062 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:50 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:50.713+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:51.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:50 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:50.925+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:51.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:50 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:50.925+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:51.140 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:50 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4[47202]: 2024-03-08T23:19:50.803+0000 7f95aa051700 -1 osd.4 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:22.083971+0000 front 2024-03-08T23:19:22.084008+0000 (oldest deadline 2024-03-08T23:19:46.784176+0000) 2024-03-08T23:19:51.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:50.937+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:51.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:50.785+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:51.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:50 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:50.785+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:51.709 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:51 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:51.358+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:51.721 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:51 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:51.414+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:51.721 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:51 smithi062 podman[72211]: 2024-03-08 23:19:51.44627777 +0000 UTC m=+5.443480862 container died 7ab24ed681a16bb5894e8f01d20a3e7d8e4a68692215789091b5c8765b61ec48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-08T23:19:51.721 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:51 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:51.466+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:51.721 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:51 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:51.466+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:52.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:51 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:51.706+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:52.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:51 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:51.893+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:52.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:51 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:51.893+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:52.080 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:51 smithi062 podman[72211]: 2024-03-08 23:19:51.818402468 +0000 UTC m=+5.815605577 container cleanup 7ab24ed681a16bb5894e8f01d20a3e7d8e4a68692215789091b5c8765b61ec48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-08T23:19:52.080 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:51 smithi062 bash[72211]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4 2024-03-08T23:19:52.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:51 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:51.921+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:52.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:51 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:51.740+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:52.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:51 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:51.740+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:52.391 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72229]: 2024-03-08 23:19:52.078024429 +0000 UTC m=+0.627464099 container remove 7ab24ed681a16bb5894e8f01d20a3e7d8e4a68692215789091b5c8765b61ec48 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:19:52.700 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:52 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:52.437+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:52.700 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:52 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:52.469+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:52.700 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:52 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:52.469+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:52.701 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72325]: 2024-03-08 23:19:52.344284836 +0000 UTC m=+0.017619338 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:19:52.701 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72325]: 2024-03-08 23:19:52.445577579 +0000 UTC m=+0.118912080 container create 89a92a542f1dbfe8e1ef0c40d1ec62065e9eb2ce3366c8e80b5aff072901383e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:19:52.704 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:52 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:52.338+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:52.974 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:52 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:52.955+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:52.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:52 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:52.705+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:52.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:52 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:52.705+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:53.007 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72325]: 2024-03-08 23:19:52.698320334 +0000 UTC m=+0.371654835 container init 89a92a542f1dbfe8e1ef0c40d1ec62065e9eb2ce3366c8e80b5aff072901383e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-08T23:19:53.007 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72325]: 2024-03-08 23:19:52.701766861 +0000 UTC m=+0.375101366 container start 89a92a542f1dbfe8e1ef0c40d1ec62065e9eb2ce3366c8e80b5aff072901383e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:19:53.008 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:52 smithi062 podman[72325]: 2024-03-08 23:19:52.773143133 +0000 UTC m=+0.446477650 container attach 89a92a542f1dbfe8e1ef0c40d1ec62065e9eb2ce3366c8e80b5aff072901383e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-08T23:19:53.008 INFO:journalctl@ceph.osd.4.smithi062.stdout:Mar 08 23:19:53 smithi062 podman[72325]: 2024-03-08 23:19:53.005490562 +0000 UTC m=+0.678825076 container died 89a92a542f1dbfe8e1ef0c40d1ec62065e9eb2ce3366c8e80b5aff072901383e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-4-deactivate, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-08T23:19:53.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:52 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:52.702+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:53.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:52 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:52.869+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:53.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:52 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:52.869+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:53.570 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:53 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:53.429+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:53.571 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:53 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:53.474+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:53.571 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:53 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:53.474+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:53.597 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.4.service' 2024-03-08T23:19:53.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:19:53.685 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-08T23:19:53.686 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-08T23:19:53.686 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.5 2024-03-08T23:19:53.715 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:53.321+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:53.994 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:53 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:53.732+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:53.994 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:53 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:53.732+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:53.998 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:53.734+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 systemd[1]: Stopping Ceph osd.5 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:53.910+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:53.910+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:53.995+0000 7efec2f8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:53.995+0000 7efec2f8d700 -1 osd.5 108 *** Got signal Terminated *** 2024-03-08T23:19:53.999 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:53 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:53.995+0000 7efec2f8d700 -1 osd.5 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:19:54.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:53 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:53.990+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:54.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:54 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:54.279+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:54.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:54 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:54.396+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:54.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:54 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:54.440+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:54.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:54 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:54.440+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:55.062 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:54 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:54.718+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:55.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:54 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:54.878+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:55.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:54 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:54.878+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:55.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:54 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:54.745+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:55.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:54 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:54.745+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:55.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:54 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:54.992+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:55.562 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:55.306+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:55.563 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:55.306+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:19:55.640 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:55 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:55.381+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:55.641 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:55 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:55.381+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:19:55.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:55 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:55.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:55.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:55 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:55.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:56.062 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:55.670+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:56.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:55.852+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:56.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:55 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:55.852+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:56.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:55 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:55.951+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:56.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:55 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:55.951+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:19:56.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:55 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:55.739+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:56.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:55 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:55.739+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:56.642 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:56 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:56.344+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:56.642 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:56 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:56.344+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:19:56.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:56 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:56.418+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:56.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:56 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:56.418+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:19:56.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:56 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:56.435+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:56.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:56 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:56.435+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:57.062 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:56 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:56.639+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:57.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:56 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:56.827+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:57.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:56 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:56.827+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:57.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:56.965+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:57.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:56.965+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:19:57.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:56.753+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:57.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:56.753+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:57.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:56 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:56.753+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:19:57.690 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:57.378+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:57.691 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:57.378+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:19:57.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:57.407+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:57.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:57.407+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:19:57.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:57.471+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:57.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:57 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:57.471+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:58.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:57.792+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:58.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:57.792+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:58.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:57.687+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:58.064 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:57 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:57.687+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:19:58.200 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:57.937+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:58.200 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:57.937+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:19:58.200 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:57.791+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:58.200 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:57.791+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:58.200 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:57 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:57.791+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:19:58.641 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:58.330+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:58.641 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:58.330+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:19:58.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:58.410+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:58.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:58.410+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:19:58.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:58.465+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:58.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:58 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:58.465+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:59.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:58.753+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:13.218534+0000 front 2024-03-08T23:19:13.218572+0000 (oldest deadline 2024-03-08T23:19:38.518377+0000) 2024-03-08T23:19:59.063 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5[48677]: 2024-03-08T23:19:58.753+0000 7efebf581700 -1 osd.5 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:24.320698+0000 front 2024-03-08T23:19:24.319809+0000 (oldest deadline 2024-03-08T23:19:48.419772+0000) 2024-03-08T23:19:59.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:58.638+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:59.063 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:58 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:58.638+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:19:59.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:58.949+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:19:59.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:58.949+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:19:59.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:58.834+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:19:59.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:58.834+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:19:59.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:58.834+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:19:59.717 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:59 smithi192 podman[74650]: 2024-03-08 23:19:59.392944956 +0000 UTC m=+5.510508303 container died 74c41ef27c47705ca78feef53179e7318217c4eedd17d9c5311c98cdc2a966eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:19:59.717 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:59 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:59.379+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:19:59.717 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:19:59 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:19:59.379+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:19:59.717 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:59 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:59.589+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:19:59.718 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:19:59 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:19:59.589+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:19:59.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:59 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:59.395+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:19:59.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:19:59 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:19:59.395+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:19:59.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:59 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:59.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:19:59.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:59 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:59.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:19:59.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:19:59 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:19:59.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:00.059 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:59 smithi192 podman[74650]: 2024-03-08 23:19:59.829997459 +0000 UTC m=+5.947560799 container cleanup 74c41ef27c47705ca78feef53179e7318217c4eedd17d9c5311c98cdc2a966eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-08T23:20:00.059 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:19:59 smithi192 bash[74650]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5 2024-03-08T23:20:00.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:59.969+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:00.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:19:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:19:59.969+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:00.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:59.819+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:00.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:59.819+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:00.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:19:59 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:19:59.819+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:00.312 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74668]: 2024-03-08 23:20:00.056263187 +0000 UTC m=+0.659823564 container remove 74c41ef27c47705ca78feef53179e7318217c4eedd17d9c5311c98cdc2a966eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-08T23:20:00.728 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74761]: 2024-03-08 23:20:00.391314486 +0000 UTC m=+0.017655817 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:00.728 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74761]: 2024-03-08 23:20:00.51551951 +0000 UTC m=+0.141860836 container create ab251ad62c45482c288829da41b17c285594d53ac0b5ca04935cd54a732c82bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, ceph=True, io.buildah.version=1.33.5) 2024-03-08T23:20:00.729 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:00 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:00.579+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:00.729 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:00 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:00.579+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:00.729 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:00 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:00.403+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:00.729 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:00 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:00.403+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:00.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:00 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:00.418+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:00.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:00 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:00.418+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:00.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:00 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:00.485+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:00.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:00 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:00.485+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:00.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:00 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:00.485+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:01.026 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74761]: 2024-03-08 23:20:00.726540541 +0000 UTC m=+0.352881873 container init ab251ad62c45482c288829da41b17c285594d53ac0b5ca04935cd54a732c82bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:01.026 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74761]: 2024-03-08 23:20:00.729708049 +0000 UTC m=+0.356049386 container start ab251ad62c45482c288829da41b17c285594d53ac0b5ca04935cd54a732c82bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-03-08T23:20:01.026 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:00 smithi192 podman[74761]: 2024-03-08 23:20:00.834691865 +0000 UTC m=+0.461033199 container attach ab251ad62c45482c288829da41b17c285594d53ac0b5ca04935cd54a732c82bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:20:01.224 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:01.005+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:01.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:01.005+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:01.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:00.808+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:01.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:00.808+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:01.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:00 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:00.808+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:01.313 INFO:journalctl@ceph.osd.5.smithi192.stdout:Mar 08 23:20:01 smithi192 podman[74761]: 2024-03-08 23:20:01.024242966 +0000 UTC m=+0.650584305 container died ab251ad62c45482c288829da41b17c285594d53ac0b5ca04935cd54a732c82bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:20:01.640 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:01.562+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:01.641 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:01.562+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:01.641 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:01.424+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:01.641 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:01 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:01.424+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:01.660 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.5.service' 2024-03-08T23:20:01.742 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:01.742 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-08T23:20:01.743 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-08T23:20:01.743 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.6 2024-03-08T23:20:01.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:01 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:01.395+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:01.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:01 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:01.395+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:01.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:01 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:01.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:01.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:01 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:01.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:01.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:01 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:01.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:02.058 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:01 smithi046 systemd[1]: Stopping Ceph osd.6 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:02.058 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:01.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:02.059 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:01.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:02.059 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:01 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:01.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:02.374 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:02.053+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:02.374 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:02.053+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:02.724 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:02.370+0000 7fab08c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:02.725 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:02.370+0000 7fab08c00700 -1 osd.6 108 *** Got signal Terminated *** 2024-03-08T23:20:02.725 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:02.370+0000 7fab08c00700 -1 osd.6 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:02.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:02.413+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:02.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:02.413+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:02.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:02.538+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:02.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:02 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:02.538+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:02.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:02 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:02.478+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:02.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:02 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:02.478+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:02.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:02 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:02.478+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:02.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:02 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:02.404+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:02.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:02 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:02.404+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:03.054 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:02.759+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:03.055 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:02.759+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:03.055 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:02 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:02.759+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:03.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:03.051+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:03.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:03.051+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:03.640 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:03.428+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:03.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:03.428+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:03.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:03.428+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:03.641 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:03.356+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:03.641 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:03 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:03.356+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:03.812 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:03 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:03.521+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:03.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:03 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:03.521+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:03.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:03 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:03.450+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:03.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:03 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:03.450+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:04.050 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:03.772+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:04.050 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:03.772+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:04.050 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:03 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:03.772+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:04.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:04.046+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:04.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:04.046+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:04.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:04.430+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:04.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:04.430+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:04.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:04.430+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:04.642 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:04.372+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:04.642 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:04 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:04.372+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:04.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:04.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:04.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:04.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:04.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:04.467+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:04.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:04.467+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:04.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:04 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:04.467+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:05.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:04.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:05.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:04.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:05.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:04.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:05.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:05.013+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:05.225 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:05.013+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:05.226 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:05.013+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:39.461510+0000 front 2024-03-08T23:19:39.461481+0000 (oldest deadline 2024-03-08T23:20:04.760781+0000) 2024-03-08T23:20:05.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:05.485+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:05.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:05.485+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:05.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:05.513+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:05.814 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:05.513+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:05.814 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:05 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:05.513+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:05.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:05.458+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:05.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:05.458+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:05.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:05.458+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:05.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:05.407+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:05.892 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:05.407+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:05.892 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:05 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:05.407+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:06.046 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:05.741+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:06.047 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:05.741+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:06.047 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:05 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:05.741+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:06.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:06.043+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:06.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:06.043+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:06.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:06.043+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:39.461510+0000 front 2024-03-08T23:19:39.461481+0000 (oldest deadline 2024-03-08T23:20:04.760781+0000) 2024-03-08T23:20:06.812 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:06.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:06.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:06.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:06.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:06.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:06.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:06.531+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:06.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:06.531+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:06.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:06 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:06.531+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:06.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:06.422+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:06.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:06.422+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:06.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:06.422+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:06.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:06.430+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:06.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:06.430+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:06.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:06 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:06.430+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:07.075 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:06.735+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:07.075 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:06.735+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:07.075 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:06.735+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:07.075 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:06 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:06.735+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:07.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:07.072+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:20.755376+0000 front 2024-03-08T23:19:20.755339+0000 (oldest deadline 2024-03-08T23:19:49.556437+0000) 2024-03-08T23:20:07.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:07.072+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.557700+0000 front 2024-03-08T23:19:29.557576+0000 (oldest deadline 2024-03-08T23:19:55.457511+0000) 2024-03-08T23:20:07.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6[57280]: 2024-03-08T23:20:07.072+0000 7faaff400700 -1 osd.6 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:39.461510+0000 front 2024-03-08T23:19:39.461481+0000 (oldest deadline 2024-03-08T23:20:04.760781+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:07.550+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:07.550+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:07.550+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:07.460+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:07.460+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:07.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:07 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:07.460+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:07.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:07.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:07.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:07.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:07.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:07.472+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:07.472+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:07.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:07 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:07.472+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:08.173 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:07 smithi046 podman[104953]: 2024-03-08 23:20:07.890511763 +0000 UTC m=+5.672386235 container died 8bc949816d6f7cb1a0421ff853c8425ddaa182651c932644c400aeb563f4097a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:08.173 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:07.778+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:08.173 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:07.778+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:08.173 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:07.778+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:08.174 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:07 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:07.778+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:08.474 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:08 smithi046 podman[104953]: 2024-03-08 23:20:08.280004096 +0000 UTC m=+6.061878605 container cleanup 8bc949816d6f7cb1a0421ff853c8425ddaa182651c932644c400aeb563f4097a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-08T23:20:08.475 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:08 smithi046 bash[104953]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6 2024-03-08T23:20:08.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:08.511+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:08.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:08.511+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:08.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:08.511+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:08.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:08.419+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:08.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:08.419+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:08.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:08 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:08.419+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:08.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:08.462+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:08.462+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:08.462+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:08.462+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:08.515+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:08.515+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:08.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:08 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:08.515+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:08.974 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:08 smithi046 podman[104973]: 2024-03-08 23:20:08.614486358 +0000 UTC m=+0.711381298 container remove 8bc949816d6f7cb1a0421ff853c8425ddaa182651c932644c400aeb563f4097a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-08T23:20:08.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:08 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:08.795+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:08.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:08 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:08.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:08.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:08 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:08.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:08.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:08 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:08.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:09.580 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:09 smithi046 podman[105058]: 2024-03-08 23:20:09.162155981 +0000 UTC m=+0.043078041 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:09.580 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:09 smithi046 podman[105058]: 2024-03-08 23:20:09.290709936 +0000 UTC m=+0.171632018 container create 85a9ad360dec665e7108a8b703781ba14efe0f1bc93b1a687df04fbbf88647c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:09.581 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:09 smithi046 podman[105058]: 2024-03-08 23:20:09.57766936 +0000 UTC m=+0.458591449 container init 85a9ad360dec665e7108a8b703781ba14efe0f1bc93b1a687df04fbbf88647c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:20:09.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:09.496+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:09.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:09.496+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:09.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:09.496+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:09.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:09.392+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:09.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:09.392+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:09.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:09 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:09.392+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:09.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:09.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:09.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:09.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:09.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:09.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:09.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:09.475+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:09.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:09.559+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:09.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:09.559+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:09.892 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:09 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:09.559+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:09.974 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:09 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:09.782+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:09.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:09 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:09.782+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:09.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:09 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:09.782+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:09.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:09 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:09.782+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:09.975 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:09 smithi046 podman[105058]: 2024-03-08 23:20:09.589074189 +0000 UTC m=+0.469996243 container start 85a9ad360dec665e7108a8b703781ba14efe0f1bc93b1a687df04fbbf88647c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:09.975 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:09 smithi046 podman[105058]: 2024-03-08 23:20:09.668113052 +0000 UTC m=+0.549035207 container attach 85a9ad360dec665e7108a8b703781ba14efe0f1bc93b1a687df04fbbf88647c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:20:10.371 INFO:journalctl@ceph.osd.6.smithi046.stdout:Mar 08 23:20:10 smithi046 podman[105058]: 2024-03-08 23:20:10.042910687 +0000 UTC m=+0.923832776 container died 85a9ad360dec665e7108a8b703781ba14efe0f1bc93b1a687df04fbbf88647c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-08T23:20:10.791 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.6.service' 2024-03-08T23:20:10.812 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:10.546+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:10.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:10.546+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:10.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:10.546+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:10.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:10.399+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:10.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:10.399+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:10.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:10 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:10.399+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:10.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:10.566+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:10.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:10.566+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:10.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:10.566+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:10.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:10.470+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:10.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:10.470+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:10.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:10.470+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:10.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:10 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:10.470+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:10.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:10.921 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-08T23:20:10.921 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-08T23:20:10.921 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.7 2024-03-08T23:20:11.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:10.747+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:11.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:10.747+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:11.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:10.747+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:11.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:10.747+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:11.390 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:10 smithi062 systemd[1]: Stopping Ceph osd.7 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:11.391 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.195+0000 7fce2f5f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:11.391 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.195+0000 7fce2f5f5700 -1 osd.7 108 *** Got signal Terminated *** 2024-03-08T23:20:11.391 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.195+0000 7fce2f5f5700 -1 osd.7 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:11.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:11.586+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:11.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:11.586+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:11.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:11.586+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:11.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:11.423+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:11.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:11.423+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:11.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:11 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:11.423+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:11.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.548+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:11.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.548+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:11.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:11.548+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:11.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:11.447+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:11.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:11.447+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:11.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:11.447+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:11.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:11 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:11.447+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:12.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:11.748+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:12.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:11.748+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:12.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:11.748+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:12.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:11 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:11.748+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:12.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:12.599+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:12.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:12.599+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:12.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:12.599+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:12.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:12.599+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:12.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:12.431+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:12.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:12.431+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:12.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:12.431+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:12.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:12 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:12.431+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:12.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:12.418+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:12.418+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:12.418+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:12.418+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:12.543+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:12.543+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:12.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:12 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:12.543+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:12.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:12 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:12.715+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:12.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:12 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:12.715+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:12.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:12 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:12.715+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:12.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:12 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:12.715+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:13.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:13.642+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:13.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:13.642+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:13.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:13.642+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:13.814 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:13.642+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:13.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:13.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:13.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:13.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:13.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:13.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:13.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:13 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:13.479+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:13.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:13.532+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:13.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:13.532+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:13.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:13.532+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:13.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:13.452+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:13.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:13.452+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:13.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:13.452+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:13.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:13.452+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:13.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:13 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:13.452+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:14.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:13 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:13.750+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:14.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:13 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:13.750+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:14.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:13 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:13.750+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:14.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:13 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:13.750+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:14.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:13 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:13.750+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:14.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:14.609+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:14.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:14.609+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:14.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:14.609+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:14.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:14.609+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:14.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:14.504+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:14.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:14.504+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:14.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:14.504+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:14.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:14 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:14.504+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:14.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:14.536+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:14.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:14.536+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:14.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:14.536+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:14.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:14.536+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:50.144997+0000 front 2024-03-08T23:19:50.145056+0000 (oldest deadline 2024-03-08T23:20:14.245217+0000) 2024-03-08T23:20:14.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:14.445+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:14.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:14.445+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:14.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:14.445+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:14.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:14.445+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:14.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:14 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:14.445+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:15.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:14 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:14.771+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:15.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:14 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:14.771+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:15.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:14 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:14.771+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:15.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:14 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:14.771+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:15.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:14 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:14.771+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:15.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:15.622+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:15.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:15.622+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:15.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:15.622+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:15.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:15.622+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:15.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:15.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:15.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:15.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:15.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:15.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:15.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:15 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:15.471+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:15.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:15.578+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.638805+0000 front 2024-03-08T23:19:23.638775+0000 (oldest deadline 2024-03-08T23:19:48.938890+0000) 2024-03-08T23:20:15.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:15.578+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:30.640039+0000 front 2024-03-08T23:19:30.640104+0000 (oldest deadline 2024-03-08T23:19:55.340008+0000) 2024-03-08T23:20:15.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:15.578+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.343023+0000 front 2024-03-08T23:19:40.342882+0000 (oldest deadline 2024-03-08T23:20:04.943589+0000) 2024-03-08T23:20:15.891 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7[53942]: 2024-03-08T23:20:15.578+0000 7fce2bbe9700 -1 osd.7 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:50.144997+0000 front 2024-03-08T23:19:50.145056+0000 (oldest deadline 2024-03-08T23:20:14.245217+0000) 2024-03-08T23:20:15.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:15.399+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:15.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:15.399+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:15.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:15.399+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:15.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:15.399+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:15.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:15.399+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:16.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:15 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:15.793+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:16.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:15 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:15.793+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:16.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:15 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:15.793+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:16.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:15 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:15.793+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:16.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:15 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:15.793+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:16.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:16.646+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:16.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:16.646+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:16.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:16.646+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:16.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:16.646+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:16.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:16.441+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:16.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:16.441+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:16.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:16.441+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:16.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:16 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:16.441+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:16.890 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:16 smithi062 podman[72430]: 2024-03-08 23:20:16.619804343 +0000 UTC m=+5.502244061 container died 46ff3805b2c9bf5b26d3dde4648254413380f5c450db8e309f15f6f7c906f368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:20:16.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:16 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:16.425+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:16.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:16 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:16.425+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:16.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:16 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:16.425+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:16.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:16 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:16.425+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:16.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:16 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:16.425+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:17.140 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72430]: 2024-03-08 23:20:17.087319343 +0000 UTC m=+5.969759052 container cleanup 46ff3805b2c9bf5b26d3dde4648254413380f5c450db8e309f15f6f7c906f368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-08T23:20:17.141 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 bash[72430]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7 2024-03-08T23:20:17.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:16 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:16.767+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:17.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:16 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:16.767+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:17.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:16 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:16.767+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:17.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:16 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:16.767+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:17.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:16 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:16.767+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:17.399 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:17.377+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:17.400 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:17.377+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:17.400 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:17.377+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:17.400 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:17.377+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:17.400 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:17 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:17.377+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:17.689 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:17.424+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:17.689 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:17.424+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:17.690 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:17.424+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:17.690 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:17.424+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:17.761 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72447]: 2024-03-08 23:20:17.397497047 +0000 UTC m=+0.778698039 container remove 46ff3805b2c9bf5b26d3dde4648254413380f5c450db8e309f15f6f7c906f368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-08T23:20:18.061 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72538]: 2024-03-08 23:20:17.662782501 +0000 UTC m=+0.018186268 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:18.061 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72538]: 2024-03-08 23:20:17.773490537 +0000 UTC m=+0.128894306 container create ae35b0e6e7243d51bff9a4195b72b34589541c25588fc370a8259705126a6925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-08T23:20:18.062 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72538]: 2024-03-08 23:20:17.993026282 +0000 UTC m=+0.348430059 container init ae35b0e6e7243d51bff9a4195b72b34589541c25588fc370a8259705126a6925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-08T23:20:18.062 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:17 smithi062 podman[72538]: 2024-03-08 23:20:17.996584948 +0000 UTC m=+0.351988706 container start ae35b0e6e7243d51bff9a4195b72b34589541c25588fc370a8259705126a6925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-08T23:20:18.062 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:18 smithi062 podman[72538]: 2024-03-08 23:20:18.059233913 +0000 UTC m=+0.414637687 container attach ae35b0e6e7243d51bff9a4195b72b34589541c25588fc370a8259705126a6925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-08T23:20:18.065 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:17.687+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:18.066 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:17.687+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:18.066 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:17.687+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:18.066 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:17 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:17.687+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:18.127 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:17 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:17.768+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:18.127 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:17 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:17.768+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:18.127 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:17 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:17.768+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:18.127 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:17 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:17.768+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:18.127 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:17 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:17.768+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:18.347 INFO:journalctl@ceph.osd.7.smithi062.stdout:Mar 08 23:20:18 smithi062 podman[72538]: 2024-03-08 23:20:18.300391586 +0000 UTC m=+0.655795343 container died ae35b0e6e7243d51bff9a4195b72b34589541c25588fc370a8259705126a6925 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-08T23:20:18.615 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:18.402+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:18.616 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:18.402+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:18.616 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:18.402+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:18.616 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:18.402+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:18.616 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:18.402+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:18.721 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:18.374+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:18.721 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:18.374+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:18.722 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:18.374+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:18.722 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:18.374+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:19.044 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.7.service' 2024-03-08T23:20:19.062 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:18.719+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:19.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:18.719+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:19.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:18.719+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:19.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:18 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:18.719+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:19.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:19.091 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-08T23:20:19.091 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-08T23:20:19.091 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.8 2024-03-08T23:20:19.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:18 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:18.780+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:19.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:18 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:18.780+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:19.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:18 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:18.780+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:19.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:18 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:18.780+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:19.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:18 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:18.780+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:19.409 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 systemd[1]: Stopping Ceph osd.8 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:19.640 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:19 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:19.444+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:19.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:19 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:19.444+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:19.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:19 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:19.444+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:19.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:19 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:19.444+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:19.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:19 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:19.444+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:19.707 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:19.411+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:19.707 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:19.411+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:19.411+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:19.411+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.406+0000 7f5c7dab9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.406+0000 7f5c7dab9700 -1 osd.8 108 *** Got signal Terminated *** 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.406+0000 7f5c7dab9700 -1 osd.8 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:19.708 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.704+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:19.709 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.704+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:19.709 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.704+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:19.709 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:19 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:19.704+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:20.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:19 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:19.803+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:20.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:19 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:19.803+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:20.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:19 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:19.803+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:20.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:19 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:19.803+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:20.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:19 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:19.803+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:20.697 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:20.385+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:20.697 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:20.385+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:20.697 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:20.385+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:20.698 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:20.385+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:20.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:20.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:20.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:20.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:20.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:20.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:20 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:20.421+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:21.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:20.694+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:21.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:20.694+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:21.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:20.694+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:21.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:20 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:20.694+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:21.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:20 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:20.806+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:21.713 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:21.342+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:21.713 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:21.342+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:21.713 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:21.342+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:21.713 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:21.342+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:21.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:21.342+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:21.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:21.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:21.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:21.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:21.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:21.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:21 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:21.456+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:22.062 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:21.711+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:22.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:21.711+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:22.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:21.711+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:22.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:21.711+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:22.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:21 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:21.711+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.539308+0000 front 2024-03-08T23:19:56.539319+0000 (oldest deadline 2024-03-08T23:20:21.239429+0000) 2024-03-08T23:20:22.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:22.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:22.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:22.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:22.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:22.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:21 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:21.788+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:22.369+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:22.369+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:22.369+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:22.752 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:22.369+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:22.752 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:22.369+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:22.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:22.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:22.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:22.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:22.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:22.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:22.451+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:23.062 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:22.748+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:23.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:22.748+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:23.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:22.748+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:23.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:22.748+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:23.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:22 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:22.748+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.539308+0000 front 2024-03-08T23:19:56.539319+0000 (oldest deadline 2024-03-08T23:20:21.239429+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:23.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:22 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:22.822+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:23.799 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:23.321+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:23.800 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:23.321+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:23.800 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:23.321+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:23.800 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:23.321+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:23.800 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:23.321+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:23.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:23.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:23.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:23.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:23.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:23.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:23 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:23.486+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:24.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:23.797+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.632513+0000 front 2024-03-08T23:19:14.632764+0000 (oldest deadline 2024-03-08T23:19:39.332512+0000) 2024-03-08T23:20:24.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:23.797+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:29.735400+0000 front 2024-03-08T23:19:29.735492+0000 (oldest deadline 2024-03-08T23:19:55.035524+0000) 2024-03-08T23:20:24.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:23.797+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.336777+0000 front 2024-03-08T23:19:40.336865+0000 (oldest deadline 2024-03-08T23:20:04.336863+0000) 2024-03-08T23:20:24.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:23.797+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.938799+0000 front 2024-03-08T23:19:48.938694+0000 (oldest deadline 2024-03-08T23:20:11.838529+0000) 2024-03-08T23:20:24.063 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:23 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8[55718]: 2024-03-08T23:20:23.797+0000 7f5c760a5700 -1 osd.8 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.539308+0000 front 2024-03-08T23:19:56.539319+0000 (oldest deadline 2024-03-08T23:20:21.239429+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:24.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:23 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:23.786+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:24.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:24 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:24.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:24.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:24 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:24.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:24.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:24 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:24.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:24.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:24 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:24.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:24.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:24 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:24.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:24.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:24.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:24.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:24.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:24.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:24.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:24 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:24.507+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:25.176 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:24 smithi192 podman[74866]: 2024-03-08 23:20:24.863785056 +0000 UTC m=+5.580529502 container died 489577fda7bdf333c28b98be86d05b942cb5d469d93669b087c32d1f250e21f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-08T23:20:25.176 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:25 smithi192 podman[74866]: 2024-03-08 23:20:25.174134148 +0000 UTC m=+5.890878593 container cleanup 489577fda7bdf333c28b98be86d05b942cb5d469d93669b087c32d1f250e21f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:25.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:24 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:24.789+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:25.461 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:25 smithi192 bash[74866]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8 2024-03-08T23:20:25.461 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:25 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:25.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:25.461 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:25 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:25.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:25.461 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:25 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:25.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:25.462 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:25 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:25.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:25.462 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:25 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:25.310+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:25.786 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:25 smithi192 podman[74883]: 2024-03-08 23:20:25.458694871 +0000 UTC m=+0.585305329 container remove 489577fda7bdf333c28b98be86d05b942cb5d469d93669b087c32d1f250e21f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8, ceph=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:25.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:25 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:25.555+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:26.047 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:25 smithi192 podman[74975]: 2024-03-08 23:20:25.784851624 +0000 UTC m=+0.111298554 container create 4f94a960980252996dd648f63816f624a9d0db5159ba9b33ad7f29c07a813a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD) 2024-03-08T23:20:26.048 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:25 smithi192 podman[74975]: 2024-03-08 23:20:25.691222522 +0000 UTC m=+0.017669467 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:26.048 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:26 smithi192 podman[74975]: 2024-03-08 23:20:26.045562285 +0000 UTC m=+0.372009230 container init 4f94a960980252996dd648f63816f624a9d0db5159ba9b33ad7f29c07a813a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:20:26.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:26.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:26.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:26.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:26.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:26.226 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:25 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:25.829+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:26.301 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:26 smithi192 podman[74975]: 2024-03-08 23:20:26.048904461 +0000 UTC m=+0.375351396 container start 4f94a960980252996dd648f63816f624a9d0db5159ba9b33ad7f29c07a813a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True) 2024-03-08T23:20:26.301 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:26 smithi192 podman[74975]: 2024-03-08 23:20:26.128908876 +0000 UTC m=+0.455355825 container attach 4f94a960980252996dd648f63816f624a9d0db5159ba9b33ad7f29c07a813a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.8.smithi192.stdout:Mar 08 23:20:26 smithi192 podman[74975]: 2024-03-08 23:20:26.338924303 +0000 UTC m=+0.665371233 container died 4f94a960980252996dd648f63816f624a9d0db5159ba9b33ad7f29c07a813a0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:26 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:26.297+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:26 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:26.297+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:26 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:26.297+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:26 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:26.297+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:26.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:26 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:26.297+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:26.894 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:26.895 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:26.895 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:26.895 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:26.895 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:26.895 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:26.563+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:27.011 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.8.service' 2024-03-08T23:20:27.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:27.053 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-08T23:20:27.054 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-08T23:20:27.054 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.9 2024-03-08T23:20:27.144 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:27.144 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:27.144 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:27.144 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:27.145 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:27.145 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:26 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:26.796+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:27.474 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 systemd[1]: Stopping Ceph osd.9 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:27.562 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:27 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:27.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:27.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:27 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:27.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:27.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:27 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:27.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:27.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:27 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:27.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:27.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:27 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:27.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:27.890 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:27.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:27.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:27.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:27.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:27.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:27 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:27.609+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:27.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.690+0000 7fe45f600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:27.975 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.690+0000 7fe45f600700 -1 osd.9 108 *** Got signal Terminated *** 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.690+0000 7fe45f600700 -1 osd.9 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:27.976 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:27 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:27.838+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:28.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:28.382+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:28.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:28.382+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:28.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:28.382+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:28.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:28.382+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:28.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:28 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:28.382+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:28.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:28.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:28.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:28.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:28.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:28.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:28.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:28 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:28.577+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:29.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.858+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:29.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.858+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:29.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.858+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:29.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.858+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:29.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.859+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:29.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:28 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:28.859+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:29.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:29.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:29.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:29.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:29.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:29.642 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:29.642 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:29 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:29.590+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:29.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:29 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:29.417+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:29.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:29 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:29.417+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:29.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:29 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:29.417+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:29.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:29 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:29.417+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:29.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:29 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:29.417+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:30.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.845+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:30.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.845+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:30.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.845+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:30.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.845+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:30.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.846+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:30.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:29 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:29.846+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:30.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:30 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:30.396+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:30.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:30 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:30.396+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:30.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:30 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:30.396+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:30.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:30 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:30.396+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:30.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:30 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:30.396+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:30.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:30.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:30 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:30.627+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:31.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:30 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:30.852+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:31.812 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:31.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:31.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:31.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:31.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:31.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:31.347+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:32.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:32.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:31 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:31.667+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.081625+0000 front 2024-03-08T23:19:14.081568+0000 (oldest deadline 2024-03-08T23:19:39.881864+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:25.085813+0000 front 2024-03-08T23:19:25.086211+0000 (oldest deadline 2024-03-08T23:19:47.386055+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.789739+0000 front 2024-03-08T23:19:33.790106+0000 (oldest deadline 2024-03-08T23:19:56.689975+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.792013+0000 front 2024-03-08T23:19:40.792083+0000 (oldest deadline 2024-03-08T23:20:06.692234+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:47.695808+0000 front 2024-03-08T23:19:47.695879+0000 (oldest deadline 2024-03-08T23:20:13.596005+0000) 2024-03-08T23:20:32.225 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:31 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9[64173]: 2024-03-08T23:20:31.821+0000 7fe455e00700 -1 osd.9 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.498715+0000 front 2024-03-08T23:19:56.499271+0000 (oldest deadline 2024-03-08T23:20:19.998919+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:32.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:32 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:32.331+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:32.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:32 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:32.625+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:33.474 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:33 smithi046 podman[105162]: 2024-03-08 23:20:33.160936446 +0000 UTC m=+5.613131108 container died 7b1f6682db06dd48da6a73bcf4372cebd39aed34b727c75e520b43efe63f1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-08T23:20:33.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:33.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:33.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:33.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:33.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:33.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:33 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:33.363+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:33.918 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:33 smithi046 podman[105162]: 2024-03-08 23:20:33.740011735 +0000 UTC m=+6.192206474 container cleanup 7b1f6682db06dd48da6a73bcf4372cebd39aed34b727c75e520b43efe63f1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:33.919 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:33 smithi046 bash[105162]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9 2024-03-08T23:20:34.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:34.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:34.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:34.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:34.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:34.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:34.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:33 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:33.646+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:34.224 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:33 smithi046 podman[105181]: 2024-03-08 23:20:33.915877584 +0000 UTC m=+0.741926254 container remove 7b1f6682db06dd48da6a73bcf4372cebd39aed34b727c75e520b43efe63f1b8b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-08T23:20:34.812 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:34.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:34.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:34.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:34.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:34.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:34.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:35.123 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:34 smithi046 podman[105269]: 2024-03-08 23:20:34.648820112 +0000 UTC m=+0.036863484 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:35.123 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:34 smithi046 podman[105269]: 2024-03-08 23:20:34.758349148 +0000 UTC m=+0.146392555 container create b32fe1e76f1a55acf9679c5b22e2f8d4db5cbd7dc017955afa37f46da7be7045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:35.124 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:35 smithi046 podman[105269]: 2024-03-08 23:20:35.120233458 +0000 UTC m=+0.508276925 container init b32fe1e76f1a55acf9679c5b22e2f8d4db5cbd7dc017955afa37f46da7be7045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307) 2024-03-08T23:20:35.140 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:35.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:34 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:34.680+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:35.474 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:35 smithi046 podman[105269]: 2024-03-08 23:20:35.132323 +0000 UTC m=+0.520366380 container start b32fe1e76f1a55acf9679c5b22e2f8d4db5cbd7dc017955afa37f46da7be7045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-08T23:20:35.475 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:35 smithi046 podman[105269]: 2024-03-08 23:20:35.210545486 +0000 UTC m=+0.598588923 container attach b32fe1e76f1a55acf9679c5b22e2f8d4db5cbd7dc017955afa37f46da7be7045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-08T23:20:35.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:35.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:35.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:35.714 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:35.715 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:35.715 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:35 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:35.354+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:35.869 INFO:journalctl@ceph.osd.9.smithi046.stdout:Mar 08 23:20:35 smithi046 podman[105269]: 2024-03-08 23:20:35.581459968 +0000 UTC m=+0.969503380 container died b32fe1e76f1a55acf9679c5b22e2f8d4db5cbd7dc017955afa37f46da7be7045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:36.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:36.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:36.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:36.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:36.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:36.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:36.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:35 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:35.729+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:36.190 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.9.service' 2024-03-08T23:20:36.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:36.326 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-08T23:20:36.327 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-08T23:20:36.327 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.10 2024-03-08T23:20:36.635 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 systemd[1]: Stopping Ceph osd.10 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:36.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:36.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:36.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:36.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:36.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:36.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:36 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:36.336+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:36.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.632+0000 7f4bb2426700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:36.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.632+0000 7f4bb2426700 -1 osd.10 108 *** Got signal Terminated *** 2024-03-08T23:20:36.891 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.632+0000 7f4bb2426700 -1 osd.10 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:36.892 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:36 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:36.702+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:37.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:37 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:37.372+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:38.140 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:38.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:37 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:37.678+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:38.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:38.346+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:39.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:39.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:38 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:38.707+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:39.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:39.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:39 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:39.301+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:40.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:40.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:40.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:40.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:40.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:40.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:40.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:39 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:39.671+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:40.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:40.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:40.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:40.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:40.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:40.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:40.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:40 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:40.281+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:14.185754+0000 front 2024-03-08T23:19:14.185773+0000 (oldest deadline 2024-03-08T23:19:39.485309+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6804 osd.1 since back 2024-03-08T23:19:23.586416+0000 front 2024-03-08T23:19:23.586375+0000 (oldest deadline 2024-03-08T23:19:48.286492+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.587822+0000 front 2024-03-08T23:19:33.587700+0000 (oldest deadline 2024-03-08T23:19:58.887665+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:40.588849+0000 front 2024-03-08T23:19:40.588736+0000 (oldest deadline 2024-03-08T23:20:06.989150+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:46.989630+0000 front 2024-03-08T23:19:46.989654+0000 (oldest deadline 2024-03-08T23:20:12.889763+0000) 2024-03-08T23:20:41.141 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:56.891441+0000 front 2024-03-08T23:19:56.891367+0000 (oldest deadline 2024-03-08T23:20:20.391201+0000) 2024-03-08T23:20:41.142 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:40 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10[60583]: 2024-03-08T23:20:40.708+0000 7f4baea1a700 -1 osd.10 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:06.193002+0000 front 2024-03-08T23:20:06.193123+0000 (oldest deadline 2024-03-08T23:20:28.492815+0000) 2024-03-08T23:20:41.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:41.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:41.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:41.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:41.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:41.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:41.814 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:41 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:41.311+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:42.252 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:41 smithi062 podman[72654]: 2024-03-08 23:20:41.977623273 +0000 UTC m=+5.453821803 container died c1fc44f2aee043297cee23e91c925f7b90e37799fd437eedaa60feddc626949a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:20:42.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:42.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:42.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:42.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:42.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:42.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:42.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:42.272+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:42.640 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:42 smithi062 podman[72654]: 2024-03-08 23:20:42.332071439 +0000 UTC m=+5.808269966 container cleanup c1fc44f2aee043297cee23e91c925f7b90e37799fd437eedaa60feddc626949a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-08T23:20:42.641 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:42 smithi062 bash[72654]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10 2024-03-08T23:20:43.086 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:42 smithi062 podman[72671]: 2024-03-08 23:20:42.708720962 +0000 UTC m=+0.729042544 container remove c1fc44f2aee043297cee23e91c925f7b90e37799fd437eedaa60feddc626949a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-03-08T23:20:43.363 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.084493491 +0000 UTC m=+0.094464238 container create ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:43.364 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.007453119 +0000 UTC m=+0.017423879 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:43.364 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.287177838 +0000 UTC m=+0.297148588 container init ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-08T23:20:43.364 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.29048893 +0000 UTC m=+0.300459683 container start ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:43.364 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.361885658 +0000 UTC m=+0.371856416 container attach ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:43.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:43.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:43 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:43.302+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:43.640 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:43 smithi062 podman[72759]: 2024-03-08 23:20:43.588688605 +0000 UTC m=+0.598659363 container died ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-08T23:20:44.129 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.10.service' 2024-03-08T23:20:44.160 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:44 smithi062 podman[72784]: 2024-03-08 23:20:44.108279051 +0000 UTC m=+0.514220302 container remove ea6b9bcad0b9e571f2546b3aa5b4cc80604b48675addd034879dd47e288279ac (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-08T23:20:44.160 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:44 smithi062 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-08T23:20:44.160 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:44 smithi062 systemd[1]: Stopped Ceph osd.10 for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:20:44.160 INFO:journalctl@ceph.osd.10.smithi062.stdout:Mar 08 23:20:44 smithi062 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.10.service: Consumed 18.393s CPU time. 2024-03-08T23:20:44.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:44.177 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-08T23:20:44.177 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-08T23:20:44.177 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.11 2024-03-08T23:20:44.504 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 systemd[1]: Stopping Ceph osd.11 for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:44.505 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:44.505 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:44.505 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:44.505 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:44.505 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:44.506 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:44.506 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.274+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:44.506 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.501+0000 7f958b382700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:44.506 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.501+0000 7f958b382700 -1 osd.11 108 *** Got signal Terminated *** 2024-03-08T23:20:44.812 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:44 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:44.501+0000 7f958b382700 -1 osd.11 108 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-08T23:20:45.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:45.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:45.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:45.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:45.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:45.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:45.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:45 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:45.303+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:46.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:46.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:46 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:46.263+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:47.562 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:47.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:47.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:47 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:47.291+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-08T23:20:23.407996+0000 front 2024-03-08T23:20:23.407969+0000 (oldest deadline 2024-03-08T23:20:46.307989+0000) 2024-03-08T23:20:48.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:48.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:48.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:48.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:48.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:48.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:48.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:48.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:48 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:48.290+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-08T23:20:23.407996+0000 front 2024-03-08T23:20:23.407969+0000 (oldest deadline 2024-03-08T23:20:46.307989+0000) 2024-03-08T23:20:49.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6806 osd.0 since back 2024-03-08T23:19:12.396698+0000 front 2024-03-08T23:19:12.396797+0000 (oldest deadline 2024-03-08T23:19:41.797295+0000) 2024-03-08T23:20:49.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-03-08T23:19:33.801357+0000 front 2024-03-08T23:19:33.801389+0000 (oldest deadline 2024-03-08T23:19:57.301419+0000) 2024-03-08T23:20:49.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6814 osd.3 since back 2024-03-08T23:19:41.903190+0000 front 2024-03-08T23:19:41.903289+0000 (oldest deadline 2024-03-08T23:20:06.002898+0000) 2024-03-08T23:20:49.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6812 osd.4 since back 2024-03-08T23:19:48.903944+0000 front 2024-03-08T23:19:48.903838+0000 (oldest deadline 2024-03-08T23:20:12.404019+0000) 2024-03-08T23:20:49.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-03-08T23:19:58.304923+0000 front 2024-03-08T23:19:58.304945+0000 (oldest deadline 2024-03-08T23:20:20.604999+0000) 2024-03-08T23:20:49.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.46:6822 osd.6 since back 2024-03-08T23:20:05.306262+0000 front 2024-03-08T23:20:05.306198+0000 (oldest deadline 2024-03-08T23:20:30.605919+0000) 2024-03-08T23:20:49.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.62:6820 osd.7 since back 2024-03-08T23:20:15.306923+0000 front 2024-03-08T23:20:15.307072+0000 (oldest deadline 2024-03-08T23:20:38.206803+0000) 2024-03-08T23:20:49.564 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11[62792]: 2024-03-08T23:20:49.245+0000 7f9587976700 -1 osd.11 108 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-03-08T23:20:23.407996+0000 front 2024-03-08T23:20:23.407969+0000 (oldest deadline 2024-03-08T23:20:46.307989+0000) 2024-03-08T23:20:50.238 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:49 smithi192 podman[75092]: 2024-03-08 23:20:49.981721941 +0000 UTC m=+5.611772998 container died e9e1a9429edb8a0c0bd4b20009dc40c03810d1dba2733a93135ca95f90165f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:20:50.562 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:50 smithi192 podman[75092]: 2024-03-08 23:20:50.33479287 +0000 UTC m=+5.964843947 container cleanup e9e1a9429edb8a0c0bd4b20009dc40c03810d1dba2733a93135ca95f90165f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-08T23:20:50.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:50 smithi192 bash[75092]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11 2024-03-08T23:20:50.954 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:50 smithi192 podman[75109]: 2024-03-08 23:20:50.602613561 +0000 UTC m=+0.622741630 container remove e9e1a9429edb8a0c0bd4b20009dc40c03810d1dba2733a93135ca95f90165f66 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-08T23:20:51.258 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:50 smithi192 podman[75198]: 2024-03-08 23:20:50.854948865 +0000 UTC m=+0.017261038 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:20:51.258 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:50 smithi192 podman[75198]: 2024-03-08 23:20:50.953227059 +0000 UTC m=+0.115539229 container create ad6d863ed87979acc0d209e4297501a0b7f1c8823a14e4837658331c5b0ea1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-08T23:20:51.258 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:51 smithi192 podman[75198]: 2024-03-08 23:20:51.172148239 +0000 UTC m=+0.334460409 container init ad6d863ed87979acc0d209e4297501a0b7f1c8823a14e4837658331c5b0ea1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-08T23:20:51.258 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:51 smithi192 podman[75198]: 2024-03-08 23:20:51.175165607 +0000 UTC m=+0.337477767 container start ad6d863ed87979acc0d209e4297501a0b7f1c8823a14e4837658331c5b0ea1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-08T23:20:51.258 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:51 smithi192 podman[75198]: 2024-03-08 23:20:51.255385108 +0000 UTC m=+0.417697296 container attach ad6d863ed87979acc0d209e4297501a0b7f1c8823a14e4837658331c5b0ea1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:20:51.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:Mar 08 23:20:51 smithi192 podman[75198]: 2024-03-08 23:20:51.468657373 +0000 UTC m=+0.630969552 container died ad6d863ed87979acc0d209e4297501a0b7f1c8823a14e4837658331c5b0ea1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-08T23:20:52.055 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@osd.11.service' 2024-03-08T23:20:52.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:52.096 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-08T23:20:52.097 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-08T23:20:52.097 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.a 2024-03-08T23:20:52.475 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:52 smithi046 systemd[1]: Stopping Ceph mds.a for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:53.224 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:52 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: 2024-03-08T23:20:52.756+0000 7fe26fa00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:53.224 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:52 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a[70673]: 2024-03-08T23:20:52.756+0000 7fe26fa00700 -1 mds.a *** got signal Terminated *** 2024-03-08T23:20:57.039 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:56 smithi046 podman[105373]: 2024-03-08 23:20:56.775146526 +0000 UTC m=+4.151679238 container died ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:20:57.385 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:57 smithi046 podman[105373]: 2024-03-08 23:20:57.208476425 +0000 UTC m=+4.585009138 container cleanup ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-08T23:20:57.385 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:57 smithi046 bash[105373]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a 2024-03-08T23:20:57.724 INFO:journalctl@ceph.mds.a.smithi046.stdout:Mar 08 23:20:57 smithi046 podman[105399]: 2024-03-08 23:20:57.382287284 +0000 UTC m=+0.588406610 container remove ea547086676b18614d6f13ae7306e2374811291b7821d0b0ec075efe642e9301 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-08T23:20:57.860 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.a.service' 2024-03-08T23:20:57.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:20:57.988 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-08T23:20:57.988 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-08T23:20:57.988 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.d 2024-03-08T23:20:58.474 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:20:58 smithi046 systemd[1]: Stopping Ceph mds.d for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:20:58.974 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:20:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d[69685]: 2024-03-08T23:20:58.641+0000 7faa45400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:20:58.975 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:20:58 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d[69685]: 2024-03-08T23:20:58.642+0000 7faa45400700 -1 mds.d *** got signal Terminated *** 2024-03-08T23:21:02.560 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:21:02 smithi046 podman[105504]: 2024-03-08 23:21:02.239310364 +0000 UTC m=+3.750817559 container died 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:21:02.880 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:21:02 smithi046 podman[105504]: 2024-03-08 23:21:02.651190392 +0000 UTC m=+4.162697580 container cleanup 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-08T23:21:02.880 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:21:02 smithi046 bash[105504]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d 2024-03-08T23:21:03.224 INFO:journalctl@ceph.mds.d.smithi046.stdout:Mar 08 23:21:02 smithi046 podman[105517]: 2024-03-08 23:21:02.877418875 +0000 UTC m=+0.624382364 container remove 4272141bb3263997f9605dfde3d8f02443b02cb8d299aad6320ed187a7eb8ed1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-08T23:21:03.460 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.d.service' 2024-03-08T23:21:03.573 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:03.574 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-08T23:21:03.574 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-08T23:21:03.574 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.g 2024-03-08T23:21:03.724 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:03 smithi046 systemd[1]: Stopping Ceph mds.g for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:04.475 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g[70323]: 2024-03-08T23:21:04.111+0000 7f8878400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:04.475 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:04 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g[70323]: 2024-03-08T23:21:04.111+0000 7f8878400700 -1 mds.g *** got signal Terminated *** 2024-03-08T23:21:08.462 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:08 smithi046 podman[105624]: 2024-03-08 23:21:08.133697334 +0000 UTC m=+4.145581136 container died 710f98c9daf461abda146349698c0648719980285cf1541ebe8e13be5f8125bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-08T23:21:08.725 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:08 smithi046 podman[105624]: 2024-03-08 23:21:08.579350328 +0000 UTC m=+4.591234133 container cleanup 710f98c9daf461abda146349698c0648719980285cf1541ebe8e13be5f8125bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:21:08.725 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:08 smithi046 bash[105624]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g 2024-03-08T23:21:09.224 INFO:journalctl@ceph.mds.g.smithi046.stdout:Mar 08 23:21:08 smithi046 podman[105638]: 2024-03-08 23:21:08.822129978 +0000 UTC m=+0.678318578 container remove 710f98c9daf461abda146349698c0648719980285cf1541ebe8e13be5f8125bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-g, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-08T23:21:09.325 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.g.service' 2024-03-08T23:21:09.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:09.449 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-08T23:21:09.449 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-08T23:21:09.450 DEBUG:teuthology.orchestra.run.smithi046:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.j 2024-03-08T23:21:09.974 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:09 smithi046 systemd[1]: Stopping Ceph mds.j for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:10.474 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j[69342]: 2024-03-08T23:21:10.095+0000 7f63e7800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:10.475 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:10 smithi046 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j[69342]: 2024-03-08T23:21:10.095+0000 7f63e7800700 -1 mds.j *** got signal Terminated *** 2024-03-08T23:21:14.381 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:14 smithi046 podman[105742]: 2024-03-08 23:21:14.11614243 +0000 UTC m=+4.196336084 container died ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-08T23:21:14.724 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:14 smithi046 podman[105742]: 2024-03-08 23:21:14.599075441 +0000 UTC m=+4.679269092 container cleanup ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-08T23:21:14.725 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:14 smithi046 bash[105742]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j 2024-03-08T23:21:15.223 INFO:journalctl@ceph.mds.j.smithi046.stdout:Mar 08 23:21:14 smithi046 podman[105768]: 2024-03-08 23:21:14.766761504 +0000 UTC m=+0.632421797 container remove ae76c76d253b1b2f4f5874af7d20bcb7f28d66de77e1b4c70372bf82d7556a01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-j, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-08T23:21:15.252 DEBUG:teuthology.orchestra.run.smithi046:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.j.service' 2024-03-08T23:21:15.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:15.379 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-08T23:21:15.379 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-08T23:21:15.379 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.b 2024-03-08T23:21:15.680 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:15 smithi062 systemd[1]: Stopping Ceph mds.b for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:15.958 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: 2024-03-08T23:21:15.675+0000 7ff9c976e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:15.958 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:15 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b[65816]: 2024-03-08T23:21:15.676+0000 7ff9c976e700 -1 mds.b *** got signal Terminated *** 2024-03-08T23:21:17.641 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:17 smithi062 podman[72864]: 2024-03-08 23:21:17.366339961 +0000 UTC m=+1.791275212 container died b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-08T23:21:18.119 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:17 smithi062 podman[72864]: 2024-03-08 23:21:17.702520798 +0000 UTC m=+2.127456049 container cleanup b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:21:18.119 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:17 smithi062 bash[72864]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b 2024-03-08T23:21:18.119 INFO:journalctl@ceph.mds.b.smithi062.stdout:Mar 08 23:21:17 smithi062 podman[72890]: 2024-03-08 23:21:17.895375925 +0000 UTC m=+0.524492974 container remove b0ed8db6071d975a6662e8b697e3bfe3c4fd0467e9088f5c8be19bd60ae67803 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-08T23:21:18.132 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.b.service' 2024-03-08T23:21:18.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:18.174 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-08T23:21:18.174 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-08T23:21:18.174 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.e 2024-03-08T23:21:18.390 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:18 smithi062 systemd[1]: Stopping Ceph mds.e for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:18.890 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e[65196]: 2024-03-08T23:21:18.496+0000 7f964ab05700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:18.891 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:18 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e[65196]: 2024-03-08T23:21:18.496+0000 7f964ab05700 -1 mds.e *** got signal Terminated *** 2024-03-08T23:21:21.493 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:21 smithi062 podman[72995]: 2024-03-08 23:21:21.204965252 +0000 UTC m=+2.809059797 container died cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:21:21.851 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:21 smithi062 podman[72995]: 2024-03-08 23:21:21.572830963 +0000 UTC m=+3.176925507 container cleanup cfa48cce9437e15967fb627067cf005439852be4f21871afcde41c813ca4e45f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-08T23:21:21.851 INFO:journalctl@ceph.mds.e.smithi062.stdout:Mar 08 23:21:21 smithi062 bash[72995]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-e 2024-03-08T23:21:22.014 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.e.service' 2024-03-08T23:21:22.054 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:22.055 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-08T23:21:22.055 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-08T23:21:22.055 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.h 2024-03-08T23:21:22.379 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:22 smithi062 systemd[1]: Stopping Ceph mds.h for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:22.640 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: 2024-03-08T23:21:22.375+0000 7f3989d3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:22.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:22 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h[65505]: 2024-03-08T23:21:22.375+0000 7f3989d3f700 -1 mds.h *** got signal Terminated *** 2024-03-08T23:21:25.641 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:25 smithi062 podman[73116]: 2024-03-08 23:21:25.374630272 +0000 UTC m=+3.098114443 container died 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-08T23:21:26.140 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:25 smithi062 podman[73116]: 2024-03-08 23:21:25.734898666 +0000 UTC m=+3.458382821 container cleanup 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-08T23:21:26.141 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:25 smithi062 bash[73116]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h 2024-03-08T23:21:26.141 INFO:journalctl@ceph.mds.h.smithi062.stdout:Mar 08 23:21:25 smithi062 podman[73128]: 2024-03-08 23:21:25.910986222 +0000 UTC m=+0.535396832 container remove 198957b123a1983ba31cfbd8a41e381f7c2b07ede326411c28a3870fa230bbd2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:21:26.167 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.h.service' 2024-03-08T23:21:26.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:26.206 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-08T23:21:26.206 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-08T23:21:26.206 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.k 2024-03-08T23:21:26.586 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:26 smithi062 systemd[1]: Stopping Ceph mds.k for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:26.890 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k[64874]: 2024-03-08T23:21:26.579+0000 7face676c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:26.891 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:26 smithi062 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k[64874]: 2024-03-08T23:21:26.579+0000 7face676c700 -1 mds.k *** got signal Terminated *** 2024-03-08T23:21:30.391 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:30 smithi062 podman[73238]: 2024-03-08 23:21:30.120823167 +0000 UTC m=+3.694240650 container died 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-08T23:21:30.662 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:30 smithi062 podman[73238]: 2024-03-08 23:21:30.482207972 +0000 UTC m=+4.055625496 container cleanup 242497e8c070554f5148e9b9d7cfff7afea8a0144af18118f9a5c9e50422bb13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:21:30.663 INFO:journalctl@ceph.mds.k.smithi062.stdout:Mar 08 23:21:30 smithi062 bash[73238]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-k 2024-03-08T23:21:30.858 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.k.service' 2024-03-08T23:21:30.896 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:30.897 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-08T23:21:30.897 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-08T23:21:30.897 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.c 2024-03-08T23:21:31.268 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:30 smithi192 systemd[1]: Stopping Ceph mds.c for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:31.563 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c[67803]: 2024-03-08T23:21:31.265+0000 7fb25e920700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:31.563 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:31 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c[67803]: 2024-03-08T23:21:31.265+0000 7fb25e920700 -1 mds.c *** got signal Terminated *** 2024-03-08T23:21:33.719 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:33 smithi192 podman[75304]: 2024-03-08 23:21:33.378723352 +0000 UTC m=+2.291103810 container died a231115212da4d1290a5d49c56d87db990c56374ce76fce96bd183452672ef43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-08T23:21:34.026 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:33 smithi192 podman[75304]: 2024-03-08 23:21:33.798521372 +0000 UTC m=+2.710901829 container cleanup a231115212da4d1290a5d49c56d87db990c56374ce76fce96bd183452672ef43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:21:34.027 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:33 smithi192 bash[75304]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c 2024-03-08T23:21:34.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.c.service' 2024-03-08T23:21:34.277 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:34 smithi192 podman[75329]: 2024-03-08 23:21:34.024646174 +0000 UTC m=+0.644551316 container remove a231115212da4d1290a5d49c56d87db990c56374ce76fce96bd183452672ef43 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:21:34.277 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:34 smithi192 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.c.service: Deactivated successfully. 2024-03-08T23:21:34.278 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:34 smithi192 systemd[1]: Stopped Ceph mds.c for ced7dd96-dd9e-11ee-95c6-87774f69a715. 2024-03-08T23:21:34.278 INFO:journalctl@ceph.mds.c.smithi192.stdout:Mar 08 23:21:34 smithi192 systemd[1]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.c.service: Consumed 5.271s CPU time. 2024-03-08T23:21:34.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:34.291 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-08T23:21:34.292 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-08T23:21:34.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.f 2024-03-08T23:21:34.562 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:34 smithi192 systemd[1]: Stopping Ceph mds.f for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:35.063 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: 2024-03-08T23:21:34.643+0000 7f9b7ee30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:35.063 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:34 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f[67498]: 2024-03-08T23:21:34.643+0000 7f9b7ee30700 -1 mds.f *** got signal Terminated *** 2024-03-08T23:21:37.647 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:37 smithi192 podman[75437]: 2024-03-08 23:21:37.35933556 +0000 UTC m=+2.839637488 container died 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-08T23:21:38.062 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:37 smithi192 podman[75437]: 2024-03-08 23:21:37.735711663 +0000 UTC m=+3.216013594 container cleanup 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-08T23:21:38.063 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:37 smithi192 bash[75437]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f 2024-03-08T23:21:38.063 INFO:journalctl@ceph.mds.f.smithi192.stdout:Mar 08 23:21:37 smithi192 podman[75452]: 2024-03-08 23:21:37.894936547 +0000 UTC m=+0.537824410 container remove 3f68d9560838a42c4c2cd50516e0918ac2be58a42c9bbde102e17c68439cc0e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:21:38.105 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.f.service' 2024-03-08T23:21:38.141 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:38.141 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-08T23:21:38.142 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-08T23:21:38.142 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.i 2024-03-08T23:21:38.449 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:38 smithi192 systemd[1]: Stopping Ceph mds.i for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:38.812 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: 2024-03-08T23:21:38.446+0000 7f2cb7ffd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:38.813 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:38 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i[68112]: 2024-03-08T23:21:38.446+0000 7f2cb7ffd700 -1 mds.i *** got signal Terminated *** 2024-03-08T23:21:41.701 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:41 smithi192 podman[75560]: 2024-03-08 23:21:41.375208241 +0000 UTC m=+3.014468952 container died eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:21:42.017 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:41 smithi192 podman[75560]: 2024-03-08 23:21:41.7805286 +0000 UTC m=+3.419789315 container cleanup eceea483525b6d9b86f4be4fc5a70e1c468487131065e1c9d31a691e898d394a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-08T23:21:42.017 INFO:journalctl@ceph.mds.i.smithi192.stdout:Mar 08 23:21:41 smithi192 bash[75560]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-i 2024-03-08T23:21:42.155 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.i.service' 2024-03-08T23:21:42.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:42.190 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-08T23:21:42.190 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-08T23:21:42.190 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.l 2024-03-08T23:21:42.563 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:42 smithi192 systemd[1]: Stopping Ceph mds.l for ced7dd96-dd9e-11ee-95c6-87774f69a715... 2024-03-08T23:21:42.563 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l[67189]: 2024-03-08T23:21:42.508+0000 7f520ff6c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:21:42.563 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:42 smithi192 ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l[67189]: 2024-03-08T23:21:42.508+0000 7f520ff6c700 -1 mds.l *** got signal Terminated *** 2024-03-08T23:21:46.812 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:46 smithi192 podman[75681]: 2024-03-08 23:21:46.516548034 +0000 UTC m=+4.112569376 container died 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:21:47.228 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:46 smithi192 podman[75681]: 2024-03-08 23:21:46.98398618 +0000 UTC m=+4.580007490 container cleanup 3c9dc191f9bc74282b7cb5de8b2bd3c7485c7cf50153220873f4508abd65d83e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=) 2024-03-08T23:21:47.228 INFO:journalctl@ceph.mds.l.smithi192.stdout:Mar 08 23:21:46 smithi192 bash[75681]: ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715-mds-l 2024-03-08T23:21:47.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ced7dd96-dd9e-11ee-95c6-87774f69a715@mds.l.service' 2024-03-08T23:21:47.389 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-08T23:21:47.389 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-08T23:21:47.390 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-08T23:21:47.722 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:21:52.691 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-08T23:21:52.892 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:21:55.419 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-08T23:21:55.610 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:21:58.167 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:21:58.279 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:21:58.317 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:21:58.346 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-08T23:21:58.349 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi046/crash 2024-03-08T23:21:58.350 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash -- . 2024-03-08T23:21:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:tar: /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-08T23:21:58.465 INFO:teuthology.orchestra.run.smithi046.stderr:tar: Error is not recoverable: exiting now 2024-03-08T23:21:58.471 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi062/crash 2024-03-08T23:21:58.472 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash -- . 2024-03-08T23:21:58.523 INFO:teuthology.orchestra.run.smithi062.stderr:tar: /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-08T23:21:58.524 INFO:teuthology.orchestra.run.smithi062.stderr:tar: Error is not recoverable: exiting now 2024-03-08T23:21:58.525 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi192/crash 2024-03-08T23:21:58.526 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash -- . 2024-03-08T23:21:58.567 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-08T23:21:58.567 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-03-08T23:21:58.568 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-08T23:21:58.568 DEBUG:teuthology.orchestra.run.smithi046:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-08T23:21:58.683 INFO:tasks.cephadm:Compressing logs... 2024-03-08T23:21:58.683 DEBUG:teuthology.orchestra.run.smithi046:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-08T23:21:58.688 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-08T23:21:58.692 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-08T23:21:58.742 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-08T23:21:58.742 INFO:teuthology.orchestra.run.smithi192.stderr:: No such file or directory 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.l.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.i.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.11.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.i.log: /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.8.log 2024-03-08T23:21:58.743 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.z.log: gzip 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr: -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.f.log 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.f.log: 90.1% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-08T23:21:58.744 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log: 94.5% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.l.log.gz 2024-03-08T23:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.c.log 2024-03-08T23:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.2.log: 87.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-08T23:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-08T23:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.c.log: 92.0% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-08T23:21:58.745 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-08T23:21:58.747 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-08T23:21:58.748 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr: /var/log/ceph/cephadm.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.10.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.b.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.10.log: gzip -5 --verbose/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log: -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.b.log: /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.4.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-08T23:21:58.749 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.k.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.h.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.e.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.7.log: 90.1% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.e.log: 94.3% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.k.log.gz 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-08T23:21:58.750 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.y.log: 94.3% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.e.log.gz 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log: 83.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log: 87.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-08T23:21:58.751 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log: 91.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-08T23:21:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-08T23:21:58.752 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-08T23:21:58.753 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-08T23:21:58.755 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-08T23:21:58.755 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-08T23:21:58.756 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-08T23:21:58.757 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.g.log 2024-03-08T23:21:58.758 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.j.log 2024-03-08T23:21:58.760 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-08T23:21:58.761 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-08T23:21:58.764 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.3.log: 93.2%/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.x.log: -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-08T23:21:58.764 INFO:teuthology.orchestra.run.smithi046.stderr: 89.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-08T23:21:58.764 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-08T23:21:58.766 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-08T23:21:58.767 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.0.log: 94.5% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.g.log.gz 2024-03-08T23:21:58.768 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log 2024-03-08T23:21:58.769 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log: 94.5% 83.7% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-08T23:21:58.769 INFO:teuthology.orchestra.run.smithi046.stderr: -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.j.log.gz 2024-03-08T23:21:58.769 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-08T23:21:58.771 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.9.log 2024-03-08T23:21:58.774 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log: /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.d.log 2024-03-08T23:21:58.775 INFO:teuthology.orchestra.run.smithi046.stderr: 87.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-08T23:21:58.775 INFO:teuthology.orchestra.run.smithi046.stderr:gzip -5 --verbose -- /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.a.log 2024-03-08T23:21:58.781 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.d.log: /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-08T23:21:58.789 INFO:teuthology.orchestra.run.smithi046.stderr:/var/log/ceph/ceph-client.admin.log: 90.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-08T23:21:58.791 INFO:teuthology.orchestra.run.smithi046.stderr: 94.3% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.d.log.gz 2024-03-08T23:21:58.791 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-08T23:21:59.059 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.c.log: 94.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.f.log.gz 2024-03-08T23:21:59.136 INFO:teuthology.orchestra.run.smithi062.stderr: 94.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.h.log.gz 2024-03-08T23:21:59.150 INFO:teuthology.orchestra.run.smithi046.stderr: 89.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-08T23:21:59.330 INFO:teuthology.orchestra.run.smithi062.stderr: 91.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-08T23:21:59.350 INFO:teuthology.orchestra.run.smithi192.stderr: 91.7% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-08T23:21:59.682 INFO:teuthology.orchestra.run.smithi192.stderr: 95.3% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.c.log.gz 2024-03-08T23:21:59.836 INFO:teuthology.orchestra.run.smithi046.stderr: 90.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-08T23:22:01.206 INFO:teuthology.orchestra.run.smithi062.stderr: 92.1% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-08T23:22:01.567 INFO:teuthology.orchestra.run.smithi062.stderr: 92.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-08T23:22:01.937 INFO:teuthology.orchestra.run.smithi062.stderr: 92.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.10.log.gz 2024-03-08T23:22:02.080 INFO:teuthology.orchestra.run.smithi192.stderr: 92.9% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.11.log.gz 2024-03-08T23:22:02.104 INFO:teuthology.orchestra.run.smithi046.stderr: 92.7% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-08T23:22:02.137 INFO:teuthology.orchestra.run.smithi192.stderr: 94.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.i.log.gz 2024-03-08T23:22:02.188 INFO:teuthology.orchestra.run.smithi192.stderr: 92.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-08T23:22:02.341 INFO:teuthology.orchestra.run.smithi046.stderr: 92.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.9.log.gz 2024-03-08T23:22:02.384 INFO:teuthology.orchestra.run.smithi192.stderr: 92.2% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-08T23:22:02.462 INFO:teuthology.orchestra.run.smithi192.stderr: 92.7% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.8.log.gz 2024-03-08T23:22:02.464 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-03-08T23:22:02.465 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m3.758s 2024-03-08T23:22:02.465 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m18.959s 2024-03-08T23:22:02.465 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.614s 2024-03-08T23:22:02.662 INFO:teuthology.orchestra.run.smithi046.stderr: 92.4% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-08T23:22:02.704 INFO:teuthology.orchestra.run.smithi046.stderr: 92.8% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-08T23:22:03.011 INFO:teuthology.orchestra.run.smithi062.stderr: 93.0% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-08T23:22:03.680 INFO:teuthology.orchestra.run.smithi046.stderr: 94.6% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.a.log.gz 2024-03-08T23:22:03.684 INFO:teuthology.orchestra.run.smithi046.stderr: 2024-03-08T23:22:03.685 INFO:teuthology.orchestra.run.smithi046.stderr:real 0m4.966s 2024-03-08T23:22:03.685 INFO:teuthology.orchestra.run.smithi046.stderr:user 0m19.768s 2024-03-08T23:22:03.685 INFO:teuthology.orchestra.run.smithi046.stderr:sys 0m1.512s 2024-03-08T23:22:09.060 INFO:teuthology.orchestra.run.smithi062.stderr: 93.3% -- replaced with /var/log/ceph/ced7dd96-dd9e-11ee-95c6-87774f69a715/ceph-mds.b.log.gz 2024-03-08T23:22:09.062 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-03-08T23:22:09.062 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m10.356s 2024-03-08T23:22:09.063 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m23.490s 2024-03-08T23:22:09.063 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.701s 2024-03-08T23:22:09.063 INFO:tasks.cephadm:Archiving logs... 2024-03-08T23:22:09.064 DEBUG:teuthology.misc:Transferring archived files from smithi046:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi046/log 2024-03-08T23:22:09.065 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-08T23:22:11.805 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi062/log 2024-03-08T23:22:11.806 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-08T23:22:15.102 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi192/log 2024-03-08T23:22:15.103 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-08T23:22:17.605 INFO:tasks.cephadm:Removing cluster... 2024-03-08T23:22:17.606 DEBUG:teuthology.orchestra.run.smithi046:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force 2024-03-08T23:22:17.947 INFO:teuthology.orchestra.run.smithi046.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:22:18.631 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force 2024-03-08T23:22:18.834 INFO:teuthology.orchestra.run.smithi062.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:22:19.119 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ced7dd96-dd9e-11ee-95c6-87774f69a715 --force 2024-03-08T23:22:19.309 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: ced7dd96-dd9e-11ee-95c6-87774f69a715 2024-03-08T23:22:19.587 INFO:tasks.cephadm:Removing cephadm ... 2024-03-08T23:22:19.587 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-08T23:22:19.649 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-08T23:22:19.665 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-08T23:22:19.681 INFO:tasks.cephadm:Teardown complete 2024-03-08T23:22:19.681 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-08T23:22:19.692 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-08T23:22:19.693 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-08T23:22:19.700 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-08T23:22:19.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-08T23:22:19.876 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-08T23:22:19.887 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-08T23:22:19.887 DEBUG:teuthology.orchestra.run.smithi046:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:22:19.895 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:22:19.899 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:22:19.911 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:22:19.912 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:22:19.914 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:22:19.915 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-03-08T23:22:19.915 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 63 +587us[ +587us] +/- 108ms 2024-03-08T23:22:19.915 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -1042us[-1041us] +/- 75ms 2024-03-08T23:22:19.915 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +554us[ +554us] +/- 65ms 2024-03-08T23:22:19.915 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 64 +572us[ +555us] +/- 107ms 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 63 -1032us[-1049us] +/- 75ms 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +568us[ +568us] +/- 65ms 2024-03-08T23:22:19.917 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:22:19.944 INFO:teuthology.orchestra.run.smithi046.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:22:19.954 INFO:teuthology.orchestra.run.smithi046.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:22:19.955 INFO:teuthology.orchestra.run.smithi046.stdout:=============================================================================== 2024-03-08T23:22:19.955 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +688us[ +688us] +/- 108ms 2024-03-08T23:22:19.955 INFO:teuthology.orchestra.run.smithi046.stdout:^* hv02.front.sepia.ceph.com 3 7 377 126 -842us[ -817us] +/- 74ms 2024-03-08T23:22:19.955 INFO:teuthology.orchestra.run.smithi046.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 +697us[ +697us] +/- 65ms 2024-03-08T23:22:19.955 INFO:teuthology.orchestra.run.smithi046.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:22:19.957 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-08T23:22:19.968 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-08T23:22:19.968 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-08T23:22:19.992 DEBUG:teuthology.orchestra.run.smithi046:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-08T23:22:20.253 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-08T23:22:20.336 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-08T23:22:20.409 DEBUG:teuthology.orchestra.run.smithi046:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:22:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:20.519 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:22:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:20.554 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:22:20.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:20.585 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-08T23:22:20.596 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-08T23:22:20.634 INFO:teuthology.task.internal:Duration was 2257.204060 seconds 2024-03-08T23:22:20.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-08T23:22:20.684 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-08T23:22:20.684 DEBUG:teuthology.orchestra.run.smithi046:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-08T23:22:20.692 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-08T23:22:20.695 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-08T23:22:20.737 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:22:20.744 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:22:20.860 INFO:teuthology.orchestra.run.smithi046.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:22:21.306 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-08T23:22:21.307 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi046.front.sepia.ceph.com 2024-03-08T23:22:21.307 DEBUG:teuthology.orchestra.run.smithi046:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-08T23:22:21.371 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-03-08T23:22:21.371 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-08T23:22:21.486 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-03-08T23:22:21.486 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-08T23:22:21.512 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-08T23:22:21.513 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-08T23:22:21.520 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-08T23:22:21.523 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-08T23:22:21.614 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-08T23:22:21.614 DEBUG:teuthology.orchestra.run.smithi046:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-08T23:22:21.620 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-08T23:22:21.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-08T23:22:22.146 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-08T23:22:22.158 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-08T23:22:22.159 DEBUG:teuthology.orchestra.run.smithi046:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-08T23:22:22.260 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-08T23:22:22.293 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-08T23:22:22.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-08T23:22:22.334 DEBUG:teuthology.orchestra.run.smithi046:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.342 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.372 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-03-08T23:22:22.387 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-03-08T23:22:22.450 INFO:teuthology.orchestra.run.smithi046.stdout:kernel.core_pattern = core 2024-03-08T23:22:22.510 DEBUG:teuthology.orchestra.run.smithi046:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:22.560 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:22.575 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-08T23:22:22.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:22:22.589 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-08T23:22:22.601 INFO:teuthology.task.internal:Transferring archived files... 2024-03-08T23:22:22.602 DEBUG:teuthology.misc:Transferring archived files from smithi046:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi046 2024-03-08T23:22:22.602 DEBUG:teuthology.orchestra.run.smithi046:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-08T23:22:22.725 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi062 2024-03-08T23:22:22.725 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-08T23:22:22.770 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268/remote/smithi192 2024-03-08T23:22:22.770 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-08T23:22:22.809 INFO:teuthology.task.internal:Removing archive directory... 2024-03-08T23:22:22.809 DEBUG:teuthology.orchestra.run.smithi046:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-08T23:22:22.817 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-08T23:22:22.819 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-08T23:22:22.870 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-08T23:22:22.885 INFO:teuthology.task.internal:Not uploading archives. 2024-03-08T23:22:22.885 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-08T23:22:22.909 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-08T23:22:22.910 DEBUG:teuthology.orchestra.run.smithi046:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-08T23:22:22.918 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-08T23:22:22.920 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-08T23:22:22.933 INFO:teuthology.orchestra.run.smithi062.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 8 23:22 /home/ubuntu/cephtest 2024-03-08T23:22:22.934 INFO:teuthology.orchestra.run.smithi192.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 8 23:22 /home/ubuntu/cephtest 2024-03-08T23:22:22.970 INFO:teuthology.orchestra.run.smithi046.stdout: 393243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 8 23:22 /home/ubuntu/cephtest 2024-03-08T23:22:22.973 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-08T23:22:22.985 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-08T23:22:23.110 INFO:teuthology.nuke:Checking targets against current locks 2024-03-08T23:22:23.138 DEBUG:teuthology.nuke:shortname: smithi046 2024-03-08T23:22:23.139 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T23:22:23.162 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi046.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.147602', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBALBRr494KWIsmHoG8SNXCt8oaJNNWBZyX2bnIn3439QfzY/L09+/PZFEz4wvhNKUhpD6T3z1xyvpAkzsBaH8cQ='} 2024-03-08T23:22:23.168 DEBUG:teuthology.nuke:shortname: smithi062 2024-03-08T23:22:23.168 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T23:22:23.191 DEBUG:teuthology.nuke:shortname: smithi192 2024-03-08T23:22:23.192 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T23:22:23.195 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.144262', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApETSWUc4zesRT8aG+3tvjZpZMzdaf+a+WX2EAxGi94LDYCONTl3r/H69D8WFbpi2Gwr5lKc56lZHYskcBu3XE='} 2024-03-08T23:22:23.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 22:35:38.146016', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDLttwb5bQnofE4tfKvDurVh8KmCGkXTY0TehT5eoqDhg1ji2Ug5C9xLD52pTrlFzc45Y0CuEVvZ8qxAfd77R0I='} 2024-03-08T23:22:23.228 INFO:teuthology.orchestra.console.smithi046:Power off 2024-03-08T23:22:23.228 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T23:22:23.253 DEBUG:teuthology.orchestra.console.smithi046:power off output: Chassis Power Control: Down/Off 2024-03-08T23:22:23.253 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:23.277 INFO:teuthology.orchestra.console.smithi062:Power off 2024-03-08T23:22:23.278 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T23:22:23.295 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-08T23:22:23.309 DEBUG:teuthology.orchestra.console.smithi062:power off output: Chassis Power Control: Down/Off 2024-03-08T23:22:23.309 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:23.327 INFO:teuthology.orchestra.console.smithi192:Power off 2024-03-08T23:22:23.328 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-08T23:22:23.337 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-08T23:22:23.353 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-03-08T23:22:23.353 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:23.377 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-08T23:22:27.296 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:27.338 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:27.379 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:27.407 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-08T23:22:27.449 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-08T23:22:27.492 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-08T23:22:31.412 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:31.450 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:31.492 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:31.523 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is on 2024-03-08T23:22:31.560 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is on 2024-03-08T23:22:31.603 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-03-08T23:22:35.526 DEBUG:teuthology.orchestra.console.smithi046:pexpect command: ipmitool -H smithi046.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:35.562 DEBUG:teuthology.orchestra.console.smithi062:pexpect command: ipmitool -H smithi062.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:35.604 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-08T23:22:35.638 DEBUG:teuthology.orchestra.console.smithi046:check power output: Chassis Power is off 2024-03-08T23:22:35.673 DEBUG:teuthology.orchestra.console.smithi062:check power output: Chassis Power is off 2024-03-08T23:22:35.714 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-03-08T23:22:35.739 INFO:teuthology.orchestra.console.smithi046:Power off completed 2024-03-08T23:22:35.774 INFO:teuthology.orchestra.console.smithi062:Power off completed 2024-03-08T23:22:35.815 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-03-08T23:22:35.921 INFO:teuthology.run:Summary data: client.0-kernel-sha1: dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/fsx}} duration: 2257.204060316086 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi046 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=ff70c05f0c8346a8a9b94f3f856a0109 status: fail success: false 2024-03-08T23:22:35.922 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T23:22:36.009 INFO:teuthology.run:FAIL